精华内容
下载资源
问答
  • 本文主要介绍了在Ubuntu系统上Hadoop单机版测试环境的搭建过程。
  • Ubuntu安装Hadoop(本地单机版

    千次阅读 2018-06-26 22:00:53
    Hadoop本地安装 ...需要安装好虚拟机Vmware,本文使用的是Ubuntu16。 * 下载JDK1.8 下载地址: http://www.oracle.com/technetwork/java/javase/downloads/index.html * 下载Openstack3.0 下载地址:h...

    Hadoop本地安装

    Hadoop有多种安装方式,本文讲述Hadoop本地安装方法。

    一、软件准备

    需要安装好虚拟机Vmware12,本文使用的是Ubuntu16。
    * 下载JDK1.8
    必须安装sun JDK,下载地址:
    http://www.oracle.com/technetwork/java/javase/downloads/index.html
    * 下载Hadoop3.0
    下载地址:http://hadoop.apache.org/releases.html

    二、安装和配置

    2.1 安装JDK1.8

    解压JDK安装包到~/soft/目录下,然后将目录拷贝到/usr/soft/目录下:mv jdk1.8.0_111 /usr/soft/

    环境变量配置:
    打开/etc/environment:
    JAVA_HOME=/usr/soft/jdk1.8.0_111

    系统本来自带open-JDK,需要对JDK进行配置,具体请看:
    https://blog.csdn.net/goodmentc/article/details/80959686

    2.2 安装Hadoop

    安装Hadoop:
    将Hadoop安装包解压到/home/tc/soft/目录下,然后将目录拷贝到/usr/soft/目录下:mv hadoop-3.0.3 /usr/soft/
    HADOOP_INSTALL=/home/tc/soft/hadoop-3.0.3

    PATH=”/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/soft/jdk1.8/bin:/home/tc/soft/hadoop-3.0.3/bin:/home/tc/soft/hadoop-3.0.3/sbin”

    打开终端执行命令使用环境变量生效:
    source environment

    查看hadoop安装是否成功:
    执行命令:hadoop version
    如果安装成功会看到hadoop版本信息:
    tc@ubuntu:/usr/soft/hadoop-3.0.3$ hadoop version
    Hadoop 3.0.3
    Source code repository https://yjzhangal@git-wip-us.apache.org/repos/asf/hadoop.git -r 37fd7d752db73d984dc31e0cdfd590d252f5e075
    Compiled by yzhang on 2018-05-31T17:12Z
    Compiled with protoc 2.5.0
    From source with checksum 736cdcefa911261ad56d2d120bf1fa
    This command was run using /usr/soft/hadoop-3.0.3/share/hadoop/common/hadoop-common-3.0.3.jar

    如果JDK环境变量配置未生效,则报错:
    tc@ubuntu:/usr/soft/hadoop-3.0.3$ hadoop version
    Error JAVA_HOME is not set and could not be found.

    解决方法:重启虚拟机。
    tc@ubuntu:~$ hadoop version
    Hadoop 3.0.3
    Source code repository https://yjzhangal@git-wip-us.apache.org/repos/asf/hadoop.git -r 37fd7d752db73d984dc31e0cdfd590d252f5e075
    Compiled by yzhang on 2018-05-31T17:12Z
    Compiled with protoc 2.5.0
    From source with checksum 736cdcefa911261ad56d2d120bf1fa
    This command was run using /usr/soft/hadoop-3.0.3/share/hadoop/common/hadoop-common-3.0.3.jar

    三、应用

    Hadoop自带了一个MapReduce程序$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar,它作为一个例子提供了MapReduce的基本功能,并且可以用于计算,包括 wordcount、terasort、join、grep 等。

    以通过执行如下命令查看该.jar文件支持哪些MapReduce功能。

    hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar
    tc@ubuntu:~$ hadoop jar /usr/soft/hadoop-3.0.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar 
    An example program must be given as the first argument.
    Valid program names are:
      aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
      aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
      bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
      dbcount: An example job that count the pageview counts from a database.
      distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
      grep: A map/reduce program that counts the matches of a regex in the input.
      join: A job that effects a join over sorted, equally partitioned datasets
      multifilewc: A job that counts words from several files.
      pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
      pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
      randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
      randomwriter: A map/reduce program that writes 10GB of random data per node.
      secondarysort: An example defining a secondary sort to the reduce.
      sort: A map/reduce program that sorts the data written by the random writer.
      sudoku: A sudoku solver.
      teragen: Generate data for the terasort
      terasort: Run the terasort
      teravalidate: Checking results of terasort
      wordcount: A map/reduce program that counts the words in the input files.
      wordmean: A map/reduce program that counts the average length of the words in the input files.
      wordmedian: A map/reduce program that counts the median length of the words in the input files.
      wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
    tc@ubuntu:~$   

    3.1 grep使用

    创建目录input,在input目录下创建文件test.txt, 执行命令:

    tancan@ubuntu:~$ hadoop jar /usr/soft/hadoop-3.0.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar grep input output "test"  

    执行成功后,会打印相关日志:

             ... ...
                  Merged Map outputs=1
                    GC time elapsed (ms)=0
                    Total committed heap usage (bytes)=838860800
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters 
                    Bytes Read=98
            File Output Format Counters 
                    Bytes Written=8

    查看结果:
    - 执行命令:ll output

    tancan@ubuntu:~$ ll output/
    total 16
    drwxr-xr-x  2 tancan tancan 4096 Jul  7 06:02 ./
    drwxr-xr-x 25 tancan tancan 4096 Jul  7 06:02 ../
    -rw-r--r--  1 tancan tancan    0 Jul  7 06:02 part-r-00000
    -rw-r--r--  1 tancan tancan    8 Jul  7 06:02 .part-r-00000.crc
    -rw-r--r--  1 tancan tancan    0 Jul  7 06:02 _SUCCESS
    -rw-r--r--  1 tancan tancan    8 Jul  7 06:02 ._SUCCESS.crc
    
    • 打印结果:cat output/*
    tancan@ubuntu:~$ cat output/*
    1   test

    3.2 wordcount应用

    修改input/test.txt内容:This is test file
    执行命令:

    tancan@ubuntu:~$ hadoop jar /usr/soft/hadoop-3.0.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount input output/
    2018-07-07 06:08:24,179 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2018-07-07 06:08:24,249 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
    2018-07-07 06:08:24,249 INFO impl.MetricsSystemImpl: JobTracker metrics system started
    2018-07-07 06:08:24,532 INFO input.FileInputFormat: Total input files to process : 1
    2018-07-07 06:08:24,553 INFO mapreduce.JobSubmitter: number of splits:1
    2018-07-07 06:08:24,716 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1885415322_0001
    2018-07-07 06:08:24,719 INFO mapreduce.JobSubmitter: Executing with tokens: []
    2018-07-07 06:08:24,853 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
    2018-07-07 06:08:24,854 INFO mapreduce.Job: Running job: job_local1885415322_0001
    2018-07-07 06:08:24,855 INFO mapred.LocalJobRunner: OutputCommitter set in config null
    2018-07-07 06:08:24,863 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
    2018-07-07 06:08:24,863 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2018-07-07 06:08:24,865 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    2018-07-07 06:08:24,916 INFO mapred.LocalJobRunner: Waiting for map tasks
    2018-07-07 06:08:24,916 INFO mapred.LocalJobRunner: Starting task: attempt_local1885415322_0001_m_000000_0
    2018-07-07 06:08:24,954 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
    2018-07-07 06:08:24,955 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2018-07-07 06:08:24,978 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
    2018-07-07 06:08:24,985 INFO mapred.MapTask: Processing split: file:/home/tancan/input/test.txt:0+18
    2018-07-07 06:08:25,082 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    2018-07-07 06:08:25,083 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    2018-07-07 06:08:25,083 INFO mapred.MapTask: soft limit at 83886080
    2018-07-07 06:08:25,083 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    2018-07-07 06:08:25,083 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    2018-07-07 06:08:25,089 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2018-07-07 06:08:25,102 INFO mapred.LocalJobRunner: 
    2018-07-07 06:08:25,102 INFO mapred.MapTask: Starting flush of map output
    2018-07-07 06:08:25,102 INFO mapred.MapTask: Spilling map output
    2018-07-07 06:08:25,102 INFO mapred.MapTask: bufstart = 0; bufend = 34; bufvoid = 104857600
    2018-07-07 06:08:25,102 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
    2018-07-07 06:08:25,121 INFO mapred.MapTask: Finished spill 0
    2018-07-07 06:08:25,135 INFO mapred.Task: Task:attempt_local1885415322_0001_m_000000_0 is done. And is in the process of committing
    2018-07-07 06:08:25,137 INFO mapred.LocalJobRunner: map
    2018-07-07 06:08:25,138 INFO mapred.Task: Task 'attempt_local1885415322_0001_m_000000_0' done.
    2018-07-07 06:08:25,144 INFO mapred.Task: Final Counters for attempt_local1885415322_0001_m_000000_0: Counters: 18
        File System Counters
            FILE: Number of bytes read=316165
            FILE: Number of bytes written=784641
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
        Map-Reduce Framework
            Map input records=1
            Map output records=4
            Map output bytes=34
            Map output materialized bytes=48
            Input split bytes=97
            Combine input records=4
            Combine output records=4
            Spilled Records=4
            Failed Shuffles=0
            Merged Map outputs=0
            GC time elapsed (ms)=0
            Total committed heap usage (bytes)=216006656
        File Input Format Counters 
            Bytes Read=18
    2018-07-07 06:08:25,145 INFO mapred.LocalJobRunner: Finishing task: attempt_local1885415322_0001_m_000000_0
    2018-07-07 06:08:25,145 INFO mapred.LocalJobRunner: map task executor complete.
    2018-07-07 06:08:25,149 INFO mapred.LocalJobRunner: Waiting for reduce tasks
    2018-07-07 06:08:25,150 INFO mapred.LocalJobRunner: Starting task: attempt_local1885415322_0001_r_000000_0
    2018-07-07 06:08:25,166 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
    2018-07-07 06:08:25,166 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2018-07-07 06:08:25,166 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
    2018-07-07 06:08:25,170 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4e822838
    2018-07-07 06:08:25,172 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
    2018-07-07 06:08:25,200 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=322594400, maxSingleShuffleLimit=80648600, mergeThreshold=212912320, ioSortFactor=10, memToMemMergeOutputsThreshold=10
    2018-07-07 06:08:25,202 INFO reduce.EventFetcher: attempt_local1885415322_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
    2018-07-07 06:08:25,243 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1885415322_0001_m_000000_0 decomp: 44 len: 48 to MEMORY
    2018-07-07 06:08:25,249 INFO reduce.InMemoryMapOutput: Read 44 bytes from map-output for attempt_local1885415322_0001_m_000000_0
    2018-07-07 06:08:25,250 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 44, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->44
    2018-07-07 06:08:25,251 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
    2018-07-07 06:08:25,252 INFO mapred.LocalJobRunner: 1 / 1 copied.
    2018-07-07 06:08:25,256 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
    2018-07-07 06:08:25,267 INFO mapred.Merger: Merging 1 sorted segments
    2018-07-07 06:08:25,268 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 37 bytes
    2018-07-07 06:08:25,269 INFO reduce.MergeManagerImpl: Merged 1 segments, 44 bytes to disk to satisfy reduce memory limit
    2018-07-07 06:08:25,270 INFO reduce.MergeManagerImpl: Merging 1 files, 48 bytes from disk
    2018-07-07 06:08:25,271 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
    2018-07-07 06:08:25,271 INFO mapred.Merger: Merging 1 sorted segments
    2018-07-07 06:08:25,273 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 37 bytes
    2018-07-07 06:08:25,273 INFO mapred.LocalJobRunner: 1 / 1 copied.
    2018-07-07 06:08:25,279 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
    2018-07-07 06:08:25,280 INFO mapred.Task: Task:attempt_local1885415322_0001_r_000000_0 is done. And is in the process of committing
    2018-07-07 06:08:25,281 INFO mapred.LocalJobRunner: 1 / 1 copied.
    2018-07-07 06:08:25,282 INFO mapred.Task: Task attempt_local1885415322_0001_r_000000_0 is allowed to commit now
    2018-07-07 06:08:25,283 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1885415322_0001_r_000000_0' to file:/home/tancan/output
    2018-07-07 06:08:25,284 INFO mapred.LocalJobRunner: reduce > reduce
    2018-07-07 06:08:25,284 INFO mapred.Task: Task 'attempt_local1885415322_0001_r_000000_0' done.
    2018-07-07 06:08:25,285 INFO mapred.Task: Final Counters for attempt_local1885415322_0001_r_000000_0: Counters: 24
        File System Counters
            FILE: Number of bytes read=316293
            FILE: Number of bytes written=784727
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
        Map-Reduce Framework
            Combine input records=0
            Combine output records=0
            Reduce input groups=4
            Reduce shuffle bytes=48
            Reduce input records=4
            Reduce output records=4
            Spilled Records=4
            Shuffled Maps =1
            Failed Shuffles=0
            Merged Map outputs=1
            GC time elapsed (ms)=0
            Total committed heap usage (bytes)=216006656
        Shuffle Errors
            BAD_ID=0
            CONNECTION=0
            IO_ERROR=0
            WRONG_LENGTH=0
            WRONG_MAP=0
            WRONG_REDUCE=0
        File Output Format Counters 
            Bytes Written=38
    2018-07-07 06:08:25,286 INFO mapred.LocalJobRunner: Finishing task: attempt_local1885415322_0001_r_000000_0
    2018-07-07 06:08:25,286 INFO mapred.LocalJobRunner: reduce task executor complete.
    2018-07-07 06:08:25,861 INFO mapreduce.Job: Job job_local1885415322_0001 running in uber mode : false
    2018-07-07 06:08:25,864 INFO mapreduce.Job:  map 100% reduce 100%
    2018-07-07 06:08:25,866 INFO mapreduce.Job: Job job_local1885415322_0001 completed successfully
    2018-07-07 06:08:25,883 INFO mapreduce.Job: Counters: 30
        File System Counters
            FILE: Number of bytes read=632458
            FILE: Number of bytes written=1569368
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
        Map-Reduce Framework
            Map input records=1
            Map output records=4
            Map output bytes=34
            Map output materialized bytes=48
            Input split bytes=97
            Combine input records=4
            Combine output records=4
            Reduce input groups=4
            Reduce shuffle bytes=48
            Reduce input records=4
            Reduce output records=4
            Spilled Records=8
            Shuffled Maps =1
            Failed Shuffles=0
            Merged Map outputs=1
            GC time elapsed (ms)=0
            Total committed heap usage (bytes)=432013312
        Shuffle Errors
            BAD_ID=0
            CONNECTION=0
            IO_ERROR=0
            WRONG_LENGTH=0
            WRONG_MAP=0
            WRONG_REDUCE=0
        File Input Format Counters 
            Bytes Read=18
        File Output Format Counters 
            Bytes Written=38
    

    查看结果:

    tancan@ubuntu:~$ cat output/*
    This    1
    file    1
    is  1
    test    1
    tancan@ubuntu:~$ 

    四、安装过程遇到的问题

    4.1 在64位系统上安装后,运行报错

    tancan@ubuntu:~$ hadoop fs -ls /
    Java HotSpot(TM) Server VM warning: You have loaded library /usr/soft/hadoop-3.0.3/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
    It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
    2018-06-28 06:52:31,341 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    ls: Call From ubuntu/127.0.1.1 to master:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

    原因:
    因为官网提供的版本本地库是32位的,在64位主机环境下无法执行。需要下载hadoop源码进行编译,编译成功后,找到native下的文件拷贝到${HADOOP_HOME}/lib/native目录下即可。

    五、附件

    我使用的native文件:
    https://download.csdn.net/download/goodmentc/10528791

    展开全文
  • Ubuntu16.04 下安装单机版 hadoop

    千次阅读 2017-12-21 12:04:15
    Ubuntu16.04 下安装单机版 hadoop

    环境 ubuntu16.04 + jdk8 + hadoop2.7.5

    Jdk安装参考链接:http://www.linuxidc.com/Linux/2016-04/130438.htm

    Hadoop安装参考链接:https://www.cnblogs.com/dennyzhangtm/p/7351996.html


    首先安装jdk,配置好JAVA_HOME环境之后再安装Hadoop

    JDK 和 Hadoop 安装包都要先下载之后解压安装

    展开全文
  • 目录 前言 ...本文安装Hadoop 及 Java 环境基于林子雨老师的《大数据技术原理与应用(第3)》中所要求,其中Java 版本为1.8.0_301,Hadoop 版本为3.3.1,其他版本的安装请参考其他博客。 ..

    目录

    前言

    一、创建Hadoop用户

    二、更新apt和安装Vim编辑器

    三、安装SSH和配置SSH无密码登录

    四、安装Java环境

    1. 安装JDK

    2. 配置JDK环境

    3. 检验安装 

    五、安装单机Hadoop

    1. 下载安装Hadoop

    2. 运行示例

    总结




    前言

    本文安装的 Hadoop 及 Java 环境基于林子雨老师的《大数据技术原理与应用(第3版)》中所要求,其中 Java 版本为1.8.0_301,Hadoop 版本为3.2.2,其他版本的安装请参考其他博客。

    Hadoop 单机安装基本配置主要包括以下几个步骤:

    • 创建 Hadoop 用户
    • 更新 apt 和安装 Vim 编辑器
    • 安装 SSH 和配置 SSH 无密码登录
    • 安装 Java 环境
    • 安装单机 Hadoop

    这里我的操作系统环境是 Ubuntu20.04,此安装方法同样适用于低版本。


    一、创建Hadoop用户

    创建用户命令如下:

     sudo useradd -m hadoop -s /bin/bash

     接着为 hadoop 用户设置密码,建议三位数,不用太长也要便于记忆:

    sudo passwd hadoop

    然后为 hadoop 用户增加管理员权限:

    sudo adduser hadoop sudo

    切换用户为hadoop登录!





    二、更新apt和安装Vim编辑器

    首先更新 apt:

    sudo apt-get update

    接着安装 Vim 编辑器:

    sudo apt-get install vim 



    若电脑已安装则可跳过此步骤。


    三、安装SSH和配置SSH无密码登录

    Ubuntu 操作系统下默认已安装了 SSH 客户端,因此这里我们只需安装 SSH 服务端:

    sudo apt-get install openssh-server

    安装后,可使用以下命令登录本机:

    ssh localhost

    输入 yes 与用户密码,就可以登录到本机,详细输出如下:

    zq@fzqs-computer [11时22分50秒] [/home/hadoop/Downloads] 
    -> %  
    ssh localhost
    The authenticity of host 'localhost (127.0.0.1)' can't be established.
    ECDSA key fingerprint is SHA256:YMFv60J4eT7***c3SA8sfuXU.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
    zq@localhost's password: 
    Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-36-generic x86_64)

     * Documentation:  https://help.ubuntu.com
     * Management:     https://landscape.canonical.com
     * Support:        https://ubuntu.com/advantage

    0 updates can be applied immediately.

    Your Hardware Enablement Stack (HWE) is supported until April 2025.

    The programs included with the Ubuntu system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
    applicable law.

    接着我们退出 SSH 登录,

    exit

    配置无密码登录:

    cd ~/.ssh/
    ssh-keygen -t rsa

    注意这里第二步要你输入文件名时不用输入,直接一路 Enter 选择默认值就好了!

    cat ./id_rsa.pub >> ./authorized_keys

    此时再用 ssh localhost 命令无需密码即可登录了。


    四、安装Java环境

    1. 安装JDK

    对于 Hadoop3.1.3 及以上版本而言,需要使用 JDK1.8 或者更新的版本,这里我们使用的 JDK 版本为1.8.0_301,安装包可以从 Oracle 官网下载:Java Downloads | Oracle,Java官网安装麻烦

    也可从我的百度网盘下载:jdk-8u301-linux-x64.tar.gz

    提取码:ul5u

    接着在 /usr/lib 目录下创建 jvm 文件夹来保存 JDK 文件:

    cd /usr/lib
    sudo mkdir jvm

    解压缩之前的 JDK 文件到上述目录中:

    cd ~/Downloads
    sudo tar -zxvf ./jdk-8u301-linux-x64.tar.gz -C /usr/lib/jvm

    2. 配置JDK环境

    使用 gedit 编辑器编辑环境变量:

    sudo gedit ~/.bashrc

    在文件末尾处添加以下几行内容:

    export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_301
    export JRE_HOME=${JAVA_HOME}/jre
    export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
    export PATH=${JAVA_HOME}/bin:$PATH

    保存并退出,接着使我们刚加入的环境变量生效:

    source ~/.bashrc

    3. 检验安装 

    输入以下命令:

    java -version

    若出现如下输出则说明安装成功:

    hadoop@fzqs-computer:~$ java -version

    java version "1.8.0_301"
    Java(TM) SE Runtime Environment (build 1.8.0_301-b09)
    Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)


    五、安装单机Hadoop

    1. 下载安装Hadoop

    下载地址:Apache Hadoop,这里官网下载较快,但注意不要下载 src 源码包!

    这里我安装的 Hadoop 版本为3.2.2,下载好后,执行以下命令安装:

    sudo tar -zxf ~/Downloads/hadoop-3.2.2.tar.gz -C /usr/local

    修改目录名称:

    cd /usr/local
    sudo mv  ./hadoop-3.2.2/ ./hadoop

    赋予可执行权限:

    sudo chown -R hadoop ./hadoop

      进入 hadoop 文件夹,查看安装的 Hadoop 版本信息:

    cd ./hadoop
    ./bin/hadoop version

    hadoop@fzqs-computer:/usr/local/hadoop$ ./bin/hadoop version

    Hadoop 3.2.2
    Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
    Compiled by ubuntu on 2021-06-15T05:13Z
    Compiled with protoc 3.7.1
    From source with checksum 88a4ddb2299aca054416d6b7f81ca55
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.2.jar

    若出现如上输出,则说明 Hadoop 安装成功。

    2. 运行示例

    首先在 Hadoop 安装目录下新建 input 子目录:

    cd /usr/local/hadoop
    sudo mkdir input

     复制 “/usr/local/hadoop/etc/hadoop” 中的配置文件到 input 目录下:

    sudo cp ./etc/hadoop/*.xml ./input

    切换 root 用户 :

    su

     执行以下代码运行 Grep 示例:

    ./bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar grep ./input ./output 'dfs[a-z.]+'

    执行完后,输入以下命令查看输出结果:

    cat ./output/*

    hadoop@fzqs-computer:/usr/local/hadoop$ cat ./output/*
    1    dfsadmin


    总结

    展开全文
  • Ubuntu18.04/64-bit Hadoop-2.7 JAVA JDK1.13.0.1 安装JDK 从这里下载对应版本的JDK,选择Linux版本,记住自己下载的版本号,后面配置时候需要用到。 新建/usr/java目录,切换到所下载的jdk-13.0.1_linux-x...

    准备

    • Ubuntu18.04/64-bit
    • Hadoop-2.7
    • JAVA JDK1.13.0.1

    安装JDK

    • 这里下载对应版本的JDK,选择Linux版本,记住自己下载的版本号,后面配置时候需要用到。
    • 新建/usr/java目录,切换到所下载的jdk-13.0.1_linux-x64_bin.tar.gz目录。将文件解压到/usr/java目录下。
      以下所有操作都以root用户登录
    sudo tar -xzvf jdk-13.0.1_linux-x64_bin.tar.gz /usr/java/
    
    • 配置环境变量
      打开~/.bashrc文件,配置java环境变量
    vim ~/.bashrc
    

    插入如下内容:

    export JAVA_HOME=/usr/java/jdk-13.0.1
    export JRE_HOME=${JAVA_HOME}/jre
    export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
    export PATH=${JAVA_HOME}/bin:$PATH
    

    使环境变量生效

    source ~/.bashrc
    

    验证java环境是否配置成功

    java -version
    #输出如下内容,说明安装成功
    java version "13.0.1" 2019-10-15
    Java(TM) SE Runtime Environment (build 13.0.1+9)
    Java HotSpot(TM) 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)
    

    安装ssh-server并实现免密码登录

    • 安装ssh-server
    sudo apt-get install openssh-server
    
    • 启动ssh
    sudo /etc/init.d/ssh start
    
    • 查看ssh服务是否启动,如果显示相关ssh字样则表示成功
    ps -ef|grep ssh
    #输出如下字样,说明启动成功
    gbh       2011  1874  0 09:25 ?        00:00:00 /usr/bin/ssh-agent /usr/bin/im-launch env GNOME_SHELL_SESSION_MODE=ubuntu gnome-session --session=ubuntu
    root      5362     1  0 10:46 ?        00:00:00 /usr/sbin/sshd -D
    gbh       6447  3514  0 10:47 pts/1    00:00:00 grep --color=auto ssh
    
    • 设置免密码登录(要用root账户登录,不然登录时候仍然要密码)
      使用如下命令,一直回车,直到生成了rsa
    ssh-keygen -t rsa
    #会输出如下信息
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    

    导入authorized_keys

    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    

    测试是否免密码登录localhost

    ssh localhost
    
    • 关闭防火墙
    ufw disable
    

    安装Hadoop单击模式和伪分布模式

    • 下载并解压hadoop-2.7.4.tar.gz
    wget hhttp://archive.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
    sudo tar zxvf hadoop-2.7.4.tar.gz -C /usr/local
    
    • 切换到/usr/local下,将hadoop-2.7.4重命名为hadoop,并给/usr/local/hadoop设置访问权限。(-R为递归的给目录权限,必须!
    cd /usr/local
    sudo mv hadoop-2.7.4 hadoop 
    sudo chmod 777 -R /usr/local/hadoop
    
    • 配置.bashsc文件
    vim ~/.bashrc
    #在文件末尾追加下面内容,然后保存。(注意路径)
    #HADOOP VARIABLES START 
    export HADOOP_HOME=/usr/local/hadoop
    export HADOOP_INSTALL=$HADOOP_HOME
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
    export HADOOP_CONF_DIR=$HADOOP_HOME
    export HADOOP_PREFIX=$HADOOP_HOME
    export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
    export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
    export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
    #HADOOP VARIABLES END
    
    • 执行下面命令,使添加的环境变量生效
    source ~/.bashrc
    
    • Hadoop配置(伪分布式搭建)
      配置hadoop-env.sh
    sudo vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
    #在末尾添加如下内容:(注意jdk文件名)
    # The java implementation to use. 
    export JAVA_HOME=/usr/java/jdk-13.0.1
    export HADOOP=/usr/local/hadoop
    export PATH=$PATH:/usr/local/hadoop/bin
    

    配置yarn-env.sh

    sudo vim /usr/local/hadoop/etc/hadoop/yarn-env.sh
    在末尾添加如下内容:(注意jdk文件名)
    # export JAVA_HOME=/usr/java/jdk-13.0.1
    JAVA_HOME=/usr/java/jdk-13.0.1
    

    配置core-site.xml,(这里的tmp.dir不用改,是缓存目录)

    sudo vim /usr/local/hadoop/etc/hadoop/core-site.xml
    #插入如下内容
    #注意:将core-site.xml中原来的有的<configuration> < /configuration >一定要删除掉,不然后面格式化的时候会出错。即.xml文件中只有一个<configuration> < /configuration >对就可以。
    <configuration>
            <property>
                 <name>hadoop.tmp.dir</name>
                 <value>file:/usr/local/hadoop/tmp</value>
                 <description>Abase for other temporary directories.</description>
            </property>
            <property>
                 <name>fs.defaultFS</name>
                 <value>hdfs://localhost:9000</value>
            </property>
    </configuration>
    

    同样修改配置文件 hdfs-site.xml

    sudo vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
    #插入下列内容:(删除原有的<configuration>)
    <configuration>
            <property>
                 <name>dfs.replication</name>
                 <value>1</value>
            </property>
            <property>
                 <name>dfs.namenode.name.dir</name>
                 <value>file:/usr/local/hadoop/tmp/dfs/name</value>
            </property>
            <property>
                 <name>dfs.datanode.data.dir</name>
                 <value>file:/usr/local/hadoop/tmp/dfs/data</value>
            </property>
    </configuration>
    
    • Hadoop配置文件说明
      Hadoop 的运行方式是由配置文件决定的(运行 Hadoop 时会读取配置文件),因此如果需要从伪分布式模式切换回非分布式模式,需要删除 core-site.xml 中的配置项。此外,伪分布式虽然只需要配置 fs.defaultFS 和 dfs.replication 就可以运行,不过若没有配置 hadoop.tmp.dir 参数,则默认使用的临时目录为 /tmp/hadoo-hadoop,而这个目录在重启时有可能被系统清理掉,导致必须重新执行 format 才行。同时也指定 dfs.namenode.name.dir 和 dfs.datanode.data.dir,否则在接下来的步骤中可能会出错。

    配置yarn-site.xml

    sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml 
    #插入下列内容
    <configuration> 
    <!-- Site specific YARN configuration properties -->
        <property> 
            <name>yarn.nodemanager.aux-services</name> 
            <value>mapreduce_shuffle</value> 
        </property> 
        <property> 
            <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
            <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
        </property> 
        <property> 
            <name>yarn.resourcemanager.address</name> 
            <value>127.0.0.1:8032</value> 
        </property> 
        <property> 
            <name>yarn.resourcemanager.scheduler.address</name> 
            <value>127.0.0.1:8030</value> 
        </property> 
        <property> 
            <name>yarn.resourcemanager.resource-tracker.address</name> 
            <value>127.0.0.1:8031</value> 
        </property> 
    </configuration>
    
    • 关机重启系统
      note: 以上三个文件都必须删除原来的对,在复制进去,否则会在后面格式化时发生如下错误 !!!!!
      在这里插入图片描述

    验证hadoop是否安装并配置成功

    • 验证Hadoop单击模式
    hadoop version
    #若输出如下内容说明配置成功
    Hadoop 2.7.4
    Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r cd915e1e8d9d0131462a0b7301586c175728a282
    Compiled by kshvachk on 2017-08-01T00:29Z
    Compiled with protoc 2.5.0
    From source with checksum 50b0468318b4ce9bd24dc467b7ce1148
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.4.jar
    
    • 启动HDFS为分布式模式
      格式化namenode
    hdfs namenode -format
    #输出
    ...
    19/12/24 14:26:41 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1645567278-127.0.1.1-1577168801272
    19/12/24 14:26:41 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
    19/12/24 14:26:41 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
    19/12/24 14:26:41 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
    19/12/24 14:26:41 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    19/12/24 14:26:41 INFO util.ExitUtil: Exiting with status 0
    19/12/24 14:26:41 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at gbh/127.0.1.1
    ************************************************************/
    

    note:有successfully formatted代表格式化成功。Existing with 1代表有错误。

    启动hdfs

    start-all.sh
    

    显示进程

    jps 
    #输出
    root@gbh:/usr/local/hadoop# jps 
    5568 ResourceManager
    5170 DataNode
    6019 Jps
    5399 SecondaryNameNode
    4985 NameNode
    5727 NodeManager
    

    note:有7个进程代表正确
    在浏览器中输入http://localhost:50070/,出现如下页面

    在这里插入图片描述输入 http://localhost:8088/, 出现如下页面
    在这里插入图片描述

    Hadop实例:词频统计

    • 启动HDFS
    start-all.sh
    

    查看HDFS下面包含的文件目录

    hadoop dfs -ls /
    #第一次运行hdfs什么都没有
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    ...
    

    在HDFS中创建一个文件目录input,将/usr/local/hadoop/README.txt上传至input中,此时再用ls查看就发现多了个input目录

    hdfs dfs -mkdir /input
    hadoop fs -put /usr/local/hadoop/README.txt /input
    hadoop dfs -ls /input
    #output
    Found 1 items
    -rw-r--r--   1 root supergroup       1366 2019-12-24 14:52 /input/README.txt
    

    执行一下命令运行wordcount 并将结果输出到output中

    hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /input /output
    #输出如下
    ...
    19/12/24 14:55:25 INFO mapreduce.Job:  map 100% reduce 100%
    19/12/24 14:55:25 INFO mapreduce.Job: Job job_local609248869_0001 completed successfully
    19/12/24 14:55:25 INFO mapreduce.Job: Counters: 35
    	File System Counters
    		FILE: Number of bytes read=599278
    		FILE: Number of bytes written=1185318
    		FILE: Number of read operations=0
    		FILE: Number of large read operations=0
    		FILE: Number of write operations=0
    		HDFS: Number of bytes read=2732
    		HDFS: Number of bytes written=1306
    		HDFS: Number of read operations=13
    		HDFS: Number of large read operations=0
    		HDFS: Number of write operations=4
    ...
    

    有以上输出,说明统计成功。
    执行成功后output 目录底下会生成两个文件 _SUCCESS 成功标志的文件,里面没有内容。 一个是 part-r-00000 ,通过以下命令查看执行的结果。

    hadoop fs -cat /output/part-r-00000
    #下面是统计的词频结果
    (BIS),	1
    (ECCN)	1
    (TSU)	1
    (see	1
    5D002.C.1,	1
    740.13)	1
    <http://www.wassenaar.org/>	1
    Administration	1
    Apache	1
    BEFORE	1
    BIS	1
    Bureau	1
    Commerce,	1
    Commodity	1
    Control	1
    Core	1
    Department	1
    ENC	1
    Exception	1
    Export	2
    For	1
    Foundation	1
    Government	1
    Hadoop	1
    Hadoop,	1
    Industry	1
    ...
    

    下面附一些HDFS常用命令

    • hadoop fs -mkdir /tmp/input 在HDFS上新建文件夹
    • hadoop fs -put input1.txt /tmp/input 把本地文件input1.txt传到HDFS的/tmp/input目录下
    • hadoop fs -get input1.txt /tmp/input/input1.txt 把HDFS文件拉到本地
    • hadoop fs -ls /tmp/output 列出HDFS的某目录
    • hadoop fs -cat /tmp/ouput/output1.txt 查看HDFS上的文件
    • hadoop fs -rmr /home/less/hadoop/tmp/output 删除HDFS上的目录
    • hadoop dfsadmin -report 查看HDFS状态,比如有哪些datanode,每个datanode的情况
    • hadoop dfsadmin -safemode leave 离开安全模式
    • hadoop dfsadmin -safemode enter 进入安全模式
    展开全文
  • 由于我是要安装Hadoop和Hbase,并且注意到这两者之间会有版本之间的兼容性问题,之前也是走了弯路,在此记录一下: 1、hbase 与java的兼容性是不一样的,在Hadoop的文档Hbase的文档中对jdk的版本进行了说明 2、...
  • Ubuntu安装Hadoop-3.3.1

    2021-09-23 21:08:30
    Hadoop包括3种安装模式 1.单机模式 2.伪分布模式 3.分布式模式 一、下载Hadoop安装文件 文件名称: hadoop-3.3.1.tar.gz 网页地址: ...
  • ubuntu安装配置hadoop jkd版本:1.8.0_191 hadoop版本:2.8.5 点此下载hadoop:https://hadoop.apache.org/releases.html 1、解压并移动到指定文件夹 tar -zxvf /home/hadoop-2.8.5.tar.gz #解压 mv /home/...
  • 文章目录Ubuntu系统安装Hadoop3.1.3并进行单机/伪分布式配置前言详细流程创建Hadoop用户安装Java配置SSH免密登陆安装Hadoop3.1.3Hadoop单机配置Hadoop单机运行实例Hadoop伪分布式配置Hadoop伪分布式运行实例Web访问...
  • (2)已安装jdk,具体安装教程请看我以前的文章。 下载Hadoop Hadoop下载地址,推荐下载2.7.7版本,因为它比较稳定。 使用wget下载: root@instance-cqxyyrb2:/# mkdir Hadoop root@instance-cqxyyrb2:/# cd ...
  • Ubuntu16.04安装Hadoop单机和伪分布式环境超详细

    万次阅读 多人点赞 2017-12-10 17:16:54
    介绍:Hadoop是Apache开源组织的一个分布式计算框架,可以在大量廉价的硬件设备组成的集群上运行应用程序,为应用程序提供了一组稳定可靠的接口,旨在构建一个具有高可靠性和良好扩展性的分布式系统。...安装
  • (前提:先安装java环境) 下载地址:http://hadoop.apache.org/releases.html (注意是binary文件,source那个是源码) (2)解压tar.gz    (3)配置hadoop 1.修改/usr/hadoop/hadoop-2.7.3/etc/hadoop/...
  • 1.配置jdk 见前文 2. 下载解压hadoop 这里以2.7.7版本为例,将...设置hadoop目录下的env文件,hadoop/etc/hadoop-env.sh文件,注意这里的etc是在hadoop目录下的etc,大家注意命令执行的路径,在里面定义jdk目...
  • ubuntu安装hadoop

    2017-12-03 15:13:12
    安装jdk 然后再安装hadoop 然后再改一下几个配置文件 最后启动 然后查看浏览器检验一下
  • 使用Docker搭建Hadoop单机版

    千次阅读 2019-07-21 01:46:03
    其实,直接安装也很简单,官方说明文档:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html。使用Docker安装只是为了不想污染我的云服务器... 首先确保你的机器关了...
  • 文章目录1. 安装2. 配置步骤一:配置JAVA_HOME步骤二:配置端口与文件格式...最新版安装网站:https://hadoop.apache.org/releases.html 历史版本:http://archive.apache.org/dist/hadoop/core/ 清华源:https://mir
  • 所以,本文会搭建一个伪分布式版hadoop集群,用于CI测试非常方便。 准备 准备的文件结构如下: ├── dockerfile ├── hadoop │ ├── core-site.xml │ ├── hadoop-2.7.7.tar.gz │ ├── hdfs-site.xml ...
  • 单机模式:安装简单,几乎不用做任何配置,但仅限于调试用途; 伪分布模式:在单节点上同时启动 NameNode、DataNode、JobTracker、TaskTracker、Secondary Namenode 等 5 个进程,模拟分布式运行的各个节点; 完全...
  • UbuntuHadoop安装教程

    千次阅读 2020-09-21 01:37:31
    UbuntuHadoop安装教程,Hadoop单机配置和伪分布式安装 教程 本教程采用的是Ubuntu18.04.4+hadoop2.7.7+jdk1.8 创建Hadoop用户 打开命令行终端(ctrl+alt+t),输入如下命令创建hadoop用户:sudo useradd -m ...
  • ubuntu18.04安装Hadoop

    万次阅读 多人点赞 2018-08-20 15:35:21
    安装部分参考https://blog.csdn.net/qjk19940101/article/details/70544197,这里只是改动了部分和部分填充并进一步说明...安装SSH server: sudo apt-get install openssh-server 配置SSH: ssh localhost e...
  • Ubuntu安装Hadoop单机

    千次阅读 2018-11-22 23:37:34
    Ubuntu安装Hadoop单机)确保已安装Java安装Hadoop运行Hadoop(伪集群)执行MapReduce 任务,使用hadoop预置的示例程序进行演示关闭hdfs 确保已安装Java Hadoop是用Java开发的,必须先安装Java环境,Oracle和...
  • 一.安装须知 Hadoop版本问题:Hadoop 有两个主要版本,Hadoop 1.x.y 和 Hadoop 2.x.y 系列,比较老的教材上用的可能是 0.20 这样的版本。Hadoop 2.x 版本在不断更新,...自学建议安装最新版本,目前是hadoop2.7.2 单机
  • Ubuntu安装Hadoop记录本次安装过程,本次安装使用32位Ubuntu12.04,JDK版本为1.8.0_11。1) 安装linux虚拟机本教程使用VMware WorkStationPro,新建Ubuntu虚拟机。在使用之前最好换源。2)新建用户组 创建hadoop用户...
  • 虚拟机安装Hadoop单机伪分布式下载Hadoop下载JDK配置JDKssh免密登陆配置Hadoop总结 下载Hadoop 虚拟机安装ubuntu16.04, 配置虚拟机和主机之间可以复制文件,参考如下网址 ...wfr=spider&...
  • 完全是个小白的我根据林子雨老师的教程和视频,进行hadoop的环境配置和安装,由于对ubuntu的了解很少,只能尽量减少错误,还希望大佬能多多指正! 首先是linux系统的安装,这里我选择ubuntu系统(严格来说,Linux并...
  • 文章目录1 系统版本1.1 解压安装包2 配置ssh免密登录2.1 安装openssh2.2 登陆本机3 配置环境变量3.1 编辑环境变量3.2 插入如下内容3.3 刷新环境变量4 验证Java5 配置Hadoop5.1 添加权限5.2 修改hadoop-env.sh文件5.3...
  • ubuntu20.04server下安装hadoop2.8.5

    千次阅读 2020-06-07 17:32:49
    参考UbuntuHadoop安装(全命令行安装环境 项目 名称 版本 电脑硬件 Huwei Matebook X Pro i7-8550U 16G 512G 操作系统 Windows 10 家庭中文 虚拟机 VMware® Workstation 15 Pro 15.0.0 ...
  • 我今天来说一下Oracle VM VirtualBox下的Ubuntu版本安装hadoop2.7.1的伪分布式。 问题描述:搭建hadoop伪分布式 首先,我简单说一下,目前hadoop环境的搭建大致可划分为四种,单机模式、伪分布式、完全分布式、高...
  • 基于ubuntu安装Hadoop,并运行word count

    千次阅读 2018-06-28 12:11:51
     实验目的以及实验环境 在Linux或者Windows下安装Hadoop并运行其中任意实例,本次试验简单运行一个wordcount 实例,选择的安装环境是在VMware Fusion 8.5.3虚拟机上,使用ubuntu安装。以下是两个版本的截图。 ...
  • 使用的虚拟机软件是:VMWare Workstation Pro 14 虚拟机系统:Ubuntu 18.04 ... 目录 ... 安装ssh ...安装vim ...在虚拟机中安装JDK并...安装Hadoop Hadoop伪分布模式修改 启动HDFS伪分布式模式 创建新账户 参考文...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 3,351
精华内容 1,340
关键字:

ubuntu安装hadoop单机版