32位 hadoop 编译native

2016-06-28 23:48:06 tterminator 阅读数 6544
  • Spark技术在银行业应用-1

    AI时代下金融数据人才培养建设与转型提升,银行业数据仓库向大数据平台转型实战,Spark技术在银行业应用,Apache Griffin原理及在互联网金融数据治理上的应用。

    482人学习 CSDN讲师
    免费试看
  1. 本文中的所有路径请根据个人情况修改。
  2. 编译好的native library见个人资源:【http://download.csdn.net/detail/tterminator/9565597

一、为什么要编译native library

mac单机模式安装Hadoop后启动,报错:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable。

有兴趣的可以自己编译下,因为网上很多的native library资源和方法都是不可用的,自己在本地编译也是官网推荐的。

二、问题原因

在官网上有说明:
The pre-built 32-bit i386-Linux native hadoop library is available as part of the hadoop distribution and is located in the lib/native directory. You can download the hadoop distribution from Hadoop Common Releases.

在官网下载的Hadoop版本中自带的native library 只针对32位Linux

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

官网已明确说明预编译的native library 不适用于mac os x。所以这也是必须要在mac下编译native library的充要理由。

详细说明请参见http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html

三、编译环境说明

  1. mac os x:10.10.4
  2. jdk:java version “1.7.0_80”
  3. hadoop:2.6.0-src

四、编译前期准备

编译开始前需要在mac下安装以下软件:
1.安装brew

类似Ubuntu中的apt软件包管理工具,用于安装缺少的软件包,这里用于安装cmake工具:brew install cmake。

2.安装cmake

没有版本要求。

3.安装protoc

版本必须是2.5.0,否则编译失败,不能用brew install protobuf的方式安装,因为该方式安装的版本不一定是2.5.0。

protocbuf 2.5.0的源码详见个人资源【http://download.csdn.net/detail/tterminator/9562400】,需要自己手动编译,其实也比较简单,步骤如下:
(1)设置编译目录:
./configure --prefix=/User/King-pan/software/tools/protobuf
其中/User/King-pan/software/tools/protobuf 为自己设定的编译安装目录。
(2)安装:
make
make install
(3)配置环境变量:
sudo vi .bash_profile
(4)添加配置文件:
export PROTOBUF=/Users/King-pan/software/tools/protobuf
export PATH=$PROTOBUF/bin:$PATH
(5)测试:
protoc --version

4.安装maven

本人编译时使用的版本:Apache Maven 3.3.3。

五、最后准备

在编译过程中会出现以下报错:
Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

解决办法:
在JAVA_HOME,也即/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home路径下新建目录Class,并执行创建软符号链接命令(推荐使用绝对路径):
sudo ln -s $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar

六、开始编译

  1. 进入下载的2.6.0源码根目录下,执行以下命令:
    mvn package -Pdist,native -DskipTests -Dtar

  2. 等待之后会出现以下编译结果:
    [INFO] Reactor Summary:
    [INFO]
    [INFO] Apache Hadoop Main ................................. SUCCESS [ 1.206 s]
    [INFO] Apache Hadoop Project POM .......................... SUCCESS [06:57 min]
    [INFO] Apache Hadoop Annotations .......................... SUCCESS [03:22 min]
    [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.272 s]
    [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [02:02 min]
    [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [02:46 min]
    [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [19:36 min]
    [INFO] Apache Hadoop Auth ................................. SUCCESS [07:47 min]
    [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [01:21 min]
    [INFO] Apache Hadoop Common ............................... SUCCESS [12:20 min]
    [INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.994 s]
    [INFO] Apache Hadoop KMS .................................. SUCCESS [ 30.076 s]
    [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.044 s]
    [INFO] Apache Hadoop HDFS ................................. SUCCESS [07:09 min]
    [INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:46 min]
    [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [03:25 min]
    [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.440 s]
    [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.027 s]
    [INFO] hadoop-yarn ........................................ SUCCESS [ 0.031 s]
    [INFO] hadoop-yarn-api .................................... SUCCESS [01:08 min]
    [INFO] hadoop-yarn-common ................................. SUCCESS [01:32 min]
    [INFO] hadoop-yarn-server ................................. SUCCESS [ 0.027 s]
    [INFO] hadoop-yarn-server-common .......................... SUCCESS [01:21 min]
    [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [01:49 min]
    [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.149 s]
    [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 4.729 s]
    [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.844 s]
    [INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 4.096 s]
    [INFO] hadoop-yarn-client ................................. SUCCESS [ 5.619 s]
    [INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.024 s]
    [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 1.940 s]
    [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.541 s]
    [INFO] hadoop-yarn-site ................................... SUCCESS [ 0.031 s]
    [INFO] hadoop-yarn-registry ............................... SUCCESS [ 3.909 s]
    [INFO] hadoop-yarn-project ................................ SUCCESS [ 3.328 s]
    [INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.040 s]
    [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 17.288 s]
    [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.444 s]
    [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 2.979 s]
    [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 7.477 s]
    [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 5.705 s]
    [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:24 min]
    [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.265 s]
    [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.280 s]
    [INFO] hadoop-mapreduce ................................... SUCCESS [ 3.765 s]
    [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 16.991 s]
    [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 31.676 s]
    [INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.504 s]
    [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.793 s]
    [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.410 s]
    [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.948 s]
    [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.599 s]
    [INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.169 s]
    [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 6.456 s]
    [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 4.146 s]
    [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [08:32 min]
    [INFO] Apache Hadoop Client ............................... SUCCESS [ 6.769 s]
    [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.112 s]
    [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.510 s]
    [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.581 s]
    [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.021 s]
    [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 26.965 s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 01:28 h
    [INFO] Finished at: 2016-06-28T01:40:35+08:00
    [INFO] Final Memory: 193M/874M
    [INFO] ------------------------------------------------------------------------

七、将编译出的native library复制到下载的二进制版本的Hadoop 2.6.0相应目录中

  1. 编译出的native library库的位置为
    hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0/lib/native
  2. 拷贝到二进制版本的Hadoop 2.6.0的目录
    hadoop-2.6.0/lib/native

八、修改/etc/hadoop/hadoop-env.sh配置

export HADOOP_OPTS=”$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=/hadoop-2.6.0/lib/native”

九、重新启动Hadoop

此时就不会出现本文开头处的那个警告了。

十、附录

  1. ssh localhost 报错:connection closed by ::1
    解决办法:
    查看mac os x 系统日志/var/log/system.log后发现有提示:
    (1)Jun 28 09:29:29 bj-m-203544a.local sshd[48173]: error: Could not load host key: /etc/ssh_host_rsa_key
    (2)Jun 28 09:29:29 bj-m-203544a.local sshd[48173]: error: Could not load host key: /etc/ssh_host_dsa_key
    根据提示把.ssh/id_dsaid_rsa拷贝到/etc目录下,并重命名为ssh_host_dsa_keyssh_host_rsa_key即可。

  2. ssh localhost 免密登录
    修改.ssh/authorized_keys文件权限为644
    (1)chmod 644 authorized_keys
    (2)修改/etc/ssh_config文件中属性PasswordAuthentication值为no【可能会影响其他需要密码的ssh登录,若有影响再把该值给为yes即可。】

  3. 单机模式安装的Hadoop配置打包,详见个人资源

  4. 其它
    (1)启动Hadoop:
    ./start-dfs.sh
    ./start-yarn.sh

    (2)关闭Hadoop
    ./stop-dfs.sh
    ./stop-yarn.sh

    (3)hadoop fs -mkdir /tmp
    (4)hadoop fs -copyFromLocal ~/word.txt /tmp
    (5)Hadoop管理页面
    http://localhost:50070
    (6)yarn管理界面
    http://localhost:8098/cluster【端口实在yarn-site.xml中设置的】
    (7)hadoop fs -rm /tmp/out/part-r-00000
    (8)hadoop fs -rmdir /tmp/out/
    备注:如果有Hadoop进程没有起来,可以查看Hadoop日志,里面会有详细的记录:
    hadoop/logs/
    -rw-r--r-- 1 junwei8 staff 0 Jun 28 10:16 SecurityAuth-junwei8.audit
    -rw-r--r-- 1 junwei8 staff 65411 Jun 28 11:39 hadoop-junwei8-datanode-bj-m-203544a.local.log
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-datanode-bj-m-203544a.local.out
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-datanode-bj-m-203544a.local.out.1
    -rw-r--r-- 1 junwei8 staff 87710 Jun 28 11:40 hadoop-junwei8-namenode-bj-m-203544a.local.log
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-namenode-bj-m-203544a.local.out
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-namenode-bj-m-203544a.local.out.1
    -rw-r--r-- 1 junwei8 staff 64683 Jun 28 11:40 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.log
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 11:39 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.out
    -rw-r--r-- 1 junwei8 staff 511 Jun 28 10:16 hadoop-junwei8-secondarynamenode-bj-m-203544a.local.out.1
    drwxr-xr-x 2 junwei8 staff 68 Jun 28 11:40 userlogs
    -rw-r--r-- 1 junwei8 staff 77547 Jun 28 11:40 yarn-junwei8-nodemanager-bj-m-203544a.local.log
    -rw-r--r-- 1 junwei8 staff 494 Jun 28 11:40 yarn-junwei8-nodemanager-bj-m-203544a.local.out
    -rw-r--r-- 1 junwei8 staff 494 Jun 28 10:17 yarn-junwei8-nodemanager-bj-m-203544a.local.out.1
    -rw-r--r-- 1 junwei8 staff 251199 Jun 28 11:40 yarn-junwei8-resourcemanager-bj-m-203544a.local.log
    -rw-r--r-- 1 junwei8 staff 494 Jun 28 11:40 yarn-junwei8-resourcemanager-bj-m-203544a.local.out
    -rw-r--r-- 1 junwei8 staff 494 Jun 28 10:17 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.1
    -rw-r--r-- 1 junwei8 staff 494 Jun 27 21:18 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.2
    -rw-r--r-- 1 junwei8 staff 494 Jun 27 21:17 yarn-junwei8-resourcemanager-bj-m-203544a.local.out.3

十一、参考链接:

  1. http://www.rockyfeng.me/hadoop_native_library_mac.html
  2. http://my.oschina.net/KingPan/blog/283881?p=1
  3. http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
  4. http://leibnitz.iteye.com/blog/2149745
2019-04-09 16:03:18 qq_37806518 阅读数 418
  • Spark技术在银行业应用-1

    AI时代下金融数据人才培养建设与转型提升,银行业数据仓库向大数据平台转型实战,Spark技术在银行业应用,Apache Griffin原理及在互联网金融数据治理上的应用。

    482人学习 CSDN讲师
    免费试看

hadoop编译native包

hadoop源码包
http://mirrors.hust.edu.cn/apache/hadoop/common

protobuf(用于编译生成hadoop的native包,这个百度很多)

centos下:

yum -y install lzo-devel zlib-devel gcc autoconf automake libtool
安装g++:yum -y install gcc-c++
安装cmake:yum -y install cmake
安装openssl-devel:yum -y install openssl-devel

进入下载的protobuf文件夹下
./configure
make
make install

进入hadoop-2.x.x-src编译hadoop
mvn clean package -Pdist,native -DskipTests -Dtar

native包在
hadoop-2.x.x-src/hadoop-dist/target/hadoop-2.x.x/lib/native

在/etc/profile添加
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

2016-09-02 14:43:00 weixin_34417635 阅读数 50
  • Spark技术在银行业应用-1

    AI时代下金融数据人才培养建设与转型提升,银行业数据仓库向大数据平台转型实战,Spark技术在银行业应用,Apache Griffin原理及在互联网金融数据治理上的应用。

    482人学习 CSDN讲师
    免费试看

本文介绍了hadoop编译的详细流程,包括依赖的安装,以及支持native和snappy库。

snappy

安装snappy

snappy github 地址:[snappy]https://github.com/google/snappy/releases "snappy")选择1.1.4版本下载至/opt/tmp

cd /opt/tmp   
unzip snappy-1.1.4.zip   
cd sanppy-1.1.4
yum install -y gcc gcc-c++ autoconf automake libtool  cmake openssl-devel

./autogen.sh   
./configure   
make   
make install   
ls -lh /usr/local/lib |grep snappy   

-rw-r--r--. 1 root root 462K Sep 1 20:38 libsnappy.a
-rwxr-xr-x. 1 root root 934 Sep 1 20:38 libsnappy.la
lrwxrwxrwx. 1 root root 18 Sep 1 20:38 libsnappy.so -> libsnappy.so.1.3.0
lrwxrwxrwx. 1 root root 18 Sep 1 20:38 libsnappy.so.1 -> libsnappy.so.1.3.0
-rwxr-xr-x. 1 root root 223K Sep 1 20:38 libsnappy.so.1.3.0

简单测试snappy

cd /tmp
vim a.cpp   

#include "snappy.h"
#include <string>
#include <iostream>
int main() {
std::string s = "ddsandslkadnsldsan;lkdsanf;ladnskal;fsan;lkfsjk;lfsna;lfdskfsa";
std::string d;
snappy::Compress(s.data(), s.size(), &d);
std::cout<<d<<std::endl;
std::cout<<s.size()<<" "<<d.size()<<std::endl;
return 0;
}

libtool --mode=compile gcc -c a.cpp   
libtool --mode=link g++ -o test a.lo /usr/local/lib/libsnappy.la   
./test

protobuf

安装protobuf

protobuf github 地址:protobuf选择2.5的版本下载至/opt/tmp

cd /opt/tmp   
unzip protobuf-2.5.0.zip
cd protobuf-2.5.0

修改autogen.sh,注释掉如下两行

curl http://googletest.googlecode.com/files/gtest-1.5.0.tar.bz2 | tar jx
mv gtest-1.5.0 gtest

并修改为

wget https://codeload.github.com/google/googletest/zip/release-1.5.0
unzip release-1.5.0
mv googletest-release-1.5.0 gtest

执行命令进行编译安装,默认安装protoc到/usr/local/bin

./autogen.sh
./configure  
make   
make check   
make install 

hadoop2.6的编译过程hadoop-common要求必须是protobuf-2.5.0版本,其他版本会错误。

测试protobuf

protoc --version

libprotoc 2.5.0

findbugs

###安装findbugs 下载findbugs 解压缩并设置环境变量:

export FINDBUGS_HOME=/opt/tmp/findbugs-3.0.0   
export PATH=$PATH:$FINDBUGS_HOME/bin

###测试findbugs findbugs -version

JDK

安装JDK

JDK7 地址:JDK7 下载jdk解压至/usr/local,修改java目录名为/usr/local/jdk

JDK环境变量

修改/etc/profile最后添加如下:

export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

测试JDK

java -version

java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

maven

安装maven

apache maven下载地址:apache maven下载解压至/usr/local,修改maven目录名为/usr/local/maven

MAVEN环境变量

修改/etc/profile最后添加如下:

export MVN_HOME=/usr/local/maven
export PATH=$MVN_HOME/bin:$PATH

测试maven

mvn -v

Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T09:29:23-08:00) Maven home: /usr/local/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/local/jdk1.7.0_79/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.4.2.el6.x86_64", arch: "amd64", family: "unix"

hadoop

编译hadoop

apache hadoop下载地址:apache hadoop下载解压至/opt/tmp,这里选择2.6.4。使用mvn编译.

cd /opt/tmp
tar zxf hadoop-2.6.4-src.tar.gz
cd hadoop-2.6.4-src
mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy   

输出(只截取最后部分):

main:
 [exec] $ tar cf hadoop-2.6.4.tar hadoop-2.6.4
 [exec] $ gzip -f hadoop-2.6.4.tar
 [exec] 
 [exec] Hadoop dist tar available at: /home/hadoop/Downloads/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4.tar.gz
 [exec] 
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/hadoop/Downloads/hadoop-2.6.4-src/hadoop-dist/target/hadoop-dist-2.6.4-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.509 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.180 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.209 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.302 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.920 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.205 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.681 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.828 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.916 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:49 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.459 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 43.569 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:29 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 56.546 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.499 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.245 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.036 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.032 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:25 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 25.410 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.040 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  9.218 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 14.246 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.401 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.167 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 16.416 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.213 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  6.172 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.035 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.356 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.561 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.033 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.421 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  6.009 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.069 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 17.897 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 14.063 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.271 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.973 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  6.585 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  5.657 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.528 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.974 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  5.153 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.880 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.325 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.785 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  5.321 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.932 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.533 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.057 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.871 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.219 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.474 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  6.465 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.969 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.082 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.779 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.584 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.045 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 32.537 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:31 min
[INFO] Finished at: 2016-09-02T00:01:32-07:00
[INFO] Final Memory: 207M/549M
[INFO] ------------------------------------------------------------------------

hadoop可执行软件包

相关软件包生成在/opt/tmp/hadoop-2.6.4-src/hadoop-dist/target

  • 压缩包为:
    /opt/tmp/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.0-cdh5.5.4.tar.gz

  • native库(包含snappy):
    /opt/tmp/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.0-cdh5.5.4/lib/native

测试native

hadoop checknative

_16/09/02 15:03:35 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/09/02 15:03:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/beh/core/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /opt/beh/core/hadoop/lib/native/libsnappy.so.1
lz4: true revision:10301
bzip2: false
openssl: false Cannot load libcrypto.so (libcrypto.so: 无法打开共享对象文件: 没有那个文件或目录)! _

openssl: false Cannot load libcrypto.so (libcrypto.so: 无法打开共享对象文件: 没有那个文件或目录)! 异常解决办法是:
ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so

转载于:https://my.oschina.net/yulongblog/blog/741527

2016-06-08 14:14:32 qqpy789 阅读数 1771
  • Spark技术在银行业应用-1

    AI时代下金融数据人才培养建设与转型提升,银行业数据仓库向大数据平台转型实战,Spark技术在银行业应用,Apache Griffin原理及在互联网金融数据治理上的应用。

    482人学习 CSDN讲师
    免费试看

最近公司让我研究一下hadoop hbase native包

先附上参考地址 http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/NativeLibraries.html

Usage

It is fairly easy to use the native hadoop library:

  1. Review the components.
  2. Review the supported platforms.
  3. Either download a hadoop release, which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Whether you download or build, the name for the library is the same: libhadoop.so
  4. Install the compression codec development packages (>zlib-1.2, >gzip-1.2): + If you download the library, install one or more development packages - whichever compression codecs you want to use with your deployment. + If you build the library, it is mandatory to install both development packages.
  5. Check the runtime log files.

上图是官网对native包的作用描述

Build

The native hadoop library is written in ANSI C and is built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward to build the library on any platform with a standards-compliant C compiler and the GNU autotools-chain (see the supported platforms).

The packages you need to install on the target platform are:

  • C compiler (e.g. GNU C Compiler)
  • GNU Autools Chain: autoconf, automake, libtool
  • zlib-development package (stable version >= 1.2.0)

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

   $ mvn package -Pdist,native -Dskiptests -Dtar

You should see the newly-built library in:

   $ hadoop-dist/target/hadoop-2.5.2/lib/native

Please note the following:

  • It is mandatory to install both the zlib and gzip development packages on the target platform in order to build the native hadoop library; however, for deployment it is sufficient to install just one package if you wish to use only one codec.
  • It is necessary to have the correct 32/64 libraries for zlib, depending on the 32/64 bit jvm for the target platform, in order to build and deploy the native hadoop library.

上图是官网关于编译本地包的步骤

首先就是os了,支持

  • RHEL4/Fedora
  • Ubuntu
  • Gentoo

这几个系统做编译.然后就是环境的设置了,安装jdk1.7 安装maven 

还需要安装gcc autoconf,automake,libtool,zlib-devel不然会出现莫名其妙的错误.

之后就是执行命令

mvn package -Pdist,native -Dskiptests -Dtar
或者 
mvn package -Pdist,native -Dmaven.test.skip=true -Dtar
或者
mvn package -Pdist,native -DeskipTests -Dtar -Dmaven.test.failure.ignore=true
这两者区别是 后面一个不进行junit测试
预祝大家编译成功.

2018-12-12 19:14:18 anyuzun 阅读数 489
  • Spark技术在银行业应用-1

    AI时代下金融数据人才培养建设与转型提升,银行业数据仓库向大数据平台转型实战,Spark技术在银行业应用,Apache Griffin原理及在互联网金融数据治理上的应用。

    482人学习 CSDN讲师
    免费试看

文档说明
    该文档为完成编译后,根据印象编写,并未经过重新编译测试,大体步骤应该没错,如有错误,望指正。
    编译的最初目的,是为了获取合适的native包,并且检测手动编译的native包和自带的native包有何差别。
    编译的版本为apache-hadoop-2.7.6和hadoop-2.6.0-cdh5.5.0。

编译说明
    源码中会有一个 BUILDING.txt 文件,编译需要的依赖和命令等大部分信息都可以在里面找到。
    比如依赖:

        Requirements:
        * Unix System
        * JDK 1.7+
        * Maven 3.0 or later
        * Findbugs 1.3.9 (if running findbugs)
        * ProtocolBuffer 2.5.0
        * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
        * Zlib devel (if compiling native code)
        * openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
        * Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
        * Internet connection for first build (to fetch all Maven and Hadoop dependencies)

编译环境
    $ uname -a    # Linux e3base01 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
    $ java -version    # java version "1.7.0_04"
    $ mvn -v    # Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-15T01:29:23+08:00)
    # findbugs未安装。
    $ protoc --version    # libprotoc 2.5.0
    $ cmake -version    # cmake version 2.8.12.2
    $ yum list installed | grep zlib-devel    # zlib-devel.x86_64       1.2.3-29.el6    @base
    $ yum list installed | grep openssl-devel    # openssl-devel.x86_64    1.0.1e-57.el6   @base
    $ yum list installed | grep fuse    # fuse.x86_64             2.8.3-4.el6     @anaconda-CentOS-201311272149.x86_64/6.5    # 系统自带,并未刻意安装。
    ## 编译目录预留推荐最少5G空间,编译完后编译目录占用空间接近4G。

安装依赖
    jdk
        下载tar包,tar开。
        配置环境变量,如~/.bash_profile中添加:
            export JAVA_HOME=/path/to/jdk
            export PATH=$JAVA_HOME/bin:$PATH
        $ source ~/.bash_profile # 立即生效。
        验证 
            $ java -version
    maven
        下载tar包,tar开。
        配置环境变量,如~/.bash_profile中添加:
            export MAVEN_HOME=/path/to/maven
            export MAVEN_OPTS="-Xmx4g -Xms4g"
            export PATH=$MAVEN_HOME/bin:$PATH
        $ source ~/.bash_profile # 立即生效。
        验证 
            $ mvn -v
    protobuff
        下载tar包,tar开。
        # ./configure
        # make
        # make install
        验证 
            $ protoc --version
    cmake,zlib-devel,openssl-devel
        $ yum install cmake zlib-devel openssl-devel
        验证 
            $ yum list installed | egrep 'cmake|zlib-devel|openssl-devel'

编译命令
    ## 编译,并生成native。编译过程如snappy和openssl包不可用,则编译直接报错。    # 该项未测试。
    $ mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Drequire.openssl
    ## 编译,并生成native,会将系统的snappy和openssl等共享库添加至native目录。注意将 snappy.lib 和 openssl.lib 指向系统正确的lib目录。
    $ mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Drequire.openssl -Dopenssl.lib=/usr/lib64 -Dbundle.openssl

编译过程中无法下载依赖的问题
    编译过程会遇到报错,一般是显示download xxx,然后就失败,提示无法下载依赖等字样。
    因为编译要求jdk1.7,部分源不支持jdk1.7访问,网络人说是网络协议的问题,换1.8就好了。
    这里编译指定了1.7,要编译的话只能用下面方法。
        换源,比如在maven的配置文件中,增加ali源。下载依赖时,同一个包会在多个库之间轮询,轮询到ali原,就下载成功了。
        偶尔出现下载卡主的情况,这时可以ctrl c掉,重新编译。
        有时候换源也不解决问题,只能手动将对应的包放入maven的本次仓库内。
        便捷的方法,是复制编译过程中提示的依赖地址,然后进到maven对应的依赖目录下,使用wget命令将包get下来。
        或者直接从其他地方拷一份较完整的仓库过来,附件提供的仓库可能不全,但涵盖了大部分依赖包,可以节省部分时间。

编译结果
# apache-2.7.6版本编译完成。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.413 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.060 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.873 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.796 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.214 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.974 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  5.477 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  8.330 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  5.958 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.701 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:28 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.182 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 10.096 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.040 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [04:20 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 38.823 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.453 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.027 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.058 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.036 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:54 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 18.703 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.040 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  8.185 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 15.118 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  4.324 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 16.786 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.241 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.823 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  5.175 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.580 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.046 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.914 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.467 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.041 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.651 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.318 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.198 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 13.687 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 14.004 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.039 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  6.411 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  5.591 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.838 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.332 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.209 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.373 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.223 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.516 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.895 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.894 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.840 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.690 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.636 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  3.118 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  5.782 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.930 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [01:19 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.164 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.182 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.910 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  5.668 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  7.558 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.031 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [04:35 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:52 min
[INFO] Finished at: 2018-12-12T16:40:32+08:00
[INFO] Final Memory: 242M/3915M
[INFO] ------------------------------------------------------------------------


# cdh-5.5.0版本编译完成。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.799 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.969 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.744 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.317 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.852 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.922 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  3.499 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.985 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.539 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:23 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.392 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 10.646 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.040 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [04:28 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:18 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  3.934 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.894 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.061 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.043 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:01 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 18.434 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.027 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  8.025 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 23.947 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  7.871 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  6.325 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 16.706 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  0.896 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.011 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.033 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.224 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.695 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.042 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.132 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  4.410 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.159 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 18.474 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 19.756 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.631 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.248 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  4.414 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.481 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.721 s]
[INFO] hadoop-mapreduce-client-nativetask ................. SUCCESS [01:05 min]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.870 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  4.490 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.039 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.679 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.838 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.622 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 58.028 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.962 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.581 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.339 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  5.728 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.728 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  7.629 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.251 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  4.500 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.157 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.543 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  9.379 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.088 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [03:02 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:23 min
[INFO] Finished at: 2018-12-12T17:22:39+08:00
[INFO] Final Memory: 258M/4065M
[INFO] ------------------------------------------------------------------------

获取官方自带native
    hadoop-2.7.6
        下载tar包,tar开。
        /path/to/hadoop/lib/native
    hadoop-2.6.0-cdh5.5.0
        下载rpm包  https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.0/RPMS/x86_64/hadoop-2.6.0+cdh5.5.0+921-1.cdh5.5.0.p0.15.el6.x86_64.rpm
        $ rpm2cpio hadoop-2.6.0+cdh5.5.0+921-1.cdh5.5.0.p0.15.el6.x86_64.rpm | cpio -div
        $ ./usr/lib/hadoop/lib/native

编译后的native对比
    hadoop-2.7.6
        在增加 -Drequire.snappy -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Drequire.openssl -Dopenssl.lib=/usr/lib64 -Dbundle.openssl 参数时,
            与不增加上述参数编译的native包对比,增加了 libcrypto.so* libk5crypto.so* libsnappy.so* 等文件。
            libk5crypto.so是一个失效的连接,需要手动找到连接对应的共享库,拷贝到native目录下。
        不增加上述参数,和自带的共享库对比,共享库文件大小会发生改变,但不会有共享库缺失和增加。
    hadoop-2.6.0-cdh5.5.0
        在增加 -Drequire.snappy -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Drequire.openssl -Dopenssl.lib=/usr/lib64 -Dbundle.openssl 参数时,
            与不增加上述参数编译的native包对比,增加了 libcrypto.so* libk5crypto.so* libsnappy.so* 等文件。
            libk5crypto.so是一个失效的连接,需要手动找到连接对应的共享库,拷贝到native目录下。
        不增加上述参数,和自带的共享库对比,增加了 libhdfs.so libhdfs.so.0.0.0 文件,缺少了 libsnappy.so libsnappy.so.1 libsnappy.so.1.1.3 文件。

部署集群后native的检查
    $ hadoop checknative    ## 均显示true,则说明native正常。

18/12/12 19:02:31 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/12/12 19:02:31 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /src/hadoop-2.6.0-cdh5.5.0/hadoop-dist/target/hadoop-2.6.0-cdh5.5.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /src/hadoop-2.6.0-cdh5.5.0/hadoop-dist/target/hadoop-2.6.0-cdh5.5.0/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /src/hadoop-2.6.0-cdh5.5.0/hadoop-dist/target/hadoop-2.6.0-cdh5.5.0/lib/native/libcrypto.so


    一般来讲,只要hadoop本身的共享库,即自带共享库可用,其他库如 zlib snappy lz4 bzip2 openssl 等都可以加载系统自带的共享库(默认安装位置)。
    有强迫症的话,可以将系统自带的共享库文件,拷贝一份放到 native 目录下,执行 checknative 所使用的库都会使用 native 下的共享库。
    手动拷贝共享库到native,和编译时指定参数,效果未经测试。

 

附件,编译所需的大部分依赖库。

https://download.csdn.net/download/anyuzun/10846380