CentOS6.8上编译Hadoop-2.2.0

由于Hadoop-2.2.0是在32位的基础上编译的，如下图所示，我们看到从官网下载的hadoop-2.2.0默认是32位的，这在我们64位的虚拟机上运行会有问题。因此我们需要对Hadoop源码进行编译，将它变成64位的。

[root@itcast05 ~]# ls
anaconda-ks.cfgapache-maven-3.0.5-bin.tar.gz Documents hadoop-2.2.0 install.log jdk-7u80-linux-x64.gz PicturesPublic Templates apache-ant-1.9.2-bin.tar.gz DesktopDownloads hadoop-2.2.0.tar.gz install.log.syslog Musicprotobuf-2.5.0.tar.gz target Videos
[root@itcast05 ~]# cd hadoop-2.2.0
[root@itcast05 hadoop-2.2.0]# ls
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[root@itcast05 hadoop-2.2.0]# cd lib
[root@itcast05 lib]# ls
native
[root@itcast05 lib]# cd native/
[root@itcast05 native]# ls
libhadoop.a libhadooppipes.a libhadoop.so libhadoop.so.1.0.0 libhadooputils.a libhdfs.a libhdfs.so libhdfs.so.0.0.0
[root@itcast05 native]# file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 32-bit LSB shared object,Intel 80386,version 1 (SYSV),dynamically linked,not stripped
[root@itcast05 native]#

接下来我们便一步一步图文解说如何编译。

第一步：确保连网

要确保我们的虚拟机能够连网，关于如何给虚拟机配置固定IP并且怎样连网大家如果不知道怎么操作的话，可以参考：http://blog.csdn.net/u012453843/article/details/52839105这篇博客进行学习。

第二步：安装gcc

依次执行下面四条命令，在CentOS6.8当中，gcc和make已经安装过了，不过为了保证后续不出错，最好都执行一遍，如果有的已经安装过了，无非会提醒你已经安装过了，这没什么关系。

[root@itcast05 ~]# yum install gcc

[root@itcast05 ~]#yum -y install gcc-c++

[root@itcast05 ~]#yum install make

[root@itcast05 ~]# yum install autoconf automake libtool cmake ncurses-devel openssl-devel gcc*

第三步：安装JDK

3.1 从官网下载JDK，我下载的是jdk-7u80-linux-x64.gz这个版本，大家也可以到我的百度云盘：http://pan.baidu.com/s/1sl3rpPb这个网址进行下载。

3.2 接着我们使用FileZilla工具把jdk-7u80-linux-x64.gz安装包上传到我们的服务器（也就是我们的虚拟机），这里我上传到root根目录下了，如下图所示。关于FileZilla的使用如果大家不知道怎么使用的话，可以参考：http://blog.csdn.net/u012453843/article/details/52422736这篇博客进行学习。

3.3 我们在/usr/目录下创建一个目录，用来安装我们的JDK。

[root@itcast05 ~]# mkdir /usr/java

3.4 解压jdk-7u80-linux-x64.gz到/usr/java目录下

[root@itcast05 ~]# tar -zxvf jdk-7u80-linux-x64.gz -C/usr/java/

3.5 配置环境变量，把export JAVA_HOME=/usr/java/jdk1.7.0_80和export PATH=$JAVA_HOME/bin:$PATH放到最后，如下所示，配置完之后先按ESC键退出输入模式然后按住Shift键不放再连续两次按Z键即可保存退出该文件。

[root@itcast05 ~]# vim /etc/profile
*)
if [ "$2" = "after" ] ; then
PATH=$PATH:$1
else
PATH=$1:$PATH
fi
esac
}

if [ -x /usr/bin/id ]; then
if [ -z "$EUID" ]; then
# ksh workaround
EUID=`id -u`
UID=`id -ru`
fi
USER="`id -un`"
LOGNAME=$USER
MAIL="/var/spool/mail/$USER"
fi

# Path manipulation
if [ "$EUID" = "0" ]; then
pathmunge /sbin
pathmunge /usr/sbin
pathmunge /usr/local/sbin
else
pathmunge /usr/local/sbin after
pathmunge /usr/sbin after
pathmunge /sbin after
fi

HOSTNAME=`/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
export HISTCONTROL=ignoreboth
else
export HISTCONTROL=ignoredups
fi

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL

# By default,we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You Could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`id -gn`" = "`id -un`" ]; then
umask 002
else
umask 022
fi

for i in /etc/profile.d/*.sh ; do
if [ -r "$i" ]; then
if [ "${-#*i}" != "$-" ]; then
. "$i"
else
. "$i" >/dev/null 2>&1
fi
fi
done

unset i
unset -f pathmunge
export JAVA_HOME=/usr/java/jdk1.7.0_80
export PATH=$JAVA_HOME/bin:$PATH

3.6 配置完/etc/profile文件需要使用命令：source /etc/profile来使刚才输入的配置生效。

[root@itcast05 ~]# source /etc/profile

3.7 接下来检测安装的jdk是否成功，我们使用命令java和java -version来检测，如下所示，说明我们安装成功了。

[root@itcast05 ~]# java
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)
where options include:
-d32 use a 32-bit data model if available
-d64 use a 64-bit data model if available
-server to select the "server" VM
The default VM is server.

-cp <class search path of directories and zip/jar files>
-classpath <class search path of directories and zip/jar files>
A : separated list of directories,JAR archives,
and ZIP archives to search for class files.
-D<name>=<value>
set a system property
-verbose:[class|gc|jni]
enable verbose output
-version print product version and exit
-version:<value>
require the specified version to run
-showversion print product version and continue
-jre-restrict-search | -no-jre-restrict-search
include/exclude user private JREs in the version search
-? -help print this help message
-X print help on non-standard options
-ea[:<packagename>...|:<classname>]
-enableassertions[:<packagename>...|:<classname>]
enable assertions with specified granularity
-da[:<packagename>...|:<classname>]
-disableassertions[:<packagename>...|:<classname>]
disable assertions with specified granularity
-esa | -enablesystemassertions
enable system assertions
-dsa | -disablesystemassertions
disable system assertions
-agentlib:<libname>[=<options>]
load native agent library <libname>,e.g. -agentlib:hprof
see also,-agentlib:jdwp=help and -agentlib:hprof=help
-agentpath:<pathname>[=<options>]
load native agent library by full pathname
-javaagent:<jarpath>[=<options>]
load Java programming language agent,see java.lang.instrument
-splash:<imagepath>
show splash screen with specified image
See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details.
[root@itcast05 ~]# java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11,mixed mode)
[root@itcast05 ~]#

第四步：安装Maven

4.1 我们安装的版本是apache-maven-3.0.5-bin.tar.gz，大家可以到官网下载，如果不知道怎样从官网下载的话，可以参考：http://blog.csdn.net/u012453843/article/details/52854255这篇博客来下载。（如果大家觉得从官网下载麻烦，可以直接到：http://download.csdn.net/detail/u012453843/9661837这个地址进行下载）

4.2 把下载好的apache-maven-3.0.5-bin.tar.gz上传到root根目录下，然后在/usr/local/目录下新建一个maven的目录

[root@itcast05 ~]# mkdir /usr/local/maven

4.3 把apache-maven-3.0.5-bin.tar.gz解压到/usr/local/maven/目录下

[root@itcast05 ~]# tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /usr/local/maven/

4.4 配置环境变量(命令：vim /etc/profile)，刚才配置了jdk，现在再把maven加加进来(红色字体部分)，如下所示。

export JAVA_HOME=/usr/java/jdk1.7.0_80
export MAVEN_HOME=/usr/local/maven/apache-maven-3.0.5
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

4.5 配置完之后，我们使用命令source /etc/profile来使环境变量生效。

[root@itcast05 ~]# source /etc/profile

4.6 环境变量生效后我们来检查是否配置成功，出现如下所示的信息说明安装maven成功。

[root@itcast05 ~]# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)
Maven home: /usr/local/maven/apache-maven-3.0.5
Java version: 1.7.0_80,vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_80/jre
Default locale: en_US,platform encoding: UTF-8
OS name: "linux",version: "2.6.32-642.el6.x86_64",arch: "amd64",family: "unix"
[root@itcast05 ~]#

第五步：安装Ant

5.1 我们从http://download.csdn.net/detail/u012453843/9661840这个地址下载我们所需要的apache-ant-1.9.2-bin.tar.gz。

5.2 上传到root根目录下，并在/usr/local/目录下创建一个ant目录

[root@itcast05 ~]# mkdir /usr/local/ant

5.3 把apache-ant-1.9.2-bin.tar.gz解压到/usr/local/ant/目录下

[root@itcast05 ~]# tar -zxvf apache-ant-1.9.2-bin.tar.gz -C /usr/local/ant/

5.4 我们配置环境变量(命令：vim /etc/profile)，，我们在原来的基础上加上关于ant的配置（红色字体部分）

export JAVA_HOME=/usr/java/jdk1.7.0_80
export MAVEN_HOME=/usr/local/maven/apache-maven-3.0.5
export ANT_HOME=/usr/local/ant/apache-ant-1.9.2
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin :$ANT_HOME/bin:$PATH

5.5 配置完环境变量，使用命令：source /etc/profile来使配置生效

[root@itcast05 ~]# source /etc/profile

5.6 检查ant安装是否成功，出现如下所示的消息便说明ant安装成功。

[root@itcast05 ~]# ant -version
Apache Ant(TM) version 1.9.2 compiled on July 8 2013
[root@itcast05 ~]#

第六步：安装protobuf

6.1 我们从http://download.csdn.net/detail/u012453843/9661841这个地址下载我们的安装包protobuf-2.5.0.tar.gz。

6.2 上传到root根目录下，并在/usr/local/目录下创建protobuf目录

[root@itcast05 ~]# mkdir /usr/local/protobuf

6.3 将protobuf-2.5.0.tar.gz解压到/usr/local/protobuf/目录下

[root@itcast05 ~]# tar -zxvf protobuf-2.5.0.tar.gz -C /usr/local/protobuf/

6.4 进入到protobuf-2.5.0目录下， 一定要按顺序依次执行./configure、make、make check、make install命令。

[root@itcast05 ~]# cd /usr/local/protobuf/
[root@itcast05 protobuf]# ls
protobuf-2.5.0
[root@itcast05 protobuf]# cd protobuf-2.5.0/
[root@itcast05 protobuf-2.5.0]# ls
aclocal.m4 config.guess config.log configure copYING.txt examples install-sh libtool Makefile missing protobuf.pc README.txt vsprojects autogen.sh config.h config.status configure.ac depcomp generate_descriptor_proto.sh INSTALL.txt ltmain.sh Makefile.am protobuf-lite.pc protobuf.pc.in src CHANGES.txt config.h.in config.sub CONTRIBUTORS.txt editors gtest java m4 Makefile.in protobuf-lite.pc.in python stamp-h1
[root@itcast05 protobuf-2.5.0]# ./configure

[root@itcast05 protobuf-2.5.0]# make

[root@itcast05 protobuf-2.5.0]# make check

[root@itcast05 protobuf-2.5.0]# make install

6.5 protobuf不用专门配置环境变量，直接使用命令：protoc --version，如果出现如下所示的信息，说明protobuf安装成功。

[root@itcast05 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@itcast05 protobuf-2.5.0]#

第七步：Hadoop-2.2.0源码解压及Bug的修复

7.1 我们从http://pan.baidu.com/s/1jHZLT90这个地址直接下载hadoop-2.2.0-src.tar.gz。

7.2 在/usr/local/目录下创建hadoop目录

[root@itcast05 ~]# mkdir /usr/local/hadoop

7.3 将hadoop-2.2.0-src.tar.gz解压到/usr/local/hadoop目录下。

[root@itcast05 ~]#tar -zxvf hadoop-2.2.0.tar.gz -C /usr/local/hadoop/

7.4 Hadoop-2.2.0有个补丁需要修复一下

我们修改的文件是pom.xml，它所在的目录是：/root/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/pom.xml，如下所示，我们需要在pom.xml中添加的内容是下面蓝色字体的部分，其他不用动。

[root@itcast05 ~]# cd /usr/local/hadoop/hadoop-2.2.0-src
[root@itcast05 hadoop-2.2.0-src]# ls
BUILDING.txt hadoop-assemblies hadoop-common-project hadoop-hdfs-project hadoop-maven-plugins hadoop-project hadoop-tools LICENSE.txt pom.xml
dev-support hadoop-client hadoop-dist hadoop-mapreduce-project hadoop-minicluster hadoop-project-dist hadoop-yarn-project NOTICE.txt README.txt
[root@itcast05 hadoop-2.2.0-src]# cd hadoop-common-project/
[root@itcast05 hadoop-common-project]# ls
dev-support hadoop-annotations hadoop-auth hadoop-auth-examples hadoop-common hadoop-nfs pom.xml target
[root@itcast05 hadoop-common-project]# cd hadoop-auth
[root@itcast05 hadoop-auth]# ls
BUILDING.txt pom.xml README.txt src target
[root@itcast05 hadoop-auth]# vim pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed under the Apache License,Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-project</artifactId>
<version>2.2.0</version>
<relativePath>../../hadoop-project</relativePath>
</parent>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-auth</artifactId>
<version>2.2.0</version>
<packaging>jar</packaging>

<name>Apache Hadoop Auth</name>
<description>Apache Hadoop Auth - Java HTTP SPNEGO</description>

<properties>
<maven.build.timestamp.format>yyyyMMdd</maven.build.timestamp.format>
<kerberos.realm>LOCALHOST</kerberos.realm>
</properties>

<dependencies>
<dependency>
<!-- Used,even though 'mvn dependency:analyze' doesn't find it -->
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
"pom.xml" 219L,7174C

第八步：Maven的setting.xml配置修改

由于Maven-3.0.5的配置文件setting.xml文件中的配置过于老，与我们安装的环境不相符，因此我们需要修改一下这个文件的配置

8.1 我们找到setting.xml文件

[root@itcast05 ~]# cd /usr/local/maven/apache-maven-3.0.5
[root@itcast05 apache-maven-3.0.5]# ls
bin boot conf lib LICENSE.txt NOTICE.txt README.txt
[root@itcast05 apache-maven-3.0.5]# cd conf/
[root@itcast05 conf]# ls
setting.xml

8.2 由于setting.xml整体配置过旧，因此我们删掉它原来的文件，并再新建一个相同名字的文件
[root@itcast05 conf]# rm -rf setting.xml

[root@itcast05 conf]# touch setting.xml

8.3 给setting.xml文件配置如下内容，把下面浅蓝色内容全部粘贴到setting.xml文件当中并保存退出。

[root@itcast05 conf]# vim setting.xml

<?xmlversionxmlversion="1.0"encoding="utf-8"?>
<settingsxmlnssettingsxmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0http://maven.apache.org/xsd/settings-1.0.0.xsd">
<pluginGroups></pluginGroups>
<proxies></proxies>
<servers></servers>
<mirrors>
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
</mirrors>
<profiles>
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>localprivatenexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>localprivatenexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
</settings>

第九步：编译Hadoop-2.2.0

9.1 我们进入到hadoop-2.2.0-src目录下，如下所示。

[root@itcast05 ~]# cd /usr/local/hadoop/hadoop-2.2.0-src
[root@itcast05 hadoop-2.2.0-src]# ls
BUILDING.txt hadoop-assemblies hadoop-common-project hadoop-hdfs-project hadoop-maven-plugins hadoop-project hadoop-tools LICENSE.txt pom.xml
dev-support hadoop-client hadoop-dist hadoop-mapreduce-project hadoop-minicluster hadoop-project-dist hadoop-yarn-project NOTICE.txt README.txt
[root@itcast05 hadoop-2.2.0-src]#

9.2 执行mvn clean

[root@itcast05 hadoop-2.2.0-src]# mvn clean

9.3 执行mvn package -Pdist,native -DskipTests -Dtar，这步的时间很长，大约一个小时左右，需要耐心等待，不要乱动。

[root@itcast05 hadoop-2.2.0-src]# mvn package -Pdist,native -DskipTests -Dtar

9.4 执行完上面那条命令之后，会出现如下图所示的界面，所列项后面全是"SUCCESS"并且"BUILD SUCCESS"说明编译成功了。

9.5 那么生成的文件在哪儿呢？在/usr/local/hadoop/hadoop-2.2.0-src/hadoop-dist/target/这个目录下，有hadoop-2.2.0和hadoop-2.2.0.tar.gz，这就是我们编译好的文件。

[root@itcast05 hadoop-2.2.0-src]# cd hadoop-dist/
[root@itcast05 hadoop-dist]# ls
pom.xml target
[root@itcast05 hadoop-dist]# cd target/
[root@itcast05 target]# ls
antrun dist-layout-stitching.sh dist-tar-stitching.sh hadoop-2.2.0 hadoop-2.2.0.tar.gz hadoop-dist-2.2.0.jar hadoop-dist-2.2.0-javadoc.jar javadoc-bundle-options maven-archiver test-dir
[root@itcast05 target]#

9.6 我们检验一下是否编译成的文件已经是64位，如下所示，我们发现确实已经变成64位的了！！！

[root@itcast05 target]# cd hadoop-2.2.0
[root@itcast05 hadoop-2.2.0]# ls
bin etc include lib libexec sbin share
[root@itcast05 hadoop-2.2.0]# cd lib
[root@itcast05 lib]# ls
native
[root@itcast05 lib]# cd native/
[root@itcast05 native]# ls
libhadoop.a libhadooppipes.a libhadoop.so libhadoop.so.1.0.0 libhadooputils.a libhdfs.a libhdfs.so libhdfs.so.0.0.0
[root@itcast05 native]# file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object,x86-64,not stripped
[root@itcast05 native]#

至此，我们终于编译成功了64位的Hadoop-2.2.0！！！