hadoop平台搭建

一hadoop安装过程提示：文中标注红色字体，修改配置文件时需与实际环境结合注意更改主机名，ip，文件路径。

1.配置IP地址vi /etc/sysconfig/network-scripts/ifcfg-eth0修改：BOOTPROTO=none添加：NETMASK=255.255.255.0IPADDR=IP地址2.修改主机名vi /etc/sysconfig/network修改：HOSTNAME=主机名3.vi /etc/hosts （3节点都修改）添加：192.168.18.201 h201192.168.18.202 h202192.168.18.203 h203IP地址主机名#有几个节点添加几个#4.每台机器创建hadoop 用户hadoop 密码:password 命令如下useradd hadooppasswd hadoop#如果直接安装在root用户下则不用创建hadoop用户由于安全问题建议安装在hadoop用户下#5.安装jdk（每台都安装）#本次安装目录在/usr,安装目录随意但注意声明环境变量时注意修改jdk位置#解压jdk -C 指定安装目录[root@master usr]# tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/修改环境变量[root@master~]# vi /etc/profileexport JA VA_HOME=/usr/jdk1.7.0_25export JA VA_BIN=/usr/jdk1.7.0_25/binexport PATH=$PATH:$JA V A_HOME/binexport CLASSPA TH=.:$JA V A_HOME/lib/dt.jar:$JA V A_HOME/lib/tools.jarexport JA VA_HOME JA V A_BIN PATH CLASSPA TH重启机器使换进变量生效[root@master~]# reboot6.切换hadoop用户su - hadoop安装ssh 证书生成公钥#每个节点执行如下命令（本次安装三个节点）#[hadoop@master ~]$ ssh-keygen -t rsa[hadoop@slave1~]$ ssh-keygen -t rsa[hadoop@slave2~]$ ssh-keygen -t rsa输入命令后按四次Enter键不要输入任何命令显示如下内容表示创建成功Generating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa):/root/.ssh/id_rsa already exists.Overwrite (y/n)? yEnter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:49:3a:5c:3c:50:d8:b3:a4:13:f4:6c:a9:b0:96:53:cb root@The key's randomart image is:+--[ RSA 2048]----+| .o+. || o=+. || . .+Xo || BoB.o || = E.S || . . . || || |+-----------------+将生成的公钥拷贝到远程服务器上ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub 主机名#每个节点都要执行#[hadoop@master ~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub master [hadoop@slave1~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub slave1 [hadoop@slave2~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub slave2 按提示输入内容检验是否配置成功远程连接其他节点ssh 其他节点主机名免密码直接登录则成功注意：如果连接不成功将/home/hadoop/.ssh 文件中的内容删掉重新直接以上命令或拷贝/home/hadoop/.ssh/id_rsa.pub 内容到所要连接节点的/home/hadoop/.ssh/authorized_keys目录下。

7.拷贝hadoop安装包到/home/hadoop 解压缩[hadoop@master hadoop]$ cp hadoop-2.6.0.tar.gz /home/hadoop[hadoop@master~]$ tar -zxvf hadoop-2.6.0.tar.gz修改环境变量[hadoop@master ~]$ vi .bash_profile声明hadoop和java的home，bin，conf目录添加以下内容：export JA VA_HOME=/usr/jdk1.7.0_25export JA VA_BIN=/usr/jdk1.7.0_25/binexport PATH=$PATH:$JA V A_HOME/binexport CLASSPA TH=.:$JA V A_HOME/lib/dt.jar:$JA V A_HOME/lib/tools.jarexport JA VA_HOME JA V A_BIN PATH CLASSPA THHADOOP_HOME=/home/hadoop/hadoop-2.6.0-cdh5.5.2HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopPATH=$HADOOP_HOME/bin:$PA THexport HADOOP_HOME HADOOP_CONF_DIR PATH执行以下命令使其生效[hadoop@master ~]$ source .bash_profile8.切换到hadoop-2.6.0-cdh5.5.2/etc/hadoop/目录下修改配置文件（现在主节点修改，修改完之后直接拷贝到其他节点）[hadoop@master ~]$ cd hadoop-2.6.0-cdh5.5.2/etc/hadoop/[hadoop@master hadoop]$ vi core-site.xml添加以下内容：<configuration><property><name>fs.defaultFS</name><value>hdfs://主机名:9000</value><description>NameNode URI.</description></property><property><name>io.file.buffer.size</name><value>131072</value><description>Size of read/write buffer used inSequenceFiles.</description></property></configuration>9.编辑hdfs-site.xml（管理hdfs的文件）首先创建三个目录用于保存nn ，snn，dn三个进程的信息[hadoop@master hadoop-2.6.0]$ mkdir -p /home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/name [hadoop@master hadoop-2.6.0]$ mkdir -p /home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/data [hadoop@masterhadoop-2.6.0]$mkdir -p/home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/namesecondary[hadoop@masterhadoop]$ vi hdfs-site.xml添加以下内容<property><name>node.secondary.http-address</name><value>主机名:50090</value><description>The secondary namenode http server address andport.</description></property><property><name>.dir</name><value>file:///home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/name</value><description>Path on the local filesystem where the NameNodestores the namespace and transactions logs persistently.</description></property><property><name>dfs.datanode.data.dir</name><value>file:///home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/data</value><description>Comma separated list of paths on the local filesystemof a DataNode where it should store its blocks.</description></property><property><name>node.checkpoint.dir</name><value>file:///home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/namesecondary</value><description>Determines where on the local filesystem the DFSsecondary name node should store the temporary images to merge. If this is acomma-delimited list of directories then the image is replicated in all of thedirectories for redundancy.</description></property><property><name>dfs.replication</name><value>2</value></property>#从节点有几个就填几，本次安装两个从节点填2 #10.编辑mapred-site.xml（执行mapreduce的文件）[hadoop@master hadoop]$ cp mapred-site.xml.template mapred-site.xml<property><name></name><value>yarn</value><description>Theruntime framework for executing MapReduce jobs. Can be one of local, classic oryarn.</description></property><property><name>mapreduce.jobhistory.address</name><value>主机名:10020</value><description>MapReduce JobHistoryServer IPC host:port</description></property><property><name>mapreduce.jobhistory.webapp.address</name><value>主机名:19888</value><description>MapReduce JobHistoryServer Web UI host:port</description></property>*****属性”“表示执行mapreduce任务所使用的运行框架，默认为local，需要将其改为”yarn”*****11.编辑yarn-site.xml（管理yarn）[hadoop@master hadoop]$ vi yarn-site.xml<property><name>yarn.resourcemanager.hostname</name><value>主机名</value><description>The hostname of theRM.</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Shuffle service that needs to be set for Map Reduceapplications.</description> </property>12.[hadoop@master hadoop]$ vi hadoop-env.shexport JA VA_HOME=/usr/jdk1.7.0_2513.[hadoop@master hadoop]$ vi slavesslave1slave2#天剑从节点主机名，有几个从节点几个#14.拷贝hadoop文件到其他从节点[hadoop@master ~]$ scp -r ./hadoop-2.6.0/ hadoop@slave1:/home/hadoop/[hadoop@master~]$ scp -r ./hadoop-2.6.0/ hadoop@slave2:/home/hadoop/15.格式化namenode[hadoop@master hadoop-2.6.0]$ bin/hdfs namenode -format开启hadoop所有进程[hadoop@master hadoop-2.6.0]$ sbin/start-all.sh查看进程主[hadoop@master hadoop-2.6.0]$ jps7054 SecondaryNameNode7844 Jps7318 NameNode7598 ResourceManager从[hadoop@slave1 hadoop-2.6.0-cdh5.5.2]$ jps3869 DataNode7087 Jps4854 NodeManager试用[hadoop@master hadoop-2.6.0]$ bin/hadoop fs -ls /[hadoop@masterhadoop-2.6.0]$ bin/hadoop fs -mkdir /aaa注意：1.WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable如果报上面的警告：[hadoop@h201 ~]$ tar -xvf hadoop-native-64-2.6.0.tar -C hadoop-2.6.0/lib/native/[root@h201 ~]# vi /etc/yum.conf[Server]name=rhel_yumbaseurl=file:///mnt/Serverenabled=1gpgcheck=1gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release[root@master]# yum -y install gcc*系统中的glibc的版本和libhadoop.so需要的版本不一致导致[hadoop@master ~]$ ls -l /lib/libc.so.6lrwxrwxrwx 1 root root 11 2015-07-04 /lib/libc.so.6 -> libc-2.5.so[hadoop@master~]$ tar -jxvf glibc-2.9.tar.bz2[hadoop@master~]$ cd glibc-2.9注意：glibc-linuxthreads解压到glibc目录下[hadoop@masterglibc-2.9]$ cp /ff/hadoopCDH5/glibc-linuxthreads-2.5.tar.bz2 .[hadoop@masterglibc-2.9]$ tar -jxvf glibc-linuxthreads-2.5.tar.bz2[hadoop@master~]$ export CFLAGS="-g -O2"(加上优化开关)[hadoop@master ~]$ ./glibc-2.9/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin[hadoop@master~]$ make[hadoop@master~]$ make install[hadoop@master~]$ ls -l /lib/libc.so.6重新启动2.如果格式化namnode报错检查hdfs-site.xml文件是否配置正确，修改后删除/home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/name ，data，namesecondary 三个文件中的内容重新执行bin/hdfs namenode -format3.如开启hadoop进程后jps不全根据缺失进程检查日志/home/hadoop/hadoop-2.6.0-cdh5.5.2/logs/hadoop-hadoop-datanode-slave1.log根据日志错误查找原因可能原因：主/home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/name/current/VERSION 与从/home/hadoop/hadoop-2.6.0-cdh5.5.2/dfs/data/current/VERSION版本号不一致改为一致即可。