大数据平台部署文档

大数据平台搭建

一．Linux环境准备

1)格式化磁盘

mkfs.ext4 /dev/dm-0

mkdir /data

mount /dev/dm-0 /data 系统重启需要手动挂载

把mount /dev/dm-0 /data 的命令放到/etc/rc.d/rc.local 重启自动挂载。df -h 查看挂载点

2)网卡绑定(当服务器有多块网卡需要网卡绑定)

•配置绑定网卡

[root@hadoop001 network-scripts]# vi

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0

TYPE=Ethernet

ONBOOT=yes

BOOTPROTO=static

IPADDR=19.106.64.7

NETMASK=255.255.255.0

GATEWAY=19.106.64.254

NDS1=19.104.4.3

DNS2=19.104.8.3

USERCTL=no

•各网卡设置

[root@hadoop001 network-scripts]# vi

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

TYPE=Ethernet

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

•配置文件添加vi /etc/modprobe.d/dist.conf 末尾追加

alias bond0 bonding

options bond0 miimon=100 mode=0#每100ms检测一次链路mode=0网卡冗余机制

•设置服务

service NetworkManager stop chkconfig NetworkManager off service network restart

chkconfig network on

•测试绑定网卡

modprobe bond0#如果报错就配置错误查看bond0网卡信息

3)添加用hadoop

useradd hadoop

passwd hadoop #hadoop

4)修改主机名文件

vi /etc/sysconfig/network#修改该文件Hostname=hadoop001

5）修改本地hosts文件

19.106.64.7 hadoop001

19.106.64.8 hadoop002

19.106.64.9 hadoop003

19.106.64.10 hadoop004

19.106.64.11 hadoop005

19.106.64.12hadoop006

6）关闭防火墙（root 用户下操作）service iptables status 查看防火墙状态service iptables stop 临时关闭防火墙chkconfig iptables off 永久关闭防火墙

7）修改selinux

vi /etc/selinux/config

selinux = disabled

8）ssh互信（hadoop用户下）

生成密钥对

ssh-keygen #一路回车就行

ssh-copy-id hadoop001 分发公钥

ssh-copy-id hadoop002

ssh-copy-id hadoop003

测试连接获取主机名文件；

cd /home/hadoop/.ssh 将私钥和hosts文件分发各个节点确保完全互信scp id_rsa known_hosts hadoop@hadoop002:/home/hadoop/.ssh scp id_rsa known_hosts hadoop@hadoop003:/home/hadoop/.ssh ……

9)配置时间同步服务

设置NTP服务自动同步硬件时间

vi /etc/sysconfig/ntpd

SYNC_HWCLOCK=yes

Server端：

restrict 19.106.64.0 mask 255.255.255.0 nomodify notrap

server 127.127.1.0

fudge 127.127.1.0 stratum 8

client端：

restrict 19.106.64.0 mask 255.255.255.0 nomodify notrap

server 19.106.64.7

service ntpd start

chkconfig ntpd on

10）卸载系统自带的jdk

[root@hadoop001 ~]# rpm -qa|grep java

java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

tzdata-java-2013g-1.el6.noarch

[root@hadoop001 ~]# rpm -e --nodeps tzdata-java-2013g-1.el6.noarch [root@hadoop001 ~]# rpm -qa|grep java

java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64