corosync+pacemaker+drbd

目录一、环境准备 (3)二、设置各节点互相解析 (3)1. 设置主机名 (3)2. 设置各节点ssh互信 (4)3. 设置各节点时间同步 (4)4. 设置各节点关闭防火墙及selinux (4)三、corosync安装及配置 (5)1. corosync安装 (5)2. corosync配置 (5)四、Pacemaker安装配置 (6)1. pacemaker安装 (6)五、crmsh安装配置 (6)1. crmsh安装 (6)六、检查集群是否安装成功 (7)1. 启动corosync (7)2. 查看启动信息 (7)七、drbd安装 (7)1. drbd安装包下载（node1 node2 同做） (7)2. drbd编译安装（node1 node2 同做） (8)3. 检查drbd模块是否编译正确（node1 node2 同做） (8)4. drbd配置 (8)5. 添加共享硬盘（node1 node2 同做） (9)6. drbd初始化（node1 node2 同做） (10)7. 启动drbd (10)8. 设置node1为主节点 (10)9. 格式化并挂载 (10)10. 设置node2为主节点 (10)八、oracle 数据库安装 (11)1. 源码包下载 (11)2. 安装必要依赖软件（node1 node2 同做） (11)3. 添加用户名及用户组（node1 node2 同做） (11)4. 调整内核参数（node1 node2 同做） (11)5. 增加shell限制（node1 node2 同做） (11)6. 设置node1为主节点 (12)7. 建立必要目录（node1 node2 同做） (12)8. 设置oracle环境变量（node1 node2 同做） (12)9. 解压oracle安装文件（只node1做） (12)10. 生成响应文件（只node1做） (13)11. 安装app (13)12. 配置监听 (14)13. 建库 (14)14. 将oracle做成服务（node1 node2 同做） (14)九、corosync配置 (15)十、mysql 加入corosync (16)1. mysql 安装（node1 node2 同做） (16)2. 修改mysql配置文件（node1 node2 同做） (16)3. 创建mysql数据目录 (16)4. 配置mysql加入corosync (16)十一、后期问题处理 (16)corosync+pacemaker+drbd一、环境准备1.虚拟机两台，centos 6.5 32位以安装镜像光盘为yum源，添加两块网卡eth0，eth12.eth0为桥接配置文件如下：DEVICE=eth0TYPE=EthernetUUID=47d360e2-96c6-455b-bd36-b744371c708cONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=staticIPADDR=10.10.10.20 ##node1为10.20 node2为10.21NETMASK=255.255.255.0GATEWAY=10.10.10.1eth1为nat配置文件如下：DEVICE=eth1TYPE=EthernetUUID=ff5bed04-b5e4-4977-912e-b0bdb04eedc3ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=dhcp3.vip为10.10.10.80二、设置各节点互相解析1.设置主机名node1:[root@node1 ~]# uname -n[root@node1 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.10.10.20 node110.10.10.21 node210.10.10.80 nodenode2:[root@node2 ~]# uname -n[root@node2 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.10.10.20 node110.10.10.21 node210.10.10.80 node2.设置各节点ssh互信node1:[root@node1 ~]# ssh-keygen -t rsa[root@node1 ~]# yum install openssh-clients –y[root@node1 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@node2:[root@node2 ~]# ssh-keygen -t rsa[root@node2 ~]# yum install openssh-clients –y[root@node2 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@3.设置各节点时间同步node1:[root@node1 ~]# hwclock –snode2:[root@node2 ~]# hwclock –s4.设置各节点关闭防火墙及selinuxnode1:[root@node1 ~]# service iptables stop[root@node1 ~]# chkconfig iptables off[root@node1 ~]# cat /etc/selinux/config# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of these two values:# targeted - T argeted processes are protected,# mls - Multi Level Security protection.SELINUXTYPE=targetednode2:[root@node2 ~]# service iptables stop[root@node2 ~]# chkconfig iptables off[root@node2 ~]# cat /etc/selinux/config# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of these two values:# targeted - T argeted processes are protected,# mls - Multi Level Security protection.SELINUXTYPE=targeted三、corosync安装及配置1.corosync安装node1:[root@node1 ~]# yum install corosyncnode2:[root@node2 ~]# yum install corosync2.corosync配置node1:创建corosync配置文件[root@node1 ~]# cat /etc/corosync/corosync.conf# Please read the corosync.conf.5 manual pagecompatibility: whitetanktotem {version: 2secauth: offthreads: 0interface {ringnumber: 0bindnetaddr: 10.10.10.1mcastaddr: 226.94.10.1mcastport: 5405ttl: 1}}logging {fileline: offto_stderr: noto_logfile: yesto_syslog: yeslogfile: /var/log/cluster/corosync.logdebug: offtimestamp: onlogger_subsys {subsys: AMFdebug: off}}amf {mode: disabled}service {ver: 0name: pacemaker}aisexec {user: rootgroup: root}●生成密匙文件[root@node1 ~]# cd /etc/corosync/[root@node1 corosync]# mv /dev/{random,random.bak}[root@node1 corosync]# ln -s /dev/urandom /dev/random[root@node1 corosync]# corosync-keygen●将corosync配置文件及密匙文件拷贝至node2[root@node1 corosync]# scp -p authkey corosync.conf node2:/etc/corosync/四、Pacemaker安装配置1.pacemaker安装node1:[root@node1 ~]# yum install pacemakernode2:[root@node2 ~]# yum install pacemaker五、crmsh安装配置1.crmsh安装node1:[root@node1 ~]# yum install crmshnode2:[root@node2 ~]# yum install crmsh六、检查集群是否安装成功1.启动corosyncnode1:[root@node1 ~]# service corosync startnode2:[root@node2 ~]# service corosync start2.查看启动信息●查看corosync引擎是否正常启动[root@node1 ~]# grep -e "Corosync Cluster Engine" -e "configuration file"/var/log/cluster/corosync.logMar 21 00:46:49 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provideservice.Mar 21 00:46:49 corosync [MAIN ] Successfully read main configuration file'/etc/corosync/corosync.conf'.Mar 21 00:47:39 corosync [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:1947.●查看初始化成员节点是否正常发出[root@node1 ~]# grep TOTEM /var/log/cluster/corosync.logMar 20 21:47:22 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).Mar 20 21:47:22 corosync [TOTEM ] Initializing transmit/receive security: libtomcryptSOBER128/SHA1HMAC (mode 0).Mar 20 21:47:22 corosync [TOTEM ] The network interface [10.10.10.20] is now up.Mar 20 21:47:22 corosync [TOTEM ] A processor joined or left the membership and a newmembership was formed.●检查启动过程是否有错误产生[root@node2 ~]# grep ERROR /var/log/cluster/corosync.logMar 21 04:40:44 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster usingthe Pacemaker plugin for Corosync. The plugin is not supported in this environment and will beremoved very soon.Mar 21 04:40:44 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clustersfrom Scratch' (/doc) for details on using Pacemaker with CMAN七、drbd安装1.drbd安装包下载（node1 node2 同做）/drbd/8.4/drbd-8.4.3.tar.gz2.drbd编译安装（node1 node2 同做）[root@node1 home]# tar -xzf drbd-8.4.3.tar.gz[root@node1 home]# cd drbd-8.4.3[root@node1 drbd-8.4.3]# yum install kernel-devel[root@node1 drbd-8.4.3]# ./configure --prefix=/ --with-km[root@node1 drbd-8.4.3]# make KDIR=/usr/src/kernels/2.6.32-431.el6.i686/[root@node1 drbd-8.4.3]# make install[root@node1 drbd-8.4.3]# depmod3.检查drbd模块是否编译正确（node1 node2 同做）[root@node1 drbd-8.4.3]# modprobe drbd[root@node1 drbd-8.4.3]# lsmod | grep drbddrbd 292339 3libcrc32c 841 1 drbd4.drbd配置[root@node1 ~]# vim /etc/drbd.d/global_common.confglobal {usage-count no;}common {handlers {pri-on-incon-degr "/lib/drbd/notify-pri-on-incon-degr.sh;/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";pri-lost-after-sb " /lib/drbd/notify-pri-lost-after-sb.sh;/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";local-io-error " /lib/drbd/notify-io-error.sh;/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";}startup {}options {}disk {on-io-error detach;}net {cram-hmac-alg "sha1";shared-secret "mydrbdlab";}syncer {rate 100M;}}[root@node1 ~]# vim /etc/drbd.d/web.resresource web {on {device /dev/drbd0;disk /dev/sdc1; #该值跟共享磁盘盘符一致address 10.10.10.20:7789;meta-disk internal;}on {device /dev/drbd0;disk /dev/sdc1;address 10.10.10.21:7789;meta-disk internal;}}[root@node1 ~]# scp /etc/drbd.d/* node2:/etc/drbd.d/5.添加共享硬盘（node1 node2 同做）添加硬件●分区[root@node1 ~]# fdisk –lDisk /dev/sdc: 16.1 GB, 16106127360 bytes255 heads, 63 sectors/track, 1958 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x5ece0c72[root@node1 ~]# fdisk /dev/sdc按键顺序为n->p->1->回车->回车->w●格式化[root@node1 ~]# mkfs.ext4 /dev/sdc16.drbd初始化（node1 node2 同做）[root@node1 ~]# drbdadm create-md web如果有错则执行[root@node1 ~]# dd if=/dev/zero bs=1M count=1 of=/dev/sdc1 7.启动drbd[root@node1 ~]# service drbd start[root@node2 ~]# service drbd start[root@node1 ~]# drbd-overview0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- [root@node2 ~]# drbd-overview0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r----- 8.设置node1为主节点[root@node1 ~]# drbdadm -- --overwrite-data-of-peer primary web[root@node1 ~]# drbd-overview0:web/0 Connected Primary/Secondary UpToDate/UpToDate C r----- 9.格式化并挂载[root@node1 ~]# mkfs.ext4 /dev/drbd0[root@node1 ~]# mkdir /mydata[root@node1 ~]# mount /dev/drbd0 /mydata/10.设置node2为主节点[root@node1 ~]# umount /mydata/[root@node1 ~]# drbdadm secondary web[root@node2 ~]# drbdadm primary web[root@node2 ~]# mkdir /mydata[root@node2 ~]# mount /dev/drbd0 /mydata八、oracle 数据库安装1.源码包下载/otn/linux/oracle11g/R2/linux_11gR2_database_1of2.zip/otn/linux/oracle11g/R2/linux_11gR2_database_2of2.zip2.安装必要依赖软件（node1 node2 同做）[root@node1 ~]# yum install libaio libaio-devel pdksh sysstat unixODBC unixODBC-devel3.添加用户名及用户组（node1 node2 同做）[root@node1 ~]# groupadd oinstall[root@node1 ~]# groupadd dba[root@node1 ~]# groupadd asmadmin[root@node1 ~]# groupadd asmdba[root@node1 ~]# useradd -g oinstall -G dba,asmdba oracle -d /home/oracle[root@node1 ~]# passwd oracle4.调整内核参数（node1 node2 同做）[root@node1 ~]# vim /etc/sysctl.conf 其后添加fs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 536870912kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586[root@node1 ~]# sysctl –p5.增加shell限制（node1 node2 同做）[root@node1 ~]# vim /etc/security/limits.conf 其后添加oracle soft nproc 2047oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536[root@node1 ~]# vim /etc/pam.d/login 其后添加session required pam_limits.so[root@node1 ~]# vim /etc/profile 其后添加if [ $USER = "oracle" ]; thenif [ $SHELL = "/bin/ksh" ]; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fifi6.设置node1为主节点[root@node2 ~]# umount /mydata/[root@node2 ~]# drbdadm secondary web[root@node1 ~]# drbdadm primary web[root@node1 ~]# mkdir /mydata[root@node1 ~]# mount /dev/drbd0 /mydata7.建立必要目录（node1 node2 同做）[root@node1 ~]# mkdir -p /mydata/app/oracle/ #只node1做[root@node1 ~]# chown -R oracle:oinstall /home/linux_11gR2_database_* #只node1做[root@node1 ~]# chown -R oracle:oinstall /mydata/[root@node1 ~]# chmod -R 775 /mydata/[root@node1 ~]# chmod 775 /home/linux_11gR2_database_*[root@node1 ~]# mkdir -p /oracle/[root@node1 ~]# chown -R oracle:oinstall /oracle/[root@node1 ~]# chmod -R 775 /oracle/8.设置oracle环境变量（node1 node2 同做）[root@node1 ~]# su – oracle[oracle@node1 ~]$ vim ~/.bash_profile 其后添加umask 022export ORACLE_HOSTNAME=node1 #node2 中该值为node2export ORACLE_BASE=/mydata/app/oracleexport ORACLE_HOME=/oracle/product/11.2.0/dbhome_1export ORACLE_SID=ORCLexport PATH=.:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/jdk/bin:$PATH export LC_ALL="en_US"export LANG="en_US"export NLS_LANG="AMERICAN_AMERICA.ZHS16GBK"export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"9.解压oracle安装文件（只node1做）[oracle@node1 ~]$ unzip /home/linux_11gR2_database_1of2.zip -d /mydata/[oracle@node1 ~]$ unzip /home/linux_11gR2_database_2of2.zip -d /mydata/[oracle@node1 mydata]$ cd /mydata/database/[oracle@node1 database]$ export DISTRIB=`pwd`10.生成响应文件（只node1做）●安装app[oracle@node1 database]$ vim $DISTRIB/db_swonly.rsporacle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v11_2_0 oracle.install.option=INSTALL_DB_SWONLYDECLINE_SECURITY_UPDATES=trueUNIX_GROUP_NAME=oinstallINVENTORY_LOCATION=/oracle/oraInventorySELECTED_LANGUAGES=en,zh_CN,zh_TWORACLE_HOME=/oracle/product/11.2.0/dbhome_1ORACLE_BASE=/mydata/app/oracleoracle.install.db.InstallEdition=EEoracle.install.db.isCustomInstall=trueoracle.install.db.customComponents=oracle.rdbms.partitioning:11.2.0.1.0,oracle.oraolap:11.2.0.1.0,oracle.rdbms.lbac:11.2.0.1.0,oracle.rdbms.dm:11.2.0.1.0,oracle.rdbms.dv:11.2.0.1.0,oracle.rdbms.rat:11.2.0.1.0oracle.install.db.DBA_GROUP=dbaoracle.install.db.OPER_GROUP=oinstall●安装库[oracle@node1 database]$ vim $DISTRIB/db_create.rsp[GENERAL]RESPONSEFILE_VERSION = "11.2.0"OPERATION_TYPE = "createDatabase"[CREATEDATABASE]GDBNAME = "ORCL.LK"TEMPLATENAME = "General_Purpose.dbc"CHARACTERSET = "ZHS16GBK"TOTALMEMORY = "1024"●删除库[oracle@node1 database]$ vim $DISTRIB/db_delete.rsp[GENERAL]RESPONSEFILE_VERSION = "11.2.0"OPERATION_TYPE = "deleteDatabase"[DELETEDATABASE]SOURCEDB = "ORCL"11.安装app[oracle@node1 database]$ ./runInstaller -silent -force -noconfig -responseFile$DISTRIB/db_swonly.rsp需要很长时间直到控制台输出successfully setup software.此时在另一控制台以root身份运行[root@node1 ~]# /oracle/oraInventory/orainstRoot.sh[root@node1 ~]# /oracle/product/11.2.0/dbhome_1/root.sh至此node1 app安装结束，node2只需在主节点时执行相同的命令即可，记得要执行export DISTRIB=`pwd`12.配置监听[oracle@node1 ~]$ [oracle@node1 ~]$ $ORACLE_HOME/bin/netca /silent /responsefile$DISTRIB/response/netca.rsp[oracle@node1 ~]$ vim /oracle/product/11.2.0/dbhome_1/network/admin/listener.ora# listener.ora Network Configuration File:/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora# Generated by Oracle configuration tools.LISTENER =(DESCRIPTION_LIST =(DESCRIPTION =(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))(ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.10.80)(PORT = 1521))))ADR_BASE_LISTENER = /mydata/app/oracle至此node1监听配置完成，node2只需在主节点时执行相同动作13.建库[oracle@node1 ~]$ $ORACLE_HOME/bin/dbca -silent -responseFile $DISTRIB/db_create.rsp 至此库已经建好,按照理论上来说，建库只需在node1上执行即可，node2无需建库，但是有些文件不清楚，导致无法双机启动，因此在node2上也建一次库，建库只是生成文件，因此node2建库会覆盖node1建库文件，最终不影响使用14.将oracle做成服务（node1 node2 同做）[root@node1 ~]# vim /etc/init.d/oracle#!/bin/bashcase "$1" instart)echo `date`" oracle starting" >> /home/mmmmecho `date``su - oracle -c dbstart` >> /home/mmmmecho $? >> /home/mmmmexit 0;;stop)echo `date`" oracle stoping" >> /home/mmmmecho `date``su - oracle -c dbshut` >> /home/mmmmecho $? >> /home/mmmmexit 0;;*)echo "not support"exit 1;;esac[root@node1 ~]# chmod a+x /etc/init.d/oracle[root@node1 ~]# vim /oracle/product/11.2.0/dbhome_1/bin/dbstart80行改变ORACLE_HOME_LISTNER=$ORACLE_HOME[root@node1 ~]# vim /oracle/product/11.2.0/dbhome_1/bin/dbshut50行改变ORACLE_HOME_LISTNER=$ORACLE_HOME[root@node1 ~]# chkconfig oracle/etc/oratab N改为Y至此oracle数据库在node1和node2上安装完毕九、corosync配置crm configure property stonith-enabled=falsecrm configure property no-quorum-policy=ignorecrm configure primitive mydrbd ocf:linbit:drbd params drbd_resource="web" op monitor role="Master"interval="30s" op monitor role="Slave" interval="31s" op start timeout="240s" op stop timeout="100s"crm configure ms MS_mydrbd mydrbd meta master-max=1 master-node-max=1 clone-max=2clone-node-max=1 notify="true"crm configure primitive MyFS ocf:heartbeat:Filesystem params device="/dev/drbd0"directory="/mydata" fstype="ext4" op start timeout="60s" op stop timeout="60s"crm configure primitive vip ocf:heartbeat:IPaddr params ip=10.10.10.80crm configure primitive oracleserver lsb:oracle op start timeout=”200s” op stop timeout=”100s”crm configure colocation MyFS_with_oracledrbd inf: MyFS MS_mydrbd:Master vipcrm configure order MyFS_after_oracledrbd inf: MS_mydrbd:promote MyFS:startcrm configure order vip_after_MyFS mandatory: MyFS vipcrm configure colocation oracleserver_with_vip inf: oracleserver vipcrm configure order oracleserver_after_vip mandatory: vip oracleserver十、mysql 加入corosync1.mysql 安装（node1 node2 同做）[root@node1 ~]# yum install mysql-server mysql-devel2.修改mysql配置文件（node1 node2 同做）[root@node1 ~]# cat /etc/f[mysqld]datadir=/mydata/datasocket=/var/lib/mysql/mysql.sockuser=mysqlsymbolic-links=0[mysqld_safe]log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pid3.创建mysql数据目录在主节点上[root@node1 ~]# mkdir /mydata/data4.配置mysql加入corosynccrm configure primitive mysqlserver lsb:mysqld op start timeout=”200s” op stop timeout=”100s”crm configure colocation mysqlserver_with_vip inf: mysqlserver vipcrm configure order mysqlserver_after_oracleserver mandatory: oracleserver mysqlserver十一、后期问题处理执行drbd-overview后出现stdandAlone则出现脑裂备节点执行drbdadm –discard-my-data connect web主节点执行drbdadm connect web这样再查看时就会发现脑裂消失参考：《corosync+pacemaker+drbd实现mysql的高可用》/ylqmf/archive/2012/04/16/2451211.html。