189 8069 5689
下文给大家带来实现keepalived双主模型lvs高可用集群的实验总结，希望能够给大家在实际运用中带来一定的帮助，负载均衡涉及的东西比较多，理论也不多，网上有很多书籍，今天我们就用创新互联在行业内累计的经验来做一个解答。
实现keepalived双主模型lvs高可用集群
一、拓扑图
二、环境准备
两台负载均衡调度器，两台web云服务器。
调度器A环境:
VS:一张网卡
DIP:192.168.0.7/24
VIP（主）：192.168.0.200
VIP(备)：192.168.0.201
软件包：yum install -y keepalived ipvsadm nginx(作用：sorry-server服务)
调度器B环境：
VS:一张网卡
DIP:192.168.0.8/24
VIP（主）：192.168.0.201
VIP(备)：192.168.0.200
软件包：yum install -y keepalived ipvsadm nginx(作用：sorry-server服务)
web服务器A环境：
RS1:192.168.0.3
lo:0:192.168.0.200
lo:1:192.168.0.201
软件包：yum install -y httpd
web服务器B环境：
RS1:192.168.0.4
lo:0:192.168.0.200
lo:1:192.168.0.201
软件包：yum install -y httpd
三、配置
调度器A配置
主要操作：ip配置，时间同步、hosts文件更改，修改主机名（确保主从节点能够通信）、keepalived.conf配置文件更改，通知脚本配置(两个脚本)，ipvsadm规则查看
ip配置
[root@node1keepalived]# ip a 1: lo:时间同步
ntpdate ntp-server-ip #时间服务器的iphosts文件
[root@node1 keepalived]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 node1 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 node1 192.168.0.8 node2 192.168.0.7 node1主机名修改
hostnamectl set-hostname node1keepalived.conf配置文件更改
[root@localhost keepalived]# cat keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 10 router_id node1 vrrp_mcast_group4 224.0.0.223 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 110 advert_int 1 authentication { auth_type PASS auth_pass JzvnmfkY } virtual_ipaddress { 192.168.0.200/24 dev ens33 label ens33:1 } notify_master "/etc/keepalived/notify1.sh master" notify_backup "/etc/keepalived/notify1.sh backup" notify_fault "/etc/keepalived/notify1.sh fault" } virtual_server 192.168.0.200 80 { delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.0.3 80 { weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.4 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } vrrp_instance VI_2 { state BACKUP interface ens33 virtual_router_id 52 priority 90 advert_int 1 authentication { auth_type PASS auth_pass JzvnmfkY } virtual_ipaddress { 192.168.0.201/24 dev ens33 label ens33:2 } notify_master "/etc/keepalived/notify2.sh master" notify_backup "/etc/keepalived/notify2.sh backup" notify_fault "/etc/keepalived/notify2.sh fault" } virtual_server 192.168.0.201 80 { delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.0.3 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.4 80 { weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } }通知脚本配置
脚本1
[root@node1 keepalived]# cat notify1.sh #!/bin/bash #description: An example of notify script # vip=192.168.0.200 contact='root@localhost' notify(){ local mailsubject="$(hostname) to be $1:$vip floating" local mailbody="$(date +'%F %T'):vrrp1 transition,$(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage:$(basename $0) {master|backup|fault}" exit 1 ;; esac脚本2
[root@node1 keepalived]# cat notify2.sh #!/bin/bash #description: An example of notify script # vip=192.168.0.201 contact='root@localhost' notify(){ local mailsubject="$(hostname) to be $1:$vip floating" local mailbody="$(date +'%F %T'):vrrp2 transition,$(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage:$(basename $0) {master|backup|fault}" exit 1 ;; esacipvsadm规则查看
[root@node1 keepalived]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.200:80 wrr -> 192.168.0.3:80 Route 2 0 0 -> 192.168.0.4:80 Route 1 0 0 TCP 192.168.0.201:80 wrr -> 192.168.0.3:80 Route 1 0 0 -> 192.168.0.4:80 Route 2 0 0调度器B配置
主要操作：ip配置，时间同步、hosts文件更改，修改主机名（确保主从节点能够通信）、keepalived.conf配置文件更改，通知脚本配置(两个脚本)，ipvsadm规则查看
ip配置
[root@node2 keepalived]# ip a 1: lo:时间同步
ntpdate ntp-server-iphosts文件更改
[root@node2 keepalived]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 node2 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 node2 192.168.0.7 node1 192.168.0.8 node2主机名修改
hostnamectl set-hostname node2keepalived.conf配置文件更改
[root@localhost keepalived]# cat keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 10 router_id node2 vrrp_mcast_group4 224.0.0.223 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass JzvnmfkY } virtual_ipaddress { 192.168.0.200/24 dev ens33 label ens33:1 } notify_master "/etc/keepalived/notify1.sh master" notify_backup "/etc/keepalived/notify1.sh backup" notify_fault "/etc/keepalived/notify1.sh fault" } virtual_server 192.168.0.200 80 { delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.0.3 80 { weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.4 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } vrrp_instance VI_2 { state MASTER interface ens33 virtual_router_id 52 priority 100 advert_int 1 authentication { auth_type PASS auth_pass JzvnmfkY } virtual_ipaddress { 192.168.0.201/24 dev ens33 label ens33:2 } notify_master "/etc/keepalived/notify2.sh master" notify_backup "/etc/keepalived/notify2.sh backup" notify_fault "/etc/keepalived/notify2.sh fault" } virtual_server 192.168.0.201 80 { delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.0.3 80 { weight 1 HTTP_GET { url { path / status_code 200 } status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.4 80 { weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } }通知脚本配置(同调度器A的脚本一致)
脚本1
[root@node2 keepalived]# cat notify1.sh #!/bin/bash #description: An example of notify script # vip=192.168.0.200 contact='root@localhost' notify(){ local mailsubject="$(hostname) to be $1:$vip floating" local mailbody="$(date +'%F %T'):vrrp1 transition,$(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage:$(basename $0) {master|backup|fault}" exit 1 ;; esac脚本2
[root@node2 keepalived]# cat notify2.sh #!/bin/bash #description: An example of notify script # vip=192.168.0.201 contact='root@localhost' notify(){ local mailsubject="$(hostname) to be $1:$vip floating" local mailbody="$(date +'%F %T'):vrrp2 transition,$(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage:$(basename $0) {master|backup|fault}" exit 1 ;; esacweb服务器A配置
ip配置
1: lo:lvs-dr-预配置脚本
脚本1
[root@localhost ~]# cat vip200-setrs.sh #!/bin/bash # vip=192.168.0.200 mask='255.255.255.255' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig lo:0 $vip netmask $mask broadcast $vip up route add -host $vip dev lo:0 ;; stop) ifconfig lo:0 down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esac脚本2
[root@localhost ~]# cat vip201-setrs.sh #!/bin/bash # vip=192.168.0.201 mask='255.255.255.255' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig lo:1 $vip netmask $mask broadcast $vip up route add -host $vip dev lo:1 ;; stop) ifconfig lo:1 down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esacweb服务器B配置
ip配置
[root@localhost ~]# ip a 1: lo:lvs-dr-预设脚本
脚本内容同web服务A相同，拷贝过去即可
vip200-setrs.sh vip201-setrs.sh
客户端测试
ip配置
[root@localhost ~]# ip a
1: lo:curl测试
[root@localhost ~]# for i in {1..10};do curl http://192.168.0.200;done backend server1 backend server1 backend server2 backend server1 backend server1 backend server2 backend server1 backend server1 backend server2 backend server1 [root@localhost ~]# for i in {1..10};do curl http://192.168.0.201;done backend server1 backend server2 backend server2 backend server1 backend server2 backend server2 backend server1 backend server2 backend server2 backend server1四、健康状态查看以及故障排查参考方式
方式1：查看keepalived的切换过程
systemctl status keepalived
方式2：mail命令查看邮件
方式3：查看web服务器日志
tail /var/log/httpd/access_log
tail /var/log/httpd/error_log
五、模拟故障
1.模拟调度器B故障
调度器B上操作
停止keepalived服务
systemctl stop keepalived [root@node2 keepalived]# systemctl status keepalived #分析以下信息，VIP2已移除 Jul 02 10:18:53 node2 systemd[1]: Stopping LVS and VRRP High Availability Monitor... Jul 02 10:18:53 node2 Keepalived_vrrp[50773]: VRRP_Instance(VI_2) sending 0 priority Jul 02 10:18:53 node2 Keepalived_vrrp[50773]: VRRP_Instance(VI_2) removing protocol VIPs. Jul 02 10:18:53 node2 systemd[1]: Stopped LVS and VRRP High Availability Monitor.查看ip
[root@node2 keepalived]# ip a #VIP2移除 1: lo:调度器A操作
[root@node1 keepalived]# systemctl status keepalived #分析以下信息，可知VIP漂移到调度A ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2017-07-02 10:07:36 CST; 17min ago Process: 20019 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 20020 (keepalived) CGroup: /system.slice/keepalived.service ├─20020 /usr/sbin/keepalived -D ├─20021 /usr/sbin/keepalived -D └─20022 /usr/sbin/keepalived -D Jul 02 10:07:37 node1 Keepalived_vrrp[20022]: Opening script file /etc/keepalived/notify1.sh Jul 02 10:07:37 node1 Keepalived_healthcheckers[20021]: Netlink reflector reports IP 192.168.0.200 added Jul 02 10:07:42 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_1) Sending gratuitous ARPs on ens33 for 192.168.0.200 Jul 02 10:18:54 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_2) Transition to MASTER STATE Jul 02 10:18:55 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_2) Entering MASTER STATE Jul 02 10:18:55 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_2) setting protocol VIPs. Jul 02 10:18:55 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_2) Sending gratuitous ARPs on ens33 for 192.168.0.201 Jul 02 10:18:55 node1 Keepalived_vrrp[20022]: Opening script file /etc/keepalived/notify2.sh Jul 02 10:18:55 node1 Keepalived_healthcheckers[20021]: Netlink reflector reports IP 192.168.0.201 added Jul 02 10:19:00 node1 Keepalived_vrrp[20022]: VRRP_Instance(VI_2) Sending gratuitous ARPs on ens33 for 192.168.0.201查看邮件
[root@node1 keepalived]# mail #node1成为VIP192.168.0.201的主节点 Heirloom Mail version 12.5 7/5/10. Type ? for help. "/var/spool/mail/root": 1 message > 1 root Sun Jul 2 10:18 19/702 "node1 to be master:192.168.0.201 floating" & 1 Message 1: From root@node1.localdomain Sun Jul 2 10:18:55 2017 Return-Path:2.模拟web服务器A故障(RS1)
webA上操作
[root@localhost ~]# systemctl stop httpd调度器A上操作
查看邮件
[root@node1 keepalived]# mail #分析以下信息，可知RS1宕机 Heirloom Mail version 12.5 7/5/10. Type ? for help. "/var/spool/mail/root": 4 messages 2 new 1 root Sun Jul 2 10:18 19/702 "node1 to be master:192.168.0.201 floating" 2 root Sun Jul 2 10:28 19/702 "node1 to be backup:192.168.0.201 floating" >N 3 keepalived@localhost Sun Jul 2 10:28 17/642 "[node1] Realserver [192.168.0.3]:80 - DOWN" N 4 keepalived@localhost Sun Jul 2 10:28 17/642 "[node1] Realserver [192.168.0.3]:80 - DOWN"客户端请求查看
[root@localhost ~]# curl http://192.168.0.200 #只能访问到RS2 backend server2调度器B上操作
[root@node2 keepalived]# mail #分析以下信息，可知RS1宕机 Heirloom Mail version 12.5 7/5/10. Type ? for help. "/var/spool/mail/root": 4 messages 2 new 1 root Sun Jul 2 10:28 19/702 "node2 to be backup:192.168.0.200 floating" 2 root Sun Jul 2 10:28 19/702 "node2 to be master:192.168.0.201 floating" >N 3 keepalived@localhost Sun Jul 2 10:28 17/642 "[node2] Realserver [192.168.0.3]:80 - DOWN" N 4 keepalived@localhost Sun Jul 2 10:28 17/642 "[node2] Realserver [192.168.0.3]:80 - DOWN"3.模拟web服务双机故障
RS1和RS2停止web服务
调度器A上操作
查看邮件 #分析以下信息，可知双机宕机
[root@node1 keepalived]# mail Heirloom Mail version 12.5 7/5/10. Type ? for help. "/var/spool/mail/root": 6 messages 2 new 1 root Sun Jul 2 10:18 19/702 "node1 to be master:192.168.0.201 floating" 2 root Sun Jul 2 10:28 19/702 "node1 to be backup:192.168.0.201 floating" 3 keepalived@localhost Sun Jul 2 10:28 18/653 "[node1] Realserver [192.168.0.3]:80 - DOWN" 4 keepalived@localhost Sun Jul 2 10:28 18/653 "[node1] Realserver [192.168.0.3]:80 - DOWN" >N 5 keepalived@localhost Sun Jul 2 10:36 17/642 "[node1] Realserver [192.168.0.4]:80 - DOWN" N 6 keepalived@localhost Sun Jul 2 10:36 17/642 "[node1] Realserver [192.168.0.4]:80 - DOWN"查看keepalived状态
[root@node1 keepalived]# systemctl status keepalived #分析以下信息，双机宕机，启动sorry服务器 ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2017-07-02 10:07:36 CST; 30min ago Process: 20019 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 20020 (keepalived) CGroup: /system.slice/keepalived.service ├─20020 /usr/sbin/keepalived -D ├─20021 /usr/sbin/keepalived -D └─20022 /usr/sbin/keepalived -D Jul 02 10:36:31 node1 Keepalived_healthcheckers[20021]: Removing alive servers from the pool for VS [192.168.0.200]:80 Jul 02 10:36:31 node1 Keepalived_healthcheckers[20021]: Remote SMTP server [127.0.0.1]:25 connected. Jul 02 10:36:31 node1 Keepalived_healthcheckers[20021]: SMTP alert successfully sent. Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Error connecting server [192.168.0.4]:80. Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Removing service [192.168.0.4]:80 from VS [192.168.0.201]:80 Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Lost quorum 1-0=1 > 0 for VS [192.168.0.201]:80 Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Adding sorry server [127.0.0.1]:80 to VS [192.168.0.201]:80 Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Removing alive servers from the pool for VS [192.168.0.201]:80 Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: Remote SMTP server [127.0.0.1]:25 connected. Jul 02 10:36:32 node1 Keepalived_healthcheckers[20021]: SMTP alert successfully sent.客户端请求
[root@localhost ~]# curl http://192.168.0.201 #请求至sorry服务器的web页面 sorry server [root@localhost ~]# curl http://192.168.0.200 sorry server看了以上关于实现keepalived双主模型lvs高可用集群的实验总结，如果大家还有什么地方需要了解的可以在创新互联行业资讯里查找自己感兴趣的或者找我们的专业技术工程师解答的，创新互联技术工程师在行业内拥有十几年的经验了。
另外有需要云服务器可以了解下创新互联cdcxhl.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。