MENU

人大金仓RWC读写分离集群-高可用测试&案例分析

• July 10, 2026 • Read: 13 • 编码👨🏻‍💻

一、Kingbase RWC 高可用集群切机原理

1.1 原理图

image-20260710110949550

1.2 流程说明

  • 本地db状态检测：hamgrd先判断本地db状态，故障则休眠，正常则进入监控流程。
  • 主机监控流程：主机hamgrd检测备机状态，发现备机故障则远程恢复流程。
  • 备机监控流程：当hamgrd 判断自身是备机时，会检测上流节点状态。若上流节点故障且为主机，便进入切换流程。
  • 备机先kil本机wal_receiver进程，随后开启备机选举。

  • 本机节点选举成功后，ping信任网关的ip：

    • 若ping 失败，判定网络不正常，退出切换升主流程。
    • 若ping成功，判定网络正常，每隔1秒检测其他备机是否kill掉wal_receiver进程。
    • 若检测时间超sibling_nodes_disconnect_timeout设置的时长，默认所有备机均已成功kill该进程。

  • 判断是否配置vip。

    • 若已配置vip，ping vip:
    • 若ping不通，直接在新主机加载vip。

  • 若ping通，判断vip是否在新主机上：

    • 若在新主机上，直接执行升主操作。
    • 若不在新主机上，尝试远程执行卸载vip命令，每次执行耗时不得超vip_timeout(60)秒。若超时vip仍能ping通，退出切机升主操作，报错卸载vip超时；若ping不通，则在新主机加载vip，之后继续执行真正的切换命令，完成本机升主。

一句话总结：

备机的hamgrd 时刻监控着自己和上流节点，检测到上流节点故障，并且上流节点是主机时就执行切换，切换前确认所有备机都 kill掉了wal_receiver进程，然后通过ping信任网关判断即将升主的备机网络是否正常，切换过程中通过是否已配置vip，执行卸载old主机vip，加载new主机vip操作。

二、实验目的

目前KES环境为一主一备读写分离集群，两台机器均在一个生产中心。通过对数据库服务、服务器、网络等典型故障进行模拟，主要测试集群环境的高可用性，保障迁移切换后，可稳定承载业务数据库。

2.1 实验环境

类别主机点备节点
IP172.24.0.156172.24.0.157
主机名node1node2
操作系统麒麟 V10 SP3麒麟 V10 SP3
虚拟 VIP 地址172.24.0.155

2.2 实验参数调整

  • 每个节点[node1,node2]都需要修改高可用参数，以便于快速跟踪相关过程。这里以node1节点为例。生产环境修改参数请以实际为准。
#控制连接失败或ping网关失败后的最多重试次数。
sed -i "s/reconnect_attempts=10/reconnect_attempts=3/g" /home/application/KingbaseES/cluster/kingbase/etc/repmgr.conf

#控制每次重试之间的间隔等待时间（秒）。
sed -i "s/reconnect_interval=6/reconnect_interval=5/g" /home/application/KingbaseES/cluster/kingbase/etc/repmgr.conf
  • 重启集群，查看集群当前的状态
[kingbase@nodel ~]s sys_monitor. sh restart

[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 13  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 12  | 0 bytes

[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 3859 | no      | 0 second(s) ago
  • 登陆主库，查看当前有哪些备库通过流复制连过来了、复制状态如何。
[kingbase@node1 ~]$ ksql -h 172.24.0.156 test system
用户 system 的口令：
授权类型: 企业版.
输入 "help" 来获取帮助信息.

test=# 
test=# \x
扩展显示已打开.
test=# select * from sys_stat_replication;
-[ RECORD 1 ]----+---------------------------
pid              | 6130
usesysid         | 16386
usename          | esrep
application_name | node2
client_addr      | 172.24.0.157
client_hostname  | 
client_port      | 14760
backend_start    | 2026-07-09 15:16:41.134751
backend_xmin     | 
state            | streaming
sent_lsn         | 0/2A0017A0
write_lsn        | 0/2A0017A0
flush_lsn        | 0/2A0017A0
replay_lsn       | 0/2A0017A0
write_lag        | 
flush_lag        | 
replay_lag       | 
sync_priority    | 1
sync_state       | quorum
reply_time       | 2026-07-09 15:33:08.771341
  • 登陆备库，查看备库的 WAL 接收器状态，连的是哪个主库、接收 WAL 的进度如何。
[kingbase@node1 ~]$ ksql -h 172.24.0.157 test system
用户 system 的口令：
授权类型: 企业版.
输入 "help" 来获取帮助信息.

test=# \x
扩展显示已打开.
test=# select * from sys_stat_wal_receiver;
-[ RECORD 1 ]---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
pid                   | 6894
status                | streaming
receive_start_lsn     | 0/2A000000
receive_start_tli     | 13
received_lsn          | 0/2A0017A0
received_tli          | 13
last_msg_send_time    | 2026-07-09 15:44:42.434904
last_msg_receipt_time | 2026-07-09 15:44:42.433877
latest_end_lsn        | 0/2A0017A0
latest_end_time       | 2026-07-09 15:16:54.870799
slot_name             | repmgr_slot_2
sender_host           | 172.24.0.156
sender_port           | 54321
conninfo              | user=esrep connect_timeout=10 host=172.24.0.156 port=54321 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000 application_name=node2

三、模拟数据库服务故障

3.1 备机数据库服务故障

测试用例高可用
测试内容正常关闭备库实例，查看数据库切换时间，查询业务恢复时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机服务能够自动拉起，集群状态不发生变化，不影响应用访问。
测试过程
  • 查看集群状态node1为主，node2为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 13  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 12  | 0 bytes
  • 正常关闭备库[node2]实例。
[kingbase@node2 ~]$ sys_ctl stop
等待服务器进程关闭 .... 完成
服务器进程已经关闭
  • 查看集群事件

2026-07-09 16:14:36 node2 节点无法连接；2026-07-09 16:15:03 node2 节点被自动拉起恢复

[kingbase@node1 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 1       | node1 | child_node_reconnect      | t  | 2026-07-09 16:15:05 | standby node "node2" (ID: 2) has reconnected after 30 seconds                         
 2       | node2 | standby_recovery          | t  | 2026-07-09 16:15:04 | reconnected to local node "node2" (ID: 2), marking active                             
 1       | node1 | node_recovery_success     | t  | 2026-07-09 16:15:03 | node "node2" (ID: 2) auto-recovery success                                            
 2       | node2 | standby_failure           | t  | 2026-07-09 16:14:36 | unable to connect to local node "node2" (ID: 2), marking inactive                     
 1       | node1 | child_node_disconnect     | t  | 2026-07-09 16:14:35 | standby node "node2" (ID: 2) has disconnected
  • 查看集群状态，关闭后备机服务会再次被自动拉起，集群状态不发生改变。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 13  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 13  | 0 bytes

3.2 主机数据库服务故障

测试用例高可用
测试内容正常关闭主库实例，查看数据库切换时间，查询业务恢复时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机升级成主机对外提供服务，集群状态发生改变，旧主加入集群需要人工干预。对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。
测试过程
  • 查看集群状态node1为主，node2为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 13  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 12  | 0 bytes
  • 正常关闭主库[node1]实例。
[kingbase@node1 ~]$  sys_ctl stop
等待服务器进程关闭 .... 完成
服务器进程已经关闭
  • 此时备机会自动升级成主机，集群状态发生改变。
  • 操作在 node2 节点上进行
[kingbase@node2 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | - failed  | ?        | default  | 100   |     |         
 2  | node2 | primary | * running |          | default  | 100   | 14  |         

[WARNING] following issues were detected
  - unable to connect to node "node1" (ID: 1)

[HINT] execute with --verbose option to see connection error messages



[kingbase@node2 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | - failed  | ?        | n/a     | n/a  | n/a     | n/a                
 2  | node2 | primary | * running |          | running | 3859 | no      | n/a                

[WARNING] following issues were detected
  - unable to  connect to node "node1" (ID: 1)

[HINT] execute with --verbose option to see connection error messages
  • 查看集群事件

2026-07-09 16:27:16 node2 备机被提升为主机

[kingbase@node2 ~]$  repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 2       | node2 | repmgrd_reload            | t  | 2026-07-09 16:27:21 | monitoring cluster primary "node2" (ID: 2)                                            
 2       | node2 | repmgrd_failover_promote  | t  | 2026-07-09 16:27:21 | node "node2" (ID: 2) promoted to primary; old primary "node1" (ID: 1) marked as failed
 2       | node2 | standby_promote           | t  | 2026-07-09 16:27:16 | server "node2" (ID: 2) was successfully promoted to primary
  • 应用层检查业务是否正常。

对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。

  • 原主机数据库恢复，需要人工干预加入到集群中。

⚠️： 登陆node1 服务器上操作

#重新加入到集群中
#-h 172.24.0.157 \   # 目标主库地址（rejoin 到谁那里，当前环境下node2 为主库地址）
#-U esrep \          # 复制连接用户
#-d esrep            # 连接的数据库名

[kingbase@node1 ~]$ repmgr node rejoin \
  -h 172.24.0.157 \
  -U esrep \
  -d esrep

[NOTICE] rejoin target is node "node2" (ID: 2)
[INFO] local node 1 can attach to rejoin target node 2
[DETAIL] local node's recovery point: 0/2B000028; rejoin target node's fork point: 0/2B0000A0
[INFO] creating replication slot as user "esrep"
[NOTICE] setting node 1's upstream to node 2
[WARNING] unable to ping "host=172.24.0.156 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000"
[DETAIL] KCIping() returned "KCIPING_NO_RESPONSE"
[NOTICE] begin to start server at 2026-07-09 16:37:39.011388
[NOTICE] starting server using "/home/application/KingbaseES/cluster/kingbase/bin/sys_ctl  -w -t 90 -D '/home/application/KingbaseES/data' -l /home/application/KingbaseES/cluster/kingbase/bin/logfile start"
[NOTICE] start server finish at 2026-07-09 16:37:39.218932
[NOTICE] replication slot "repmgr_slot_2" deleted on node 1
[NOTICE] NODE REJOIN successful
[DETAIL] node 1 is now attached to node 2
  • 待集群恢复至正常状态后，查看集群状态node2为主，node1为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | standby |   running | node2    | default  | 100   | 13  | 0 bytes 
 2  | node2 | primary | * running |          | default  | 100   | 14  |         
 
 
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | standby |   running | node2    | running | 5587 | no      | 1 second(s) ago    
 2  | node2 | primary | * running |          | running | 3859 | no      | n/a
  • 应用层再次检查业务是否正常

四、模拟服务器故障

4.1 备机数据库服务器故障

测试用例高可用
测试内容重启备库服务器，查看数据库切换时间，查询业务恢复时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机服务能够自动拉起，集群状态不发生变化，不影响应用访问。
测试过程
  • 查看集群状态node2为主，node1为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | standby |   running | node2    | default  | 100   | 13  | 0 bytes 
 2  | node2 | primary | * running |          | default  | 100   | 14  |         
 
 
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | standby |   running | node2    | running | 5587 | no      | 1 second(s) ago    
 2  | node2 | primary | * running |          | running | 3859 | no      | n/a
  • 正常关闭备库[node1]实例。
[kingbase@node1 ~]$ sys_ctl stop
等待服务器进程关闭 .... 完成
服务器进程已经关闭
  • 查看集群事件

2026-07-10 09:35:27 node1 节点无法连接；2026-07-10 09:35:55 node1 节点被自动拉起恢复

[kingbase@node2 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 2       | node2 | child_node_reconnect      | t  | 2026-07-10 09:35:57 | standby node "node1" (ID: 1) has reconnected after 30 seconds                         
 1       | node1 | standby_recovery          | t  | 2026-07-10 09:35:55 | reconnected to local node "node1" (ID: 1), marking active                             
 2       | node2 | node_recovery_success     | t  | 2026-07-10 09:35:55 | node "node1" (ID: 1) auto-recovery success                                            
 2       | node2 | child_node_disconnect     | t  | 2026-07-10 09:35:27 | standby node "node1" (ID: 1) has disconnected                                         
 1       | node1 | standby_failure           | t  | 2026-07-10 09:35:27 | unable to connect to local node "node1" (ID: 1), marking inactive
  • 查看集群状态，关闭后备机服务会再次被自动拉起，集群状态不发生改变。
[kingbase@node2 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | standby |   running | node2    | default  | 100   | 14  | 0 bytes 
 2  | node2 | primary | * running |          | default  | 100   | 14  |

4.2 主机数据库服务器故障

测试用例高可用
测试内容重启主库服务器，查看数据库切换时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机升级成主机对外提供服务，集群状态发生改变，旧主加入集群需要人工干预。对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。
测试过程
  • 查看集群状态node2为主，node1为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | standby |   running | node2    | default  | 100   | 13  | 0 bytes 
 2  | node2 | primary | * running |          | default  | 100   | 14  |         
 
 
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | standby |   running | node2    | running | 5587 | no      | 1 second(s) ago    
 2  | node2 | primary | * running |          | running | 3859 | no      | n/a
  • 重启主数据库服务器。
[kingbase@node2 ~]$ su - root
[root@node2 ~]# reboot
  • 此时备机会自动升级成主机，集群状态发生改变。
  • 操作在 node1 节点上进行
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 15  |         
 2  | node2 | primary | - failed  | ?        | default  | 100   |     |         

[WARNING] following issues were detected
  - unable to connect to node "node2" (ID: 2)

[HINT] execute with --verbose option to see connection error messages


[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | primary | - failed  | ?        | n/a     | n/a  | n/a     | n/a                

[WARNING] following issues were detected
  - unable to  connect to node "node2" (ID: 2)

[HINT] execute with --verbose option to see connection error messages
  • 查看集群事件

2026-07-10 09:43:57 node1 备机被提升为主机

[kingbase@node1 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 1       | node1 | repmgrd_reload            | t  | 2026-07-10 09:44:02 | monitoring cluster primary "node1" (ID: 1)                                            
 1       | node1 | repmgrd_failover_promote  | t  | 2026-07-10 09:44:02 | node "node1" (ID: 1) promoted to primary; old primary "node2" (ID: 2) marked as failed
 1       | node1 | standby_promote           | t  | 2026-07-10 09:43:57 | server "node1" (ID: 1) was successfully promoted to primary
  • 应用层检查业务是否正常。

对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。

  • 当前配置下，原主机[node2]需要人工干预才能加入集群。
  • 操作在 node2 节点上进行
#备份 wal日志
[kingbase@node2 ~]$ cp -r $KINGBASE_DATA/sys_wal /tmp/sys_wal_bak_$(date +%Y%m%d_%H%M)


#关闭 kbha 守护进程 and repmgrd 守护进程，关闭数据库服务
[kingbase@node2 ~]$ sys_monitor.sh stoplocal

2026-07-10 09:52:43 Ready to stop local kbha daemon and repmgrd daemon ...
2026-07-10 09:52:47 begin to stop repmgrd on "[localhost]".
2026-07-10 09:52:48 repmgrd on "[localhost]" already stopped.
2026-07-10 09:52:48 Done.
2026-07-10 09:52:48 begin to stop DB on "[localhost]".
sys_ctl: PID file "/home/application/KingbaseES/data/kingbase.pid"              does not exist
Is server running?
2026-07-10 09:52:49 DB on "[localhost]" stop success.


#重新加入到集群中
#-h 172.24.0.156 \   # 目标主库地址（rejoin 到谁那里，当前环境下node1 为主库地址）
#-U esrep \          # 复制连接用户
#-d esrep            # 连接的数据库名
#--force-rewind      # 强制回退旧主的分叉数据，哪怕旧主有独有写入也抹掉重来，只为让它能重新当备库。

[kingbase@node2 ~]$ repmgr node rejoin \
  -h 172.24.0.156 \
  -U esrep \
  -d esrep \
  --force-rewind
  

[NOTICE] rejoin target is node "node1" (ID: 1)
[NOTICE] sys_rewind execution required for this node to attach to rejoin target node 1
[DETAIL] rejoin target server's timeline 15 forked off current database system timeline 14 before current recovery point 0/2C000028
[NOTICE] executing sys_rewind
[DETAIL] sys_rewind command is "/home/application/KingbaseES/cluster/kingbase/bin/sys_rewind -D '/home/application/KingbaseES/data' --source-server='host=172.24.0.156 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000' -X stream"
sys_rewind: servers diverged at WAL location 0/2B002DA0 on timeline 14
sys_rewind: rewinding from last common checkpoint at 0/2B001EA8(redo lsn at 0/2B001E78) on timeline 14
sys_rewind: find last common checkpoint start time from 2026-07-10 09:58:03.286209 CST to 2026-07-10 09:58:03.400675 CST, in "0.114466" seconds.
sys_rewind: collect the number of WAL files to be processed:2, start time from 2026-07-10 09:58:03.400951 CST to 2026-07-10 09:58:03.533380 CST, cost "0.132429" seconds.
sys_rewind: read WAL in target server from 0/2B001EA8 to 0/2C0000A0 (endpoint 0/2C0000A0)
sys_rewind: read the local Wal file information, start time from 2026-07-10 09:58:03.400951 CST to 2026-07-10 09:58:03.534516 CST, cost "0.001112" seconds.
sys_rewind: waiting for background process to finish streaming ...
sys_rewind: file replication start time from 2026-07-10 09:58:03.534543 CST to 2026-07-10 09:58:04.685155 CST, cost "1.150612" seconds.
sys_rewind: update the control file: minRecoveryPoint is '0/2B014528', minRecoveryPointTLI is '15', and database state is 'in archive recovery'
sys_rewind: we will remove the dir '/home/application/KingbaseES/data/sys_replslot/repmgr_slot_1.rewind' and all the file/dir in it.
sys_rewind: rewind start wal location 0/2B001E78 (file 0000000E000000000000002B), end wal location 0/2B014528 (file 0000000F000000000000002B). wal data increment:73(kB). time from 2026-07-10 09:58:03.534543 CST to 2026-07-10 09:58:05.059098 CST, in "1.772889" seconds.
sys_rewind: Done!
[NOTICE] 0 files copied to /home/application/KingbaseES/data
[INFO] creating replication slot as user "esrep"
[NOTICE] setting node 2's upstream to node 1
[WARNING] unable to ping "host=172.24.0.157 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000"
[DETAIL] KCIping() returned "KCIPING_NO_RESPONSE"
[NOTICE] begin to start server at 2026-07-10 09:58:05.065516
[NOTICE] starting server using "/home/application/KingbaseES/cluster/kingbase/bin/sys_ctl  -w -t 90 -D '/home/application/KingbaseES/data' -l /home/application/KingbaseES/cluster/kingbase/bin/logfile start"
[NOTICE] start server finish at 2026-07-10 09:58:06.821248
[NOTICE] NODE REJOIN successful
[DETAIL] node 2 is now attached to node 1
  • 待集群恢复至正常状态后，查看集群状态node1为主，node2为备节点。
[kingbase@node2 ~]$  repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 15  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 14  | 0 bytes 
 
 
[kingbase@node2 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 5200 | no      | 0 second(s) ago

ps: 如果遇到node2 的节点的repmgrd 状态是not running , 可使用sys_monitor.sh startlocal 启动repmgrd守护进程。

  • 应用层再次检查业务是否正常

五、模拟网络故障

5.1 备机数据库网络故障

测试用例高可用
测试内容断开备机网络，查看数据库切换时间，查询业务恢复时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机服务能够自动拉起，集群状态不发生变化，不影响应用访问。
测试过程
  • 查看集群状态node1为主，node2为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 15  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 14  | 0 bytes 


[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 5200 | no      | 0 second(s) ago
  • 将备机服务器网线down 掉，并检查集群状态。
  • 操作在 node2 节点上进行
#切换到 root 用户下
[kingbase@node2 ~]$ su - root

#查看网卡信息
[root@node2 ~]# ifconfig
ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.24.0.157  netmask 255.255.255.0  broadcast 172.24.0.255
        inet6 fe80::250:56ff:fea3:ad77  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:a3:ad:77  txqueuelen 1000  (Ethernet)
        RX packets 75204  bytes 42562539 (40.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15227  bytes 2376908 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 24092  bytes 5712359 (5.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24092  bytes 5712359 (5.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        
#临时down了ens32网卡
[root@node2 ~]# ip link set ens32 down
  • 查看集群事件

2026-07-10 10:16:00 node2 节点无法连接。

[kingbase@node1 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 1       | node1 | child_node_disconnect     | t  | 2026-07-10 10:16:00 | standby node "node2" (ID: 2) has disconnected                                         
 2       | node2 | repmgrd_start             | t  | 2026-07-10 10:02:47 | monitoring connection to upstream node "node1" (ID: 1)
  • 恢复网络连接
  • 操作在 node2 节点上进行
#切换到 root 用户下
[kingbase@node2 ~]$ su - root

[root@node2 ~]# ip link set ens32 up
[2163.5995921 T117] e10a: ens32 NIC Link is Up 100a Mbps Ful1 Duplex, Flow Control: None
[2163.6003561I T117】 IPv6: ADDRCONF(NETDEV_CHANGE): ens32: 1ink becomes ready
  • 查看集群事件

2026-07-10 10:19:49 node2 节点恢复连接。

[kingbase@node1 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 1       | node1 | child_node_reconnect      | t  | 2026-07-10 10:19:49 | standby node "node2" (ID: 2) has reconnected after 229 seconds                        
 2       | node2 | repmgrd_start             | t  | 2026-07-10 10:19:44 | monitoring connection to upstream node "node1" (ID: 1)                                
 1       | node1 | child_node_disconnect     | t  | 2026-07-10 10:16:00 | standby node "node2" (ID: 2) has disconnected                                         
 2       | node2 | repmgrd_start             | t  | 2026-07-10 10:02:47 | monitoring connection to upstream node "node1" (ID: 1)                                
 1       | node1 | child_node_new_connect    | t  | 2026-07-10 09:58:06 | new standby "node2" (ID: 2) has connected                                             
 2       | node2 | node_rejoin               | t  | 2026-07-10 09:58:06 | node 2 is now attached to node 1
  • 查看集群状态node1为主，node2为备节点，节点角色没有发生变化。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 15  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 14  | 0 bytes 
 

[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 8548 | no      | 1 second(s) ago

5.2 主机数据库网络故障

测试用例高可用
测试内容断开主节点网络，查看数据库切换时间，查看集群状态。
前置条件1、操作系统正常运行。2、数据库安装成功，且数据库服务已经启动。
预期结果备机升级成主机对外提供服务，集群状态发生改变，旧主加入集群需要人工干预。对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。
测试过程
  • 查看集群状态node1为主，node2为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 15  |         
 2  | node2 | standby |   running | node1    | default  | 100   | 14  | 0 bytes 
 
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 5587 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 8548 | no      | 1 second(s) ago
  • 将主机服务器网线down 掉，并检查集群状态。
  • 操作在 node1 节点上进行。
#切换到 root 用户下
[kingbase@node1 ~]$ su - root

#查看网卡信息
[root@node1 ~]# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.24.0.156  netmask 255.255.255.0  broadcast 172.24.0.255
        inet6 fe80::3917:6e9e:dc35:d5fc  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:a3:b6:03  txqueuelen 1000  (Ethernet)
        RX packets 2003440  bytes 398139781 (379.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 715078  bytes 183618860 (175.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2078478  bytes 482424595 (460.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2078478  bytes 482424595 (460.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        
#临时down了ens192网卡
[root@node1 ~]# ip link set ens192 down
  • 此时备机会自动升级成主机，集群状态发生改变。
  • 操作在 node2 节点上进行
[kingbase@node2 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | - failed  | ?        | default  | 100   |     |         
 2  | node2 | primary | * running |          | default  | 100   | 16  |         

[WARNING] following issues were detected
  - unable to connect to node "node1" (ID: 1)
[HINT] execute with --verbose option to see connection error messages


[kingbase@node2 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | - failed  | ?        | n/a     | n/a  | n/a     | n/a                
 2  | node2 | primary | * running |          | running | 8548 | no      | n/a                

[WARNING] following issues were detected
  - unable to  connect to node "node1" (ID: 1)

[HINT] execute with --verbose option to see connection error messages
  • 查看集群事件

2026-07-10 10:29:50 备机被提升为主机

[kingbase@node2 ~]$ repmgr cluster event
 Node ID | Name  | Event                     | OK | Timestamp           | Details                                                                               
---------+-------+---------------------------+----+---------------------+----------------------------------------------------------------------------------------
 2       | node2 | repmgrd_reload            | t  | 2026-07-10 10:29:50 | monitoring cluster primary "node2" (ID: 2)                                            
 2       | node2 | repmgrd_failover_promote  | t  | 2026-07-10 10:29:50 | node "node2" (ID: 2) promoted to primary; old primary "node1" (ID: 1) marked as failed
 2       | node2 | standby_promote           | t  | 2026-07-10 10:29:45 | server "node2" (ID: 2) was successfully promoted to primary
  • 应用层检查业务是否正常。

对于有重连机制的应用不需要重启，没有重连机制的应用需要重启。切机过程1-2分钟，不需要重启的应用会有1-2分钟的断连。

  • 恢复网络连接
  • 操作在 node1 节点上进行
#切换到 root 用户下
[kingbase@node1 ~]$ su - root

[root@node1 ~]# ip link set ens192 up
  • 当前配置下，原主机[node1]需要人工干预才能加入集群。
  • 操作在 node1节点上进行
#备份 wal日志
[kingbase@node1 ~]$ cp -r $KINGBASE_DATA/sys_wal /tmp/sys_wal_bak_$(date +%Y%m%d_%H%M)


#重新加入到集群中
#-h 172.24.0.157 \   # 目标主库地址（rejoin 到谁那里，当前环境下node2 为主库地址）
#-U esrep \          # 复制连接用户
#-d esrep            # 连接的数据库名
#--force-rewind      # 强制回退旧主的分叉数据，哪怕旧主有独有写入也抹掉重来，只为让它能重新当备库。

[kingbase@node1 ~]$ repmgr node rejoin \
>   -h 172.24.0.157 \
>   -U esrep \
>   -d esrep \
>   --force-rewind



[NOTICE] rejoin target is node "node2" (ID: 2)
[NOTICE] sys_rewind execution required for this node to attach to rejoin target node 2
[DETAIL] rejoin target server's timeline 16 forked off current database system timeline 15 before current recovery point 0/2C000028
[NOTICE] executing sys_rewind
[DETAIL] sys_rewind command is "/home/application/KingbaseES/cluster/kingbase/bin/sys_rewind -D '/home/application/KingbaseES/data' --source-server='host=172.24.0.157 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000' -X stream"
sys_rewind: servers diverged at WAL location 0/2B017688 on timeline 15
sys_rewind: rewinding from last common checkpoint at 0/2B017280(redo lsn at 0/2B017250) on timeline 15
sys_rewind: find last common checkpoint start time from 2026-07-10 10:47:20.885475 CST to 2026-07-10 10:47:21.024507 CST, in "0.139032" seconds.
sys_rewind: collect the number of WAL files to be processed:2, start time from 2026-07-10 10:47:21.024754 CST to 2026-07-10 10:47:21.102221 CST, cost "0.077467" seconds.
sys_rewind: read WAL in target server from 0/2B017280 to 0/2C0000A0 (endpoint 0/2C0000A0)
sys_rewind: read the local Wal file information, start time from 2026-07-10 10:47:21.024754 CST to 2026-07-10 10:47:21.103494 CST, cost "0.001235" seconds.
sys_rewind: waiting for background process to finish streaming ...
sys_rewind: file replication start time from 2026-07-10 10:47:21.103525 CST to 2026-07-10 10:47:22.464846 CST, cost "1.361321" seconds.
sys_rewind: update the control file: minRecoveryPoint is '0/2B028D30', minRecoveryPointTLI is '16', and database state is 'in archive recovery'
sys_rewind: we will remove the dir '/home/application/KingbaseES/data/sys_replslot/repmgr_slot_2.rewind' and all the file/dir in it.
sys_rewind: rewind start wal location 0/2B017250 (file 0000000F000000000000002B), end wal location 0/2B028D30 (file 00000010000000000000002B). wal data increment:70(kB). time from 2026-07-10 10:47:21.103525 CST to 2026-07-10 10:47:23.133801 CST, in "2.248326" seconds.
sys_rewind: Done!
[NOTICE] 0 files copied to /home/application/KingbaseES/data
[INFO] creating replication slot as user "esrep"
[NOTICE] setting node 1's upstream to node 2
[WARNING] unable to ping "host=172.24.0.156 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=2 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=9000"
[DETAIL] KCIping() returned "KCIPING_NO_RESPONSE"
[NOTICE] begin to start server at 2026-07-10 10:47:23.197980
[NOTICE] starting server using "/home/application/KingbaseES/cluster/kingbase/bin/sys_ctl  -w -t 90 -D '/home/application/KingbaseES/data' -l /home/application/KingbaseES/cluster/kingbase/bin/logfile start"
[NOTICE] start server finish at 2026-07-10 10:47:23.906334
[NOTICE] NODE REJOIN successful
[DETAIL] node 1 is now attached to node 2
  • 待集群恢复至正常状态后，查看集群状态node2为主，node1为备节点。
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | standby |   running | node2    | default  | 100   | 15  | 0 bytes 
 2  | node2 | primary | * running |          | default  | 100   | 16  |      


[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID    | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+--------+---------+--------------------
 1  | node1 | standby |   running | node2    | running | 179378 | no      | 0 second(s) ago    
 2  | node2 | primary | * running |          | running | 8548   | no      | n/a

ps: 如果遇到node2 的节点的repmgrd 状态是not running , 可使用sys_monitor.sh startlocal 启动repmgrd守护进程。

  • 应用层再次检查业务是否正常

六、其他案例

6.1 主备节点互换

提升node01 提升为primary 主节点，替换node2 为standby 备节点。
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | standby |   running | node2    | running | 4463 | no      | 0 second(s) ago    
 2  | node2 | primary | * running |          | running | 3982 | no      | n/a
  • 操作在 node1 节点上进行
[kingbase@node1 ~]$ repmgr standby switchover \
  -h 172.24.0.156 \
  -U esrep \
  -d esrep \
  --siblings-follow

参数说明：

  • standby switchover → 将备节点提升为主节点，并将原来的主节点降级为备节点。
  • -h 172.24.0.156 → 指向即将升主的 standby（node1）
  • --siblings-follow → 让其他 standby 跟随新主

它会自动完成：停 node1 恢复 → promote node1 成主 → node2 降级重连为 standby → 更新 repmgr 元数据。整个过程秒级，业务短暂中断几秒。

[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 7   |         
 2  | node2 | standby |   running | node1    | default  | 100   | 6   | 0 bytes
 
[kingbase@node1 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | * running |          | running | 4463 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 3982 | no      | 0 second(s) ago

6.2 双主脑裂

双主集群脑裂，两个 primary 共存的原因分析：

  • 集群健康状态下： node1 primary ，node2 standy。
  • 主机node1 宕机，此时node2备机会自动升级成主机，集群状态发生改变。node2 提升为了 primary
  • 原主机node1 在没有人工干预的情况下(rejoin)，直接启动数据库。 sys_ctl start
  • 发生了双主集群脑裂现象，两个 primary 共存
[kingbase@node2 ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID  | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+------+---------+--------------------
 1  | node1 | primary | ! running |          | running | 4463 | no      | n/a                
 2  | node2 | primary | * running |          | running | 3982 | no      | n/a                

[WARNING] following issues were detected
  - node "node1" (ID: 1) is running but the repmgr node record is inactive
  • 操作在 node1 节点上进行
#停旧主node1数据库
[kingbase@node1 ~]$ sys_ctl stop

#备份node1 数据
[kingbase@node1 ~]$ mv $KINGBASE_DATA ${KINGBASE_DATA}_bak_$(date +%Y%m%d_%H%M)

#node2(157) 克隆重建
[kingbase@node1 ~]$ repmgr standby clone -h 172.24.0.157 -U esrep -d esrep

#启动node1数据库
[kingbase@node1 ~]$ sys_ctl start

#注册到集群中
[kingbase@node1 ~]$ repmgr standby register --force
  • 查看集群状态
[kingbase@node1 ~]$ repmgr cluster show --compact
 ID | Name  | Role    | Status    | Upstream | Location | Prio. | TLI | LSN_Lag
----+-------+---------+-----------+----------+----------+-------+-----+---------
 1  | node1 | primary | * running |          | default  | 100   | 9   |         
 2  | node2 | standby |   running | node1    | default  | 100   | 9   | 0 bytes 
 
 
[kingbase@localhost ~]$ repmgr service status
 ID | Name  | Role    | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+-------+---------+--------------------
 1  | node1 | primary | * running |          | running | 14956 | no      | n/a                
 2  | node2 | standby |   running | node1    | running | 3982  | no      | 1 second(s) ago

七、参考

  • [1] 人大金仓. 沙盘演练：读写分离集群高可用测试（V0.25）[R]. 金仓培训资料, 2026.06.13

本文是原创文章，采用 CC BY-NC-ND 4.0 协议，完整转载请注明来自 运维小弟

国产数据库
Archives Tip
QR Code for this page
Tipping QR Code