shihaomq opened a new issue, #10812: URL: https://github.com/apache/rocketmq/issues/10812
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment 4.19.91-007.ali4000.alios7.x86_64 ### RocketMQ version rocketmq 5.2.0 ### JDK Version java 11 ### Describe the Bug 根因 ReplicasManager.sendHeartbeatToController() 只向 availableControllerAddresses（由 scanAvailableControllerAddresses() 通过 TCP 可达性检查填充）发送心跳，不向 controllerLeaderAddress 发送心跳，即使该地址已被发现并被其他方法（getReplicaInfo、registerBrokerToController、brokerElect、alterSyncStateSet）成功使用。 当 DR 容灾切换导致短暂网络隔离时，Leader 的 IP 从 availableControllerAddresses 中被移除，且可能因 Netty Channel 生命周期问题无法重新加入。此后 Broker 只向 LEARNER/Follower Controller 发送心跳，而这些节点无法处理 Raft 事件（onBrokerHeartbeat 返回 CONTROLLER_NOT_LEADER，心跳被静默丢弃）。 <img width="2996" height="1192" alt="Image" src="https://github.com/user-attachments/assets/1da4c322-5c5a-49e0-8cb5-a5476f999eca"; /> ### Steps to Reproduce 复现步骤 1. 部署跨机房 RocketMQ 集群，启用 Controller 模式（Zone A = PRIMARY，Zone B = SECONDARY） 2. 隔离 Zone A 网络（模拟主机房孤岛）→ Controller Leader 切换到 Zone B 3. 恢复 Zone A 网络 4. 观察 Zone A 的 SLAVE Broker 无法进入 SyncStateSet，拓扑查询接口返回 healthy=false 关键日志证据 1. Broker 端：Zone B Controller 地址在隔离期间被移除，之后未恢复 2026-07-28 17:21:45 WARN ReplicasManager_scan_thread_1 - scanAvailableControllerAddresses remove unconnected address 10.83.32.236:9878 2026-07-28 17:21:45 WARN ReplicasManager_scan_thread_2 - scanAvailableControllerAddresses remove unconnected address 10.83.31.207:9878 2026-07-28 17:22:25 WARN ReplicasManager_scan_thread_6 - scanAvailableControllerAddresses remove unconnected address 10.83.32.235:9878 总共只有 3 条移除记录（全部在启动当天）。之后未重新加入。 2. Broker 端：Controller Leader 地址能正确发现（通过 LEARNER 查询） 2026-08-04 09:10:56 INFO ReplicasManager_ScheduledService_2 - Update controller leader address to mq-mryarmrajcf-controller-1-2.mq-mryarmrajcf-controller-svc-headless.rocketmq:9878 每 ~10 秒更新一次，始终指向 Zone B 的 LEADER。getReplicaInfo、registerBrokerToController 等方法使用此地址均正常工作。 3. Zone A LEARNER Controller：收到了 SLAVE 心跳（但无法处理） 2026-08-04 09:10:47 INFO ControllerRequestExecutorThread_14 - broker BrokerIdentityInfo{clusterName='cluster-mryarmrajcf', brokerName='cluster-mryarmrajcf_broker-0', brokerId=2} heart beat 2026-08-04 09:10:48 INFO ControllerRequestExecutorThread_6 - broker BrokerIdentityInfo{clusterName='cluster-mryarmrajcf', brokerName='cluster-mryarmrajcf_broker-1', brokerId=1} heart beat SLAVE 心跳（broker-0/brokerId=2、broker-1/brokerId=1）到达了 LEARNER，但 LEARNER 返回 CONTROLLER_NOT_LEADER → 心跳被静默丢弃，未存入 brokerLiveTable。 4. Zone B LEADER Controller：只收到 MASTER 心跳，没有 SLAVE 心跳 2026-08-04 09:10:43 INFO ControllerRequestExecutorThread_1 - broker BrokerIdentityInfo{clusterName='cluster-mryarmrajcf', brokerName='cluster-mryarmrajcf_broker-1', brokerId=2} heart beat 2026-08-04 09:10:43 INFO ControllerRequestExecutorThread_10 - broker BrokerIdentityInfo{clusterName='cluster-mryarmrajcf', brokerName='cluster-mryarmrajcf_broker-0', brokerId=1} heart beat 只有 MASTER 心跳（broker-0/brokerId=1、broker-1/brokerId=2）。没有 SLAVE 心跳（broker-0/brokerId=2、broker-1/brokerId=1）。 5. LEADER：因 SLAVE "don't alive" 拒绝 SyncStateSet 变更 2026-08-04 09:10:38 ERROR JRaft-FSMCaller-Disruptor-0 - Rejecting alter syncStateSet request because the replicas {2} don't alive 2026-08-04 09:10:38 ERROR JRaft-FSMCaller-Disruptor-0 - Rejecting alter syncStateSet request because the replicas {1} don't alive 6. LEADER：null channel 过期（由 scanNeedReelectBrokerSets 创建 null BrokerIdentityInfo 导致） 2026-08-04 09:10:55 WARN RaftBrokerHeartbeatManager_scheduledService_1 - The broker channel null expired, brokerInfo BrokerIdentityInfo{clusterName='null', brokerName='cluster-mryarmrajcf_broker-1', brokerId=null} 2026-08-04 09:11:00 WARN RaftBrokerHeartbeatManager_scheduledService_1 - The broker channel null expired, brokerInfo BrokerIdentityInfo{clusterName='null', brokerName='cluster-mryarmrajcf_broker-0', brokerId=null} ### What Did You Expect to See? 111 ### What Did You See Instead? 问题所在 —— sendHeartbeatToController() 只使用 availableControllerAddresses： // ReplicasManager.java line 401-420（5.2.0 ） public void sendHeartbeatToController() { final List<String> controllerAddresses = this.getAvailableControllerAddresses(); for (String controllerAddress : controllerAddresses) { if (StringUtils.isNotEmpty(controllerAddress)) { this.brokerOuterAPI.sendHeartbeatToController( controllerAddress, // ← 只从 availableControllerAddresses 取 this.brokerConfig.getBrokerClusterName(), ... this.brokerControllerId, ... ); } } } 作为对比，controllerLeaderAddress 被其他所有 Controller RPC 方法使用： this.brokerOuterAPI.getReplicaInfo(this.controllerLeaderAddress, ...); // line 644 this.brokerOuterAPI.registerBrokerToController(..., controllerLeaderAddress); // line 561 this.brokerOuterAPI.brokerElect(this.controllerLeaderAddress, ...); // line 378 this.brokerOuterAPI.alterSyncStateSet(this.controllerLeaderAddress, ...); // line 761 controllerLeaderAddress 通过 updateControllerMetadata() 发现——该方法向 LEARNER/Follower Controller 发送查询（这些节点可以回答元数据查询，无需走 Raft）。但心跳处理需要 Raft Leader——只有 LEADER 能将 onBrokerHeartbeat 事件提交到 brokerLiveTable。 核心矛盾：LEARNER 能回答"谁是 Leader"（元数据查询，本地读），但不能处理心跳（需要 Raft 日志提交，只有 Leader 能做）。availableControllerAddresses 中的 LEARNER 地址可以用来发现 Leader，但不能用来处理心跳。而 Leader 的地址恰好不在 availableControllerAddresses 中。 ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
