Hi Chris, Hi All, Thanks for the comment.
> So the question is: what's controld doing that's keeping it from responding > to attrd? Looking at the subsequent logs, `attrd_updater` from the `ping` resource is failing, which suggests that `attrd` is not responding. `controld`(4671) itself appears to be running. ---- Oct 24 07:15:38.105 rh102-03 pacemaker-execd [4668] (log_op_output) info: ping_monitor_10000[731043] error output [ attrd_updater: Could not update ping-status=1: Connection reset by peer (104) ] Oct 24 07:15:38.110 rh102-03 pacemaker-controld [4671] (log_executor_event) notice: Result of monitor operation for ping on rh102-03: Error occurred | graph action unconfirmed; call=108 key=ping_monitor_10000 rc=1 Oct 24 07:15:38.110 rh102-03 pacemaker-controld [4671] (log_executor_event) notice: ping_monitor_10000@rh102-03 output [ attrd_updater: Could not update ping-status=1: Connection reset by peer (104) ] Oct 24 07:15:38.112 rh102-03 pacemaker-based [4666] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='rh102-03']/transient_attributes: OK (rc=0, origin=rh102-05/pacemaker-attrd/77237, version=100000.111769.1) Oct 24 07:15:38.112 rh102-03 pacemaker-based [4666] (forward_request) info: Forwarding cib_modify operation for section status to all (origin=rh102-03/pacemaker-controld/38017) Oct 24 07:15:38.114 rh102-03 pacemaker-controld [4671] (abort_transition_graph) info: Transition 6764 aborted by deletion of transient_attributes[@id='3']: Transient attribute change | cib=100000.111769.1 source=abort_unless_down:142 path=/cib/status/node_state[@id='3']/transient_attributes[@id='3'] complete=true Oct 24 07:15:38.114 rh102-03 pacemaker-controld [4671] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graphOct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: --- 100000.111769.1 2 Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: Diff: +++ 100000.111769.2 (null)Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: + /cib: @num_updates=2 Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: + /cib/status/node_state[@id='3']: @crm-debug-origin=controld_update_resource_history Oct 24 07:15:38.115 rh102-03 pacemaker-based [4666] (cib_perform_op) info: ++ /cib/status/node_state[@id='3']/lrm[@id='3']/lrm_resources/lrm_resource[@id='ping']: <lrm_rsc_op id="ping_last_failure_0" operation_key="ping_monitor_10000" operation="monitor" crm-debug-origin="controld_update_resource_history" crm_feature_set="3.20.1" transition-key="171:1:0:6417ad8d-8516-427c-9141-01252845936f" transition-magic="0:1;171:1:0:6417ad8d-8516-427c-9141-01252845936f" exit-reason="" on_node="rh102- ---- > The first thing that comes to mind to me is the bug fixed by 39f99f1b. > Does the commit message on that seem like it might be relevant to your setup? > Looking at the source code for Pacemaker included in RHEL 10.2, it appears that this fix(https://github.com/ClusterLabs/pacemaker/commit/39f99f1b) is not included. In RHEL 10.2, the patch `006-fewer_messages.patch` has been applied to address the relevant section. ---- @controld_messages.c (snip) if (input == I_PE_CALC) { GList *ele = NULL; if (prepend) { ele = g_list_first(controld_globals.fsa_message_queue); } else { ele = g_list_last(controld_globals.fsa_message_queue); } if (ele != NULL) { fsa_data_t *message = (fsa_data_t *) ele->data; if (message->fsa_input == I_PE_CALC) { crm_debug("%s item in fsa queue is I_PE_CALC, not adding another", (prepend ? "First" : "Last")); return; } } } (snip) ---- > In particular, do you have a lot of resources or nodes that might be > generating a lot of transitions all at once? As previously mentioned, the setup consists of 5 nodes, with approximately 40 resources (such as "dummy" resources). Note that resource placement is determined based on utilization. Furthermore, this issue is extremely difficult to reproduce; under normal circumstances, this error does not occur. ---- Cluster name: test10-cluster-0 Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: rh102-03 (3) (version 3.0.1-5.el10-be343ca) - partition with quorum * Last updated: Thu Oct 22 17:33:18 2026 on rh102-04 * Last change: Thu Oct 22 17:31:55 2026 by root via root on rh102-04 * 5 nodes configured * 34 resource instances configured Node List: * Node rh102-01 (1): online, feature set 3.20.1 * Node rh102-02 (2): online, feature set 3.20.1 * Node rh102-03 (3): online, feature set 3.20.1 * Node rh102-04 (4): online, feature set 3.20.1 * Node rh102-05 (5): online, feature set 3.20.1 Full List of Resources: * Resource Group: primary-group1: * ipaddr_gwf_101 (ocf:heartbeat:Dummy): Started rh102-01 * ipaddr_dbf_101 (ocf:heartbeat:Dummy): Started rh102-01 * ipaddr_api_101 (ocf:heartbeat:Dummy): Started rh102-01 * ipaddr_kafka_101 (ocf:heartbeat:Dummy): Started rh102-01 * ipaddr_unyo_101 (ocf:heartbeat:Dummy): Started rh102-01 * fs_share_101 (ocf:heartbeat:Dummy): Started rh102-01 * mw_jp1base_101 (ocf:heartbeat:Dummy): Started rh102-01 * mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Started rh102-01 * Resource Group: primary-group2: * ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh102-02 * ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh102-02 * ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh102-02 * ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh102-02 * ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh102-02 * fs_share_102 (ocf:heartbeat:Dummy): Started rh102-02 * mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh102-02 * mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh102-02 * Resource Group: primary-group3: * ipaddr_gwf_103 (ocf:heartbeat:Dummy): Started rh102-03 * ipaddr_dbf_103 (ocf:heartbeat:Dummy): Started rh102-03 * ipaddr_api_103 (ocf:heartbeat:Dummy): Started rh102-03 * ipaddr_kafka_103 (ocf:heartbeat:Dummy): Started rh102-03 * ipaddr_unyo_103 (ocf:heartbeat:Dummy): Started rh102-03 * fs_share_103 (ocf:heartbeat:Dummy): Started rh102-03 * mw_jp1base_103 (ocf:heartbeat:Dummy): Started rh102-03 * mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Started rh102-03 * Clone Set: ping-clone [ping]: * ping (ocf:pacemaker:ping): Started rh102-04 * ping (ocf:pacemaker:ping): Started rh102-03 * ping (ocf:pacemaker:ping): Started rh102-05 * ping (ocf:pacemaker:ping): Started rh102-02 * ping (ocf:pacemaker:ping): Started rh102-01 * fence1-vmware_rest (stonith:fence_vmware_rest): Started rh102-05 * fence2-vmware_rest (stonith:fence_vmware_rest): Started rh102-03 * fence3-vmware_rest (stonith:fence_vmware_rest): Started rh102-05 * fence4-vmware_rest (stonith:fence_vmware_rest): Started rh102-02 * fence5-vmware_rest (stonith:fence_vmware_rest): Started rh102-01 Node Attributes: (snip) ---- Many thanks, Hideo Yamauchi. _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
