Found logs in corosync(!?) log directory these are for primary node ga1-ext
Sep 22 00:45:29 corosync [TOTEM ] A processor failed, forming new configuration. Sep 22 00:45:31 corosync [CMAN ] quorum lost, blocking activity Sep 22 00:45:31 corosync [QUORUM] This node is within the non-primary component and will NOT provide any services. Sep 22 00:45:31 corosync [QUORUM] Members[1]: 1 Sep 22 00:45:31 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Sep 22 00:45:31 corosync [CPG ] chosen downlist: sender r(0) ip(10.12.23.1) ; members(old:2 left:1) Sep 22 00:45:31 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 22 00:45:31 [4418] ga1-ext cib: info: pcmk_cpg_membership: Left[4.0] cib.2 Sep 22 00:45:31 [4418] ga1-ext cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now offline Sep 22 00:45:31 [4418] ga1-ext cib: info: pcmk_cpg_membership: Member[4.0] cib.1 Sep 22 00:45:31 [4423] ga1-ext crmd: info: pcmk_cpg_membership: Left[4.0] crmd.2 Sep 22 00:45:31 [4423] ga1-ext crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now offline Sep 22 00:45:31 [4423] ga1-ext crmd: info: peer_update_callback: Client ga2-ext/peer now has status [offline] (DC=true) Sep 22 00:45:31 [4423] ga1-ext crmd: warning: match_down_event: No match for shutdown action on ga2-ext Sep 22 00:45:31 [4423] ga1-ext crmd: notice: peer_update_callback: Stonith/shutdown of ga2-ext not matched Sep 22 00:45:31 [4423] ga1-ext crmd: info: crm_update_peer_join: peer_update_callback: Node ga2-ext[2] - join-2 phase 4 -> 0 Sep 22 00:45:31 [4423] ga1-ext crmd: info: abort_transition_graph: peer_update_callback:214 - Triggered transition abort (complete=1) : Node failure Sep 22 00:45:31 [4423] ga1-ext crmd: info: pcmk_cpg_membership: Member[4.0] crmd.1 Sep 22 00:45:31 [4423] ga1-ext crmd: notice: cman_event_callback: Membership 900: quorum lost Sep 22 00:45:31 [4423] ga1-ext crmd: notice: crm_update_peer_state: cman_event_callback: Node ga2-ext[2] - state is now lost (was member) Sep 22 00:45:31 [4423] ga1-ext crmd: info: peer_update_callback: ga2-ext is now lost (was member) Sep 22 00:45:31 [4423] ga1-ext crmd: warning: match_down_event: No match for shutdown action on ga2-ext Sep 22 00:45:31 [4423] ga1-ext crmd: notice: peer_update_callback: Stonith/shutdown of ga2-ext not matched Sep 22 00:45:31 [4423] ga1-ext crmd: info: abort_transition_graph: peer_update_callback:214 - Triggered transition abort (complete=1) : Node failure Sep 22 00:45:31 [4423] ga1-ext crmd: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state ] Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_dc_join_offer_one: An unknown node joined - (re-)offer to any unconfirmed nodes Sep 22 00:45:31 [4423] ga1-ext crmd: info: join_make_offer: Making join offers based on membership 900 Sep 22 00:45:31 [4423] ga1-ext crmd: info: join_make_offer: Skipping ga1-ext: already known 4 Sep 22 00:45:31 [4423] ga1-ext crmd: info: abort_transition_graph: do_te_invoke:158 - Triggered transition abort (complete=1) : Peer Halt Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Sep 22 00:45:31 [4423] ga1-ext crmd: info: crmd_join_phase_log: join-2: ga2-ext=none Sep 22 00:45:31 [4423] ga1-ext crmd: info: crmd_join_phase_log: join-2: ga1-ext=confirmed Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Sep 22 00:45:31 [4423] ga1-ext crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/703, version=0.143.10) Sep 22 00:45:31 [4419] ga1-ext stonith-ng: info: pcmk_cpg_membership: Left[3.0] stonith-ng.2 Sep 22 00:45:31 [4419] ga1-ext stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now offline Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/704, version=0.143.10) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/705, version=0.143.11) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/706, version=0.143.11) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/707, version=0.143.12) Sep 22 00:45:31 [4421] ga1-ext attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd) Sep 22 00:45:31 [4421] ga1-ext attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/708, version=0.143.12) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/709, version=0.143.13) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/710, version=0.143.13) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/711, version=0.143.13) Sep 22 00:45:31 [4422] ga1-ext pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Sep 22 00:45:31 [4422] ga1-ext pengine: info: determine_online_status: Node ga1-ext is online Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource dovecot active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: notice: unpack_rsc_op: Operation monitor found resource drbd0:0 active in master mode on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource ClusterIP active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource mail active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource mysql active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource drbdlinks active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: unpack_rsc_op: Operation monitor found resource SharedFS active on ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: clone_print: Master/Slave Set: ms_drbd0 [drbd0] Sep 22 00:45:31 [4422] ga1-ext pengine: info: short_print: Masters: [ ga1-ext ] Sep 22 00:45:31 [4422] ga1-ext pengine: info: short_print: Stopped: [ ga2-ext ] Sep 22 00:45:31 [4422] ga1-ext pengine: info: group_print: Resource Group: service_group Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: SharedFS (ocf::heartbeat:Filesystem): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: drbdlinks (ocf::tummy:drbdlinks): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: ClusterIP (ocf::heartbeat:IPaddr): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: mail (ocf::heartbeat:MailTo): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: mysql (lsb:mysqld): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_print: dovecot (lsb:dovecot): Started ga1-ext Sep 22 00:45:31 [4422] ga1-ext pengine: info: native_color: Resource drbd0:1 cannot run anywhere Sep 22 00:45:31 [4422] ga1-ext pengine: info: master_color: Promoting drbd0:0 (Master ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: master_color: ms_drbd0: Promoted 1 instances of a possible 1 to master Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave drbd0:0 (Master ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave drbd0:1 (Stopped) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave SharedFS (Started ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave drbdlinks (Started ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave ClusterIP (Started ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave mail (Started ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave mysql (Started ga1-ext) Sep 22 00:45:31 [4422] ga1-ext pengine: info: LogActions: Leave dovecot (Started ga1-ext) Sep 22 00:45:31 [4419] ga1-ext stonith-ng: info: pcmk_cpg_membership: Member[3.0] stonith-ng.1 Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='ga1-ext']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/51, version=0.143.13) Sep 22 00:45:31 [4421] ga1-ext attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd0 (10000) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/52, version=0.143.13) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='ga1-ext']//transient_attributes//nvpair[@name='master-drbd0']: OK (rc=0, origin=local/attrd/53, version=0.143.13) Sep 22 00:45:31 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/54, version=0.143.13) Sep 22 00:45:31 [4422] ga1-ext pengine: notice: process_pe_message: Calculated Transition 621: /var/lib/pacemaker/pengine/pe-input-288.bz2 Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_te_invoke: Processing graph 621 (ref=pe_calc-dc-1379803531-659) derived from /var/lib/pacemaker/pengine/pe-input-288.bz2 Sep 22 00:45:31 [4423] ga1-ext crmd: notice: run_graph: Transition 621 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-288.bz2): Complete Sep 22 00:45:31 [4423] ga1-ext crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE Sep 22 00:45:31 [4423] ga1-ext crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Sep 22 00:45:45 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Sep 22 00:45:45 corosync [CMAN ] quorum regained, resuming activity Sep 22 00:45:45 corosync [QUORUM] This node is within the primary component and will provide service. Sep 22 00:45:45 corosync [QUORUM] Members[2]: 1 2 Sep 22 00:45:45 corosync [QUORUM] Members[2]: 1 2 Sep 22 00:45:45 [4423] ga1-ext crmd: notice: cman_event_callback: Membership 904: quorum acquired Sep 22 00:45:45 [4423] ga1-ext crmd: notice: crm_update_peer_state: cman_event_callback: Node ga2-ext[2] - state is now member (was lost) Sep 22 00:45:45 [4423] ga1-ext crmd: info: peer_update_callback: ga2-ext is now member (was lost) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/712, version=0.143.14) Sep 22 00:45:45 [4423] ga1-ext crmd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=37): Try again (6) Sep 22 00:45:45 [4423] ga1-ext crmd: info: cman_event_callback: Membership 904: quorum retained Sep 22 00:45:45 [4418] ga1-ext cib: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=64): Try again (6) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/713, version=0.143.15) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/714, version=0.143.15) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/715, version=0.143.16) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/716, version=0.143.16) Sep 22 00:45:45 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/717, version=0.143.16) Sep 22 00:45:45 [4423] ga1-ext crmd: info: crm_cs_flush: Sent 0 CPG messages (2 remaining, last=37): Try again (6) Sep 22 00:45:46 [4418] ga1-ext cib: info: crm_cs_flush: Sent 0 CPG messages (3 remaining, last=64): Try again (6) Sep 22 00:45:46 corosync [CPG ] chosen downlist: sender r(0) ip(10.12.23.1) ; members(old:1 left:0) Sep 22 00:45:46 [4423] ga1-ext crmd: info: crm_cs_flush: Sent 0 CPG messages (2 remaining, last=37): Try again (6) Sep 22 00:45:46 [4419] ga1-ext stonith-ng: info: pcmk_cpg_membership: Joined[4.0] stonith-ng.2 Sep 22 00:45:46 [4419] ga1-ext stonith-ng: info: pcmk_cpg_membership: Member[4.0] stonith-ng.1 Sep 22 00:45:46 [4419] ga1-ext stonith-ng: info: pcmk_cpg_membership: Member[4.1] stonith-ng.2 Sep 22 00:45:46 [4419] ga1-ext stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now online Sep 22 00:45:46 [4423] ga1-ext crmd: info: pcmk_cpg_membership: Joined[5.0] crmd.2 Sep 22 00:45:46 [4423] ga1-ext crmd: info: pcmk_cpg_membership: Member[5.0] crmd.1 Sep 22 00:45:46 [4423] ga1-ext crmd: info: pcmk_cpg_membership: Member[5.1] crmd.2 Sep 22 00:45:46 [4423] ga1-ext crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now online Sep 22 00:45:46 [4423] ga1-ext crmd: info: peer_update_callback: Client ga2-ext/peer now has status [online] (DC=true) Sep 22 00:45:46 [4418] ga1-ext cib: info: pcmk_cpg_membership: Joined[5.0] cib.2 Sep 22 00:45:46 [4418] ga1-ext cib: info: pcmk_cpg_membership: Member[5.0] cib.1 Sep 22 00:45:46 [4418] ga1-ext cib: info: pcmk_cpg_membership: Member[5.1] cib.2 Sep 22 00:45:46 [4418] ga1-ext cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node ga2-ext[2] - corosync-cpg is now online Sep 22 00:45:46 [4423] ga1-ext crmd: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback ] Sep 22 00:45:46 [4423] ga1-ext crmd: info: do_dc_join_offer_one: An unknown node joined - (re-)offer to any unconfirmed nodes Sep 22 00:45:46 [4423] ga1-ext crmd: info: join_make_offer: Making join offers based on membership 904 Sep 22 00:45:46 [4423] ga1-ext crmd: info: join_make_offer: join-2: Sending offer to ga2-ext Sep 22 00:45:46 [4423] ga1-ext crmd: info: crm_update_peer_join: join_make_offer: Node ga2-ext[2] - join-2 phase 0 -> 1 Sep 22 00:45:46 [4423] ga1-ext crmd: info: join_make_offer: Skipping ga1-ext: already known 4 Sep 22 00:45:46 [4423] ga1-ext crmd: info: abort_transition_graph: do_te_invoke:158 - Triggered transition abort (complete=1) : Peer Halt Sep 22 00:45:46 [4418] ga1-ext cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/718, version=0.143.17) Sep 22 00:45:46 [4419] ga1-ext stonith-ng: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=5): Try again (6) Sep 22 00:45:46 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 22 00:45:46 [4412] ga1-ext pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=9): Try again (6) Sep 22 00:45:46 [4418] ga1-ext cib: info: crm_cs_flush: Sent 4 CPG messages (0 remaining, last=68): OK (1) Sep 22 00:45:46 [4423] ga1-ext crmd: info: crm_cs_flush: Sent 3 CPG messages (0 remaining, last=40): OK (1) Sep 22 00:45:48 [4418] ga1-ext cib: info: cib_process_request: Completed cib_sync_one operation for section 'all': OK (rc=0, origin=ga2-ext/ga2-ext/(null), version=0.143.17) Sep 22 00:45:48 [4412] ga1-ext pacemakerd: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Sep 22 00:45:48 [4419] ga1-ext stonith-ng: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Sep 22 00:45:48 [4421] ga1-ext attrd: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Sep 22 00:45:48 [4418] ga1-ext cib: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Sep 22 00:45:48 [4418] ga1-ext cib: error: cib_cs_destroy: Corosync connection lost! Exiting. Sep 22 00:45:48 [4418] ga1-ext cib: info: terminate_cib: cib_cs_destroy: Exiting fast... Sep 22 00:45:48 [4423] ga1-ext crmd: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Sep 22 00:45:48 [4418] ga1-ext cib: info: qb_ipcs_us_withdraw: withdrawing server sockets Sep 22 00:45:48 [4418] ga1-ext cib: info: crm_client_destroy: Destroying 0 events Sep 22 00:45:48 [4418] ga1-ext cib: info: crm_client_destroy: Destroying 0 events Sep 22 00:45:48 [4418] ga1-ext cib: info: qb_ipcs_us_withdraw: withdrawing server sockets Sep 22 00:45:48 [4418] ga1-ext cib: info: crm_client_destroy: Destroying 0 events Sep 22 00:45:48 [4418] ga1-ext cib: info: qb_ipcs_us_withdraw: withdrawing server sockets Sep 22 00:45:48 [4423] ga1-ext crmd: error: crmd_cs_destroy: connection terminated Sep 22 00:45:48 [4412] ga1-ext pacemakerd: error: mcp_cpg_destroy: Connection destroyed Sep 22 00:45:48 [4418] ga1-ext cib: info: crm_xml_cleanup: Cleaning up memory from libxml2 Sep 22 00:45:48 [4419] ga1-ext stonith-ng: error: stonith_peer_cs_destroy: Corosync connection terminated Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: stonith_shutdown: Terminating with 1 clients Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: cib_connection_destroy: Connection to the CIB closed. Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: crm_client_destroy: Destroying 0 events Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: qb_ipcs_us_withdraw: withdrawing server sockets Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: main: Done Sep 22 00:45:48 [4419] ga1-ext stonith-ng: info: crm_xml_cleanup: Cleaning up memory from libxml2 Sep 22 00:45:48 [4412] ga1-ext pacemakerd: info: crm_xml_cleanup: Cleaning up memory from libxml2 Sep 22 00:45:48 [4421] ga1-ext attrd: crit: attrd_cs_destroy: Lost connection to Corosync service! Sep 22 00:45:48 [4421] ga1-ext attrd: notice: main: Exiting... Sep 22 00:45:48 [4421] ga1-ext attrd: notice: main: Disconnecting client 0x1987990, pid=4423... Sep 22 00:45:48 [4421] ga1-ext attrd: error: attrd_cib_connection_destroy: Connection to the CIB terminated... Sep 22 00:45:48 [4423] ga1-ext crmd: info: qb_ipcs_us_withdraw: withdrawing server sockets Sep 22 00:45:48 [4423] ga1-ext crmd: info: tengine_stonith_connection_destroy: Fencing daemon disconnected Sep 22 00:45:48 [4423] ga1-ext crmd: notice: crmd_exit: Forcing immediate exit: Link has been severed (67) Sep 22 00:45:48 [4423] ga1-ext crmd: info: crm_xml_cleanup: Cleaning up memory from libxml2 Sep 22 00:45:48 [4420] ga1-ext lrmd: info: cancel_recurring_action: Cancelling operation ClusterIP_monitor_30000 Sep 22 00:45:48 [4420] ga1-ext lrmd: warning: qb_ipcs_event_sendv: new_event_notification (4420-4423-6): Bad file descriptor (9) Sep 22 00:45:48 [4420] ga1-ext lrmd: warning: send_client_notify: Notification of client crmd/84c7e6b7-398c-40da-bec9-48b5e36dce2b failed Sep 22 00:45:48 [4420] ga1-ext lrmd: info: crm_client_destroy: Destroying 1 events Sep 22 00:45:48 [4422] ga1-ext pengine: info: crm_client_destroy: Destroying 0 events no logs on ga2-ext, seems strange no corosync configuration on cluster nodes [root@ga2-ext ~]# find /etc/corosync/ /etc/corosync/ /etc/corosync/uidgid.d /etc/corosync/amf.conf.example /etc/corosync/corosync.conf.old /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf.example.udpu /etc/corosync/service.d [root@ga1-ext ~]# find /etc/corosync/ /etc/corosync/ /etc/corosync/corosync.conf.example /etc/corosync/service.d /etc/corosync/corosync.conf.example.udpu /etc/corosync/uidgid.d /etc/corosync/corosync.conf.old /etc/corosync/amf.conf.example same packages on both nodes corosync-1.4.1-15.el6_4.1.x86_64 corosynclib-1.4.1-15.el6_4.1.x86_64 drbd-bash-completion-8.3.15-1.el6.x86_64 drbdlinks-1.23-1.el6.noarch drbd-pacemaker-8.3.15-1.el6.x86_64 drbd-udev-8.3.15-1.el6.x86_64 drbd-utils-8.3.15-1.el6.x86_64 pacemaker-1.1.10-1.el6.x86_64 pacemaker-cli-1.1.10-1.el6.x86_64 pacemaker-cluster-libs-1.1.10-1.el6.x86_64 pacemaker-debuginfo-1.1.10-1.el6.x86_64 pacemaker-libs-1.1.10-1.el6.x86_64 On Sun, 22 Sep 2013 07:14:27 +0000, Alessandro Bono wrote: > Hi > > I have a problem with a cluster where pacemaker dies without logs or something > Problem started when I switched to centos 6.4 and converted cluster from > corosync to cman > this happen typically when system is under high load > tonight I received notification of drbd split brian and found on primary > machine only these programs running > > 4420 ? Ss 1:29 /usr/libexec/pacemaker/lrmd > 4422 ? Ss 0:42 /usr/libexec/pacemaker/pengine > > on secondary machine pacemaker is ok > on logs only drbd disconnect and split brain notification > I tried pacemaker 1.1.8 from centos and 1.1.9 and 1.1.10 from clusterlabs > with same result > > howto debug this problem? > /etc/sysconfig/pacemaker has lots configuration but not sure which one to use > > > pacemaker configuration is: > > node ga1-ext \ > attributes standby="off" > node ga2-ext \ > attributes standby="off" > primitive ClusterIP ocf:heartbeat:IPaddr \ > params ip="10.12.23.3" cidr_netmask="24" \ > op monitor interval="30s" > primitive SharedFS ocf:heartbeat:Filesystem \ > params device="/dev/drbd/by-res/r0" directory="/shared" fstype="ext4" > options="noatime,nobarrier" > primitive dovecot lsb:dovecot > primitive drbd0 ocf:linbit:drbd \ > params drbd_resource="r0" \ > op monitor interval="15s" > primitive drbdlinks ocf:tummy:drbdlinks > primitive mail ocf:heartbeat:MailTo \ > params email="r...@company.com" subject="ga-ext cluster - " > primitive mysql lsb:mysqld > group service_group SharedFS drbdlinks ClusterIP mail mysql dovecot \ > meta target-role="Started" > ms ms_drbd0 drbd0 \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" > colocation service_on_drbd inf: service_group ms_drbd0:Master > order service_after_drbd inf: ms_drbd0:promote service_group:start > property $id="cib-bootstrap-options" \ > dc-version="1.1.10-1.el6-368c726" \ > cluster-infrastructure="cman" \ > expected-quorum-votes="2" \ > stonith-enabled="false" \ > no-quorum-policy="ignore" \ > last-lrm-refresh="1379831462" \ > maintenance-mode="false" > rsc_defaults $id="rsc-options" \ > resource-stickiness="100" > > > cman configuration > > cat /etc/cluster/cluster.conf > > <cluster config_version="6" name="ga-ext_cluster"> > <logging debug="off"/> > <clusternodes> > <clusternode name="ga1-ext" nodeid="1"> > <fence> > <method name="pcmk-redirect"> > <device name="pcmk" port="ga1-ext"/> > </method> > </fence> > </clusternode> > <clusternode name="ga2-ext" nodeid="2"> > <fence> > <method name="pcmk-redirect"> > <device name="pcmk" port="ga2-ext"/> > </method> > </fence> > </clusternode> > </clusternodes> > <fencedevices> > <fencedevice agent="fence_pcmk" name="pcmk"/> > </fencedevices> > </cluster> > > tell me you need other information > > thank you > > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
