Here are the relevant corosync logs.
It appears that the stop action for resource p_mysql_002 failed, and that
caused a cascading series of service changes. However, I don't understand why,
since no other resources are dependent on p_mysql_002.
[root@001db01a cluster]# cat corosync_filtered.log
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_apply_diff operation for section 'all' to all
(origin=local/cibadmin/2)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.345.30 2
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.0 cc0da1b030418ec8b7c72db1115e2af1
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @epoch=346, @num_updates=0
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++
/cib/configuration/resources/primitive[@id='p_mysql_002']: <meta_attributes
id="p_mysql_002-meta_attributes"/>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++
<nvpair
id="p_mysql_002-meta_attributes-target-role" name="target-role"
value="Stopped"/>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++
</meta_attributes>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request:
Completed cib_apply_diff operation for section 'all': OK (rc=0,
origin=001db01a/cibadmin/2, version=0.346.0)
Feb 16 14:06:24 [3913] 001db01a crmd: info: abort_transition_graph:
Transition aborted by meta_attributes.p_mysql_002-meta_attributes 'create':
Configuration change | cib=0.346.0 source=te_update_diff:456
path=/cib/configuration/resources/primitive[@id='p_mysql_002'] complete=true
Feb 16 14:06:24 [3913] 001db01a crmd: notice: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph
Feb 16 14:06:24 [3912] 001db01a pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01b is online
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01a is online
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_002 (lsb:mysql_002): Started 001db01a (disabled)
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print:
p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd0:1 (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color:
ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color:
ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a pengine: info: native_color:
Resource p_mysql_002 cannot run anywhere
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_vip_clust01 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd0:1 (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:0 (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_fs_clust01 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_fs_clust02 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_vip_clust02 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_001 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_000 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_002 ( 001db01a ) due to node
availability
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_003 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_004 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_005 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_006 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_007 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_008 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_622 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: notice: process_pe_message:
Calculated transition 17789, saving inputs in
/var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_te_invoke:
Processing graph 17789 (ref=pe_calc-dc-1550343984-19545) derived from
/var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_002_stop_0 locally on 001db01a | action 90
Feb 16 14:06:24 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_002_status_15000
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_002_stop_0
Feb 16 14:06:24 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_002 action:stop call_id:1094
Feb 16 14:06:24 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_002 on 001db01a: Cancelled | call=1088
key=p_mysql_002_monitor_15000 confirmed=true
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20017)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.0 2
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.1 (null)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=1
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']:
@operation_key=p_mysql_002_stop_0, @operation=stop,
@transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343984,
@last-rc-change=1550343984, @exec-time=0
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20017, version=0.346.1)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_file_backup:
Archived previous version as /var/lib/pacemaker/cib/cib-72.raw
Feb 16 14:06:24 [3908] 001db01a cib: info:
cib_file_write_with_digest: Wrote version 0.346.0 of the CIB to disk
(digest: 5efac46096f70afefaf0a677cf4ca823)
Feb 16 14:06:24 [3908] 001db01a cib: info:
cib_file_write_with_digest: Reading cluster configuration file
/var/lib/pacemaker/cib/cib.cp2XNp (digest: /var/lib/pacemaker/cib/cib.5hNGXZ)
Feb 16 14:06:29 [3908] 001db01a cib: info: cib_process_ping:
Reporting our current digest to 001db01a: 1064b8ca173ab45545a221e13e4716d8 for
0.346.1 (0x55ac23ea4ef0 0)
Feb 16 14:06:39 [3910] 001db01a lrmd: warning: child_timeout_callback:
p_mysql_002_stop_0 process (PID 36067) timed out
Feb 16 14:06:39 [3910] 001db01a lrmd: warning: operation_finished:
p_mysql_002_stop_0:36067 - timed out after 15000ms
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_002 action:stop call_id:1094 pid:36067 exit-code:1
exec-time:15004ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a crmd: error: process_lrm_event:
Result of stop operation for p_mysql_002 on 001db01a: Timed Out | call=1094
key=p_mysql_002_stop_0 timeout=15000ms
Feb 16 14:06:39 [3913] 001db01a crmd: notice: process_lrm_event:
001db01a-p_mysql_002_stop_0:1094 [ killing mysqld with pid
69229\n.............. ]
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20018)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.1 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.2 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']:
@transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1094, @rc-code=1, @op-status=2, @exec-time=15004
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_failure_0']:
@operation_key=p_mysql_002_stop_0, @operation=stop,
@transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1094, @rc-code=1, @op-status=2, @last-run=1550343984,
@last-rc-change=1550343984, @exec-time=15004
Feb 16 14:06:39 [3913] 001db01a crmd: warning: status_from_rc: Action
90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a crmd: notice: abort_transition_graph:
Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event
failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2
source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a crmd: info: update_failcount:
Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1
(update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_graph_event:
Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20018, version=0.346.2)
Feb 16 14:06:39 [3913] 001db01a crmd: warning: status_from_rc: Action
90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph:
Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event
failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2
source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a crmd: info: update_failcount:
Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1
(update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_peer_update:
Setting fail-count-p_mysql_002#stop_0[001db01a]: (null) -> INFINITY from
001db01a
Feb 16 14:06:39 [3911] 001db01a attrd: info: write_attribute: Sent
update 666 with 2 changes for fail-count-p_mysql_002#stop_0, id=<n/a>,
set=(null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/attrd/666)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_peer_update:
Setting last-failure-p_mysql_002#stop_0[001db01a]: (null) -> 1550343999 from
001db01a
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_graph_event:
Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3913] 001db01a crmd: notice: run_graph:
Transition 17789 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1,
Source=/var/lib/pacemaker/pengine/pe-input-480.bz2): Complete
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:39 [3911] 001db01a attrd: info: write_attribute: Sent
update 667 with 2 changes for last-failure-p_mysql_002#stop_0, id=<n/a>,
set=(null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.2 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.3 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=3
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: ++
/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:
<nvpair id="status-1-fail-count-p_mysql_002.stop_0"
name="fail-count-p_mysql_002#stop_0" value="INFINITY"/>
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/attrd/666, version=0.346.3)
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph:
Transition aborted by status-1-fail-count-p_mysql_002.stop_0 doing create
fail-count-p_mysql_002#stop_0=INFINITY: Transient attribute change |
cib=0.346.3 source=abort_unless_down:343
path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']
complete=true
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 666 for fail-count-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 666 for fail-count-p_mysql_002#stop_0[001db01a]=INFINITY: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 666 for fail-count-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/attrd/667)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.3 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.4 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=4
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: ++
/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:
<nvpair id="status-1-last-failure-p_mysql_002.stop_0"
name="last-failure-p_mysql_002#stop_0" value="1550343999"/>
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/attrd/667, version=0.346.4)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 667 for last-failure-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 667 for last-failure-p_mysql_002#stop_0[001db01a]=1550343999: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback:
Update 667 for last-failure-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph:
Transition aborted by status-1-last-failure-p_mysql_002.stop_0 doing create
last-failure-p_mysql_002#stop_0=1550343999: Transient attribute change |
cib=0.346.4 source=abort_unless_down:343
path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']
complete=true
Feb 16 14:06:39 [3912] 001db01a pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure:
Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure:
Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_add_running:
resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_002 (lsb:mysql_002): FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: pe_get_failcount:
p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning:
check_migration_threshold: Forcing p_mysql_002 away from 001db01a after
1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a pengine: info: rsc_merge_weights:
p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_color:
Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp:
Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp:
Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_vip_clust01 ( 001db01a ) blocked
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: *
Promote p_drbd0:0 ( Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: *
Demote p_drbd0:1 ( Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Move
p_fs_clust01 ( 001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_fs_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_vip_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_001 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_000 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_002 (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_003 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_004 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_005 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_006 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_007 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_008 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_622 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: process_pe_message:
Calculated transition 17790, saving inputs in
/var/lib/pacemaker/pengine/pe-input-481.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: info: handle_response: pe_calc
calculation pe_calc-dc-1550343999-19547 is obsolete
Feb 16 14:06:39 [3912] 001db01a pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status:
Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status:
Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure:
Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure:
Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_add_running:
resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1
is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print:
Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print:
Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_002 (lsb:mysql_002): FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print:
p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: pe_get_failcount:
p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning:
check_migration_threshold: Forcing p_mysql_002 away from 001db01a after
1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a pengine: info: rsc_merge_weights:
p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color:
ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_color:
Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp:
Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp:
Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start
recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_vip_clust01 ( 001db01a ) blocked
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: *
Promote p_drbd0:0 ( Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: *
Demote p_drbd0:1 ( Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Move
p_fs_clust01 ( 001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_fs_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_vip_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_001 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_000 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_002 (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_003 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_004 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_005 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_006 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_007 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave
p_mysql_008 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop
p_mysql_622 ( 001db01a ) due to colocation with
p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: process_pe_message:
Calculated transition 17791, saving inputs in
/var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_te_invoke:
Processing graph 17791 (ref=pe_calc-dc-1550343999-19548) derived from
/var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating cancel operation p_drbd0_monitor_60000 on 001db01b | action 6
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_001_stop_0 locally on 001db01a | action 89
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_001_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_001_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_001 action:stop call_id:1096
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20022)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_000_stop_0 locally on 001db01a | action 92
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_000_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_000_stop_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.4 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.5 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=5
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']:
@operation_key=p_mysql_001_stop_0, @operation=stop,
@transition-key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_000 action:stop call_id:1098
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_003_stop_0 locally on 001db01a | action 96
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20022, version=0.346.5)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_003_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.5 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.6 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_60000']
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=6
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_003_stop_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_delete operation for section status: OK (rc=0,
origin=001db01b/crmd/561, version=0.346.6)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20023)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20024)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_003 action:stop call_id:1100
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_004_stop_0 locally on 001db01a | action 99
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.6 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.7 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=7
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']:
@operation_key=p_mysql_000_stop_0, @operation=stop,
@transition-key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_004_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20023, version=0.346.7)
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_004_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_004 action:stop call_id:1102
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_005_stop_0 locally on 001db01a | action 102
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.7 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.8 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=8
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']:
@operation_key=p_mysql_003_stop_0, @operation=stop,
@transition-key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20024, version=0.346.8)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20025)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_005_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_005_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_005 action:stop call_id:1104
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating stop operation p_mysql_622_stop_0 locally on 001db01a | action 111
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.8 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.9 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=9
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']:
@operation_key=p_mysql_004_stop_0, @operation=stop,
@transition-key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20025, version=0.346.9)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action:
Cancelling lsb operation p_mysql_622_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20026)
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_mysql_622_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_mysql_622 action:stop call_id:1106
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.9 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.10 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=10
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']:
@operation_key=p_mysql_005_stop_0, @operation=stop,
@transition-key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20026, version=0.346.10)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20027)
Feb 16 14:06:39 [3913] 001db01a crmd: info: te_update_diff:
Cancellation of p_drbd0_monitor_60000 on 2 confirmed (6)
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_001 on 001db01a: Cancelled | call=1086
key=p_mysql_001_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.10 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.11 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=11
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_000 on 001db01a: Cancelled | call=1087
key=p_mysql_000_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']:
@operation_key=p_mysql_622_stop_0, @operation=stop,
@transition-key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999,
@last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_003 on 001db01a: Cancelled | call=1089
key=p_mysql_003_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_004 on 001db01a: Cancelled | call=1090
key=p_mysql_004_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_005 on 001db01a: Cancelled | call=1091
key=p_mysql_005_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event:
Result of monitor operation for p_mysql_622 on 001db01a: Cancelled | call=1092
key=p_mysql_622_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20027, version=0.346.11)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating notify operation p_drbd0_pre_notify_demote_0 on 001db01b | action 125
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command:
Initiating notify operation p_drbd0_pre_notify_demote_0 locally on 001db01a |
action 127
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op:
Performing key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5
op=p_drbd0_notify_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20028)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute:
executing - rsc:p_drbd0 action:notify call_id:1107
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.11 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.12 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=12
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']:
@transition-key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20028, version=0.346.12)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.12 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.13 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=13
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']:
@transition-key=125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@transition-magic=-1:193;125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01b/crmd/562, version=0.346.13)
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event:
Action p_drbd0_notify_0 (125) confirmed on 001db01b (rc=0)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_drbd0 action:notify call_id:1107 pid:36727 exit-code:0
exec-time:53ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event:
Action p_drbd0_notify_0 (127) confirmed on 001db01a (rc=0)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: process_lrm_event:
Result of notify operation for p_drbd0 on 001db01a: 0 (ok) | call=1107
key=p_drbd0_notify_0 confirmed=true cib-update=0
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_ping:
Reporting our current digest to 001db01a: 10a59a3eddb032581416a9dc95da7ade for
0.346.13 (0x55ac23ea4ef0 0)
Feb 16 14:06:44 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_622 action:stop call_id:1106 pid:36712 exit-code:0
exec-time:5082ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20029)
Feb 16 14:06:44 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_622 on 001db01a: 0 (ok) | call=1106
key=p_mysql_622_stop_0 confirmed=true cib-update=20029
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.13 2
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.14 (null)
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=14
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']:
@transition-magic=0:0;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1106, @rc-code=0, @op-status=0, @exec-time=5082
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20029, version=0.346.14)
Feb 16 14:06:44 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_622_stop_0 (111) confirmed on 001db01a (rc=0)
Feb 16 14:06:44 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_000 action:stop call_id:1098 pid:36695 exit-code:0
exec-time:5093ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20030)
Feb 16 14:06:44 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_000 on 001db01a: 0 (ok) | call=1098
key=p_mysql_000_stop_0 confirmed=true cib-update=20030
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.14 2
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.15 (null)
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=15
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']:
@transition-magic=0:0;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1098, @rc-code=0, @op-status=0, @exec-time=5093
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20030, version=0.346.15)
Feb 16 14:06:44 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_000_stop_0 (92) confirmed on 001db01a (rc=0)
Feb 16 14:06:49 [3908] 001db01a cib: info: cib_process_ping:
Reporting our current digest to 001db01a: 1bc996395d5a4f64ed405bf122c8e890 for
0.346.15 (0x55ac23ea4ef0 0)
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_003 action:stop call_id:1100 pid:36696 exit-code:0
exec-time:12102ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_004 action:stop call_id:1102 pid:36701 exit-code:0
exec-time:12101ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_005 action:stop call_id:1104 pid:36705 exit-code:0
exec-time:12100ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished:
finished - rsc:p_mysql_001 action:stop call_id:1096 pid:36694 exit-code:0
exec-time:12107ms queue-time:0ms
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20031)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_003 on 001db01a: 0 (ok) | call=1100
key=p_mysql_003_stop_0 confirmed=true cib-update=20031
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_004 on 001db01a: 0 (ok) | call=1102
key=p_mysql_004_stop_0 confirmed=true cib-update=20032
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_005 on 001db01a: 0 (ok) | call=1104
key=p_mysql_005_stop_0 confirmed=true cib-update=20033
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event:
Result of stop operation for p_mysql_001 on 001db01a: 0 (ok) | call=1096
key=p_mysql_001_stop_0 confirmed=true cib-update=20034
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.15 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.16 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=16
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']:
@transition-magic=0:0;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1100, @rc-code=0, @op-status=0, @exec-time=12102
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20031, version=0.346.16)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_003_stop_0 (96) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20032)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20033)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Forwarding cib_modify operation for section status to all
(origin=local/crmd/20034)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.16 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.17 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=17
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']:
@transition-magic=0:0;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1102, @rc-code=0, @op-status=0, @exec-time=12101
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20032, version=0.346.17)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_004_stop_0 (99) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.17 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.18 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=18
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']:
@transition-magic=0:0;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1104, @rc-code=0, @op-status=0, @exec-time=12100
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20033, version=0.346.18)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_005_stop_0 (102) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
--- 0.346.18 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff:
+++ 0.346.19 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib: @num_updates=19
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']:
@transition-magic=0:0;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5,
@call-id=1096, @rc-code=0, @op-status=0, @exec-time=12107
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0,
origin=001db01a/crmd/20034, version=0.346.19)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event:
Action p_mysql_001_stop_0 (89) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3913] 001db01a crmd: warning: run_graph:
Transition 17791 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=28,
Source=/var/lib/pacemaker/pengine/pe-input-482.bz2): Terminated
Feb 16 14:06:51 [3913] 001db01a crmd: warning: te_graph_trigger:
Transition failed: terminated
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_graph: Graph
17791 with 39 actions: batch-limit=0 jobs, network-delay=60000ms
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
126]: Pending rsc op p_drbd0_post_notify_demote_0 on 001db01b
(priority: 1000000, waiting: 50)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
125]: Completed rsc op p_drbd0_pre_notify_demote_0 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
122]: Pending rsc op p_drbd0_post_notify_promote_0 on 001db01b
(priority: 1000000, waiting: 44)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
121]: Pending rsc op p_drbd0_pre_notify_promote_0 on 001db01b
(priority: 0, waiting: 42)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
23]: Pending rsc op p_drbd0_promote_0 on 001db01b
(priority: 0, waiting: 40)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
6]: Completed rsc op p_drbd0_monitor_60000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
128]: Pending rsc op p_drbd0_post_notify_demote_0 on 001db01a
(priority: 1000000, waiting: 50)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
127]: Completed rsc op p_drbd0_pre_notify_demote_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
124]: Pending rsc op p_drbd0_post_notify_promote_0 on 001db01a
(priority: 1000000, waiting: 44)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
123]: Pending rsc op p_drbd0_pre_notify_promote_0 on 001db01a
(priority: 0, waiting: 42)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
27]: Pending rsc op p_drbd0_monitor_60000 on 001db01a
(priority: 0, waiting: 25 45 51)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
25]: Pending rsc op p_drbd0_demote_0 on 001db01a
(priority: 0, waiting: 46)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
51]: Pending pseudo op ms_drbd0_confirmed-post_notify_demoted_0 on N/A
(priority: 1000000, waiting: 50 126 128)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
50]: Pending pseudo op ms_drbd0_post_notify_demoted_0 on N/A (priority:
1000000, waiting: 47)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
49]: Completed pseudo op ms_drbd0_confirmed-pre_notify_demote_0 on N/A
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
48]: Completed pseudo op ms_drbd0_pre_notify_demote_0 on N/A (priority: 0,
waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
47]: Pending pseudo op ms_drbd0_demoted_0 on N/A (priority:
1000000, waiting: 25 46)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
46]: Pending pseudo op ms_drbd0_demote_0 on N/A (priority: 0,
waiting: 82)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
45]: Pending pseudo op ms_drbd0_confirmed-post_notify_promoted_0 on N/A
(priority: 1000000, waiting: 44 122 124)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
44]: Pending pseudo op ms_drbd0_post_notify_promoted_0 on N/A (priority:
1000000, waiting: 41 43)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
43]: Pending pseudo op ms_drbd0_confirmed-pre_notify_promote_0 on N/A
(priority: 0, waiting: 42 121 123)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
42]: Pending pseudo op ms_drbd0_pre_notify_promote_0 on N/A (priority: 0,
waiting: 51)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
41]: Pending pseudo op ms_drbd0_promoted_0 on N/A (priority:
1000000, waiting: 23)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
40]: Pending pseudo op ms_drbd0_promote_0 on N/A (priority: 0,
waiting: 43 47)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
84]: Pending rsc op p_fs_clust01_monitor_20000 on 001db01b
(priority: 0, waiting: 83)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
83]: Pending rsc op p_fs_clust01_start_0 on 001db01b
(priority: 0, waiting: 45 82)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
82]: Pending rsc op p_fs_clust01_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 18]: Unresolved dependency rsc op p_vip_clust01_stop_0 on 001db01a
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
91]: Pending rsc op p_mysql_001_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 90]: Unresolved dependency rsc op p_mysql_001_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
89]: Completed rsc op p_mysql_001_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
94]: Pending rsc op p_mysql_000_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 93]: Unresolved dependency rsc op p_mysql_000_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
92]: Completed rsc op p_mysql_000_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
98]: Pending rsc op p_mysql_003_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 97]: Unresolved dependency rsc op p_mysql_003_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
96]: Completed rsc op p_mysql_003_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
101]: Pending rsc op p_mysql_004_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 100]: Unresolved dependency rsc op p_mysql_004_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
99]: Completed rsc op p_mysql_004_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
104]: Pending rsc op p_mysql_005_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 103]: Unresolved dependency rsc op p_mysql_005_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
102]: Completed rsc op p_mysql_005_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
113]: Pending rsc op p_mysql_622_monitor_15000 on 001db01b
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: *
[Input 112]: Unresolved dependency rsc op p_mysql_622_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action
111]: Completed rsc op p_mysql_622_stop_0 on 001db01a
(priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: info: do_log: Input
I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Feb 16 14:06:51 [3913] 001db01a crmd: notice: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:56 [3908] 001db01a cib: info: cib_process_ping:
Reporting our current digest to 001db01a: cf6fae5210d45a88d51e78fe8acb1178 for
0.346.19 (0x55ac23ea4ef0 0)
> -----Original Message-----
> From: Users <users-boun...@clusterlabs.org> On Behalf Of Eric Robinson
> Sent: Saturday, February 16, 2019 12:51 PM
> To: Cluster Labs - All topics related to open-source clustering welcomed
> <users@clusterlabs.org>
> Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just One
> Fails?
>
> Hi Valentin --
>
> Which logs? You mean /var/log/cluster/corosync.log?
>
> But even if the stop action is resulting in an error, why would the cluster
> also
> try to stop the other services which are not dependent?
>
> > -----Original Message-----
> > From: Users <users-boun...@clusterlabs.org> On Behalf Of Valentin
> > Vidic
> > Sent: Saturday, February 16, 2019 12:44 PM
> > To: users@clusterlabs.org
> > Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just
> > One Fails?
> >
> > On Sat, Feb 16, 2019 at 08:34:21PM +0000, Eric Robinson wrote:
> > > Why is it that when one of the resources that start with p_mysql_*
> > > goes into a FAILED state, all the other MySQL services also stop?
> >
> > Perhaps stop is not working correctly for these lsb services, so for
> > example stopping lsb:mysql_004 also stops the other lsb:mysql_nnn.
> >
> > You would need to send the logs from the event to confirm this.
> >
> > --
> > Valentin
> > _______________________________________________
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> _______________________________________________
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
