Somebody want to look at this log and tell me why the cluster failed over? All we did was add a new resource. We've done it many times before without any problems.
-- Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.605.2 2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.0 65654c97e62cd549f22f777a5290fe3a Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @epoch=607, @num_updates=0 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_745" class="lsb" type="mysql_745"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_746" class="lsb" type="mysql_746"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_745"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_746"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_745"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_746"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=ha14a/cibadmin/2, version=0.607.0) Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Wrote version 0.607.0 of the CIB to disk (digest: 1afdb9e480f870a095aa9e39719d29c4) Apr 03 08:50:30 [22762] ha14a cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs (digest: /var/lib/pacemaker/cib/cib.hPwa66) Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_745' not found (17 active resources) Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_745' to the rsc list (18 active resources) Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_monitor_0 Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_746' not found (18 active resources) Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_746' to the rsc list (19 active resources) Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_monitor_0 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.0 2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.1 (null) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=1 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7665, version=0.607.1) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.1 2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.2 (null) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7666, version=0.607.2) Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_monitor_0: not running (node=ha14a, call=142, rc=7, cib-update=88, confirmed=true) Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_745_monitor_0:142 [ not started\n ] Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/88) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.2 2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.3 (null) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=3 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/88, version=0.607.3) Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_monitor_0: not running (node=ha14a, call=146, rc=7, cib-update=89, confirmed=true) Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_monitor_0:146 [ not started\n ] Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/89) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.3 2 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.4 (null) Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=4 Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/89, version=0.607.4) Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_start_0 Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:start call_id:147 Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_start_0 Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:start call_id:148 Apr 03 08:50:31 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:start call_id:147 pid:24627 exit-code:0 exec-time:1064ms queue-time:0ms Apr 03 08:50:31 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_start_0: ok (node=ha14a, call=147, rc=0, cib-update=90, confirmed=true) Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/90) Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.4 2 Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.5 (null) Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=5 Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=147, @rc-code=0, @exec-time=1064 Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/90, version=0.607.5) Apr 03 08:50:36 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: a8fdd49311821ec2fcc7be040e4b02b0 for 0.607.5 (0x21e09d0 0) Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:start call_id:148 pid:24628 exit-code:1 exec-time:10064ms queue-time:0ms Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_start_0: unknown error (node=ha14a, call=148, rc=1, cib-update=91, confirmed=true) Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_start_0:148 [ starting mysqld... \ngave up waiting\n ] Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/91) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.5 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.6 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=6 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=148, @rc-code=1, @exec-time=10064 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="148" rc-code="1" op-status="0" in Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/91, version=0.607.6) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 83: fail-count-p_mysql_746=INFINITY Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/83) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 85: last-failure-p_mysql_746=1491234640 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/85) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.6 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.7 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=7 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/83, version=0.607.7) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 87: fail-count-p_mysql_746=INFINITY Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/87) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.7 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.8 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=8 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234640"/> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/85, version=0.607.8) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 89: last-failure-p_mysql_746=1491234640 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/89) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 91: fail-count-p_mysql_746=INFINITY Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/87, version=0.607.8) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/91) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 93: last-failure-p_mysql_746=1491234640 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/93) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 95: fail-count-p_mysql_746=INFINITY Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/89, version=0.607.8) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/91, version=0.607.8) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/93, version=0.607.8) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/95) Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 97: last-failure-p_mysql_746=1491234640 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/95, version=0.607.8) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/97) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/97, version=0.607.8) Apr 03 08:50:40 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_drbd0_monitor_30000 Apr 03 08:50:40 [22767] ha14a crmd: info: process_lrm_event: Operation p_drbd0_monitor_30000: Cancelled (node=ha14a, call=92, confirmed=true) Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_drbd0_monitor_30000:92 [ \n ] Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_delete operation for section status to master (origin=local/crmd/92) Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_057_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_057 action:stop call_id:150 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_092_stop_0 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.8 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.9 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_30000'] Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=9 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_092 action:stop call_id:151 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_187_stop_0 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14a/crmd/92, version=0.607.9) Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_187 action:stop call_id:152 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_213_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_213 action:stop call_id:153 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_702_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_702 action:stop call_id:154 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_743_stop_0 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.9 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.10 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_31000'] Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=10 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14b/crmd/7670, version=0.607.10) Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_743 action:stop call_id:155 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_312_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_312 action:stop call_id:156 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_289_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_289 action:stop call_id:157 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_250_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_250 action:stop call_id:158 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:stop call_id:159 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_stop_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:stop call_id:160 Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=123:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:161 Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:161 pid:25907 exit-code:0 exec-time:25ms queue-time:0ms Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=161, rc=0, cib-update=0, confirmed=true) Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:stop call_id:160 pid:25792 exit-code:0 exec-time:41ms queue-time:0ms Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_stop_0: ok (node=ha14a, call=160, rc=0, cib-update=93, confirmed=true) Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/93) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.10 2 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.11 (null) Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=11 Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=41 Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/93, version=0.607.11) Apr 03 08:50:42 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_743 action:stop call_id:155 pid:25787 exit-code:0 exec-time:2105ms queue-time:0ms Apr 03 08:50:42 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_743_stop_0: ok (node=ha14a, call=155, rc=0, cib-update=94, confirmed=true) Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/94) Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.11 2 Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.12 (null) Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=12 Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_stop_0, @operation=stop, @transition-key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=2105 Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/94, version=0.607.12) Apr 03 08:50:43 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:stop call_id:159 pid:25791 exit-code:0 exec-time:3107ms queue-time:0ms Apr 03 08:50:43 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_stop_0: ok (node=ha14a, call=159, rc=0, cib-update=95, confirmed=true) Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/95) Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.12 2 Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.13 (null) Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=13 Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_stop_0, @operation=stop, @transition-key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=3107 Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/95, version=0.607.13) Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_213 action:stop call_id:153 pid:25785 exit-code:0 exec-time:4109ms queue-time:0ms Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_312 action:stop call_id:156 pid:25788 exit-code:0 exec-time:4108ms queue-time:0ms Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_289 action:stop call_id:157 pid:25789 exit-code:0 exec-time:4108ms queue-time:0ms Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_092 action:stop call_id:151 pid:25783 exit-code:0 exec-time:4110ms queue-time:0ms Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_213_stop_0: ok (node=ha14a, call=153, rc=0, cib-update=96, confirmed=true) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/96) Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_312_stop_0: ok (node=ha14a, call=156, rc=0, cib-update=97, confirmed=true) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/97) Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_289_stop_0: ok (node=ha14a, call=157, rc=0, cib-update=98, confirmed=true) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/98) Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_092_stop_0: ok (node=ha14a, call=151, rc=0, cib-update=99, confirmed=true) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/99) Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_057 action:stop call_id:150 pid:25782 exit-code:0 exec-time:4111ms queue-time:0ms Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_187 action:stop call_id:152 pid:25784 exit-code:0 exec-time:4110ms queue-time:0ms Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_702 action:stop call_id:154 pid:25786 exit-code:0 exec-time:4110ms queue-time:0ms Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_057_stop_0: ok (node=ha14a, call=150, rc=0, cib-update=100, confirmed=true) Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_187_stop_0: ok (node=ha14a, call=152, rc=0, cib-update=101, confirmed=true) Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_702_stop_0: ok (node=ha14a, call=154, rc=0, cib-update=102, confirmed=true) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.13 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.14 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=14 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_stop_0, @operation=stop, @transition-key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4109 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/96, version=0.607.14) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.14 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.15 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=15 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_stop_0, @operation=stop, @transition-key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/97, version=0.607.15) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.15 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.16 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=16 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_stop_0, @operation=stop, @transition-key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=157, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/98, version=0.607.16) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.16 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.17 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=17 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_stop_0, @operation=stop, @transition-key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/99, version=0.607.17) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/100) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/101) Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/102) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.17 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.18 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=18 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_stop_0, @operation=stop, @transition-key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=150, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4111 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/100, version=0.607.18) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.18 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.19 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=19 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_stop_0, @operation=stop, @transition-key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=152, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/101, version=0.607.19) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.19 2 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.20 (null) Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=20 Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_stop_0, @operation=stop, @transition-key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=154, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110 Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/102, version=0.607.20) Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_250 action:stop call_id:158 pid:25790 exit-code:0 exec-time:5108ms queue-time:0ms Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_250_stop_0: ok (node=ha14a, call=158, rc=0, cib-update=103, confirmed=true) Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/103) Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.20 2 Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.21 (null) Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=21 Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_stop_0, @operation=stop, @transition-key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=5108 Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/103, version=0.607.21) Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_fs_clust19_stop_0 Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_fs_clust19 action:stop call_id:162 Apr 03 08:50:45 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_vip_clust19_monitor_30000 Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_vip_clust19_stop_0 Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_vip_clust19 action:stop call_id:164 Apr 03 08:50:45 [22767] ha14a crmd: info: process_lrm_event: Operation p_vip_clust19_monitor_30000: Cancelled (node=ha14a, call=96, confirmed=true) Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_vip_clust19 action:stop call_id:164 pid:26174 exit-code:0 exec-time:44ms queue-time:0ms Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_vip_clust19_stop_0: ok (node=ha14a, call=164, rc=0, cib-update=104, confirmed=true) Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/104) Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.21 2 Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.22 (null) Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=22 Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_stop_0, @operation=stop, @transition-key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=44 Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/104, version=0.607.22) Apr 03 08:50:50 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: bca4bc21567f887c56ae7ece4ac72814 for 0.607.22 (0x21e09d0 0) Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ] Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ] Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_fs_clust19 action:stop call_id:162 pid:26173 exit-code:0 exec-time:13216ms queue-time:0ms Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_fs_clust19_stop_0: ok (node=ha14a, call=162, rc=0, cib-update=105, confirmed=true) Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/105) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.22 2 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.23 (null) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=23 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_stop_0, @operation=stop, @transition-key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=13216 Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/105, version=0.607.23) Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_lv_on_drbd0_stop_0 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_lv_on_drbd0 action:stop call_id:165 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_lv_on_drbd0 action:stop call_id:165 pid:26495 exit-code:0 exec-time:88ms queue-time:0ms Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_lv_on_drbd0_stop_0: ok (node=ha14a, call=165, rc=0, cib-update=106, confirmed=true) Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/106) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.23 2 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.24 (null) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=24 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_stop_0, @operation=stop, @transition-key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=88 Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/106, version=0.607.24) Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_demote_0 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:demote call_id:166 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:demote call_id:166 pid:26533 exit-code:0 exec-time:33ms queue-time:0ms Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_demote_0: ok (node=ha14a, call=166, rc=0, cib-update=107, confirmed=true) Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/107) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.24 2 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.25 (null) Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=25 Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_demote_0, @operation=demote, @transition-key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=33 Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/107, version=0.607.25) Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=124:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:167 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:167 pid:26567 exit-code:0 exec-time:42ms queue-time:0ms Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=167, rc=0, cib-update=0, confirmed=true) Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=119:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:168 Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:168 pid:26594 exit-code:0 exec-time:27ms queue-time:0ms Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=168, rc=0, cib-update=0, confirmed=true) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.25 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.26 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=26 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_promote_0, @operation=promote, @transition-key=18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=31 Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7671, version=0.607.26) Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=120:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0 Apr 03 08:50:59 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:169 Apr 03 08:50:59 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:169 pid:26616 exit-code:0 exec-time:42ms queue-time:0ms Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=169, rc=0, cib-update=0, confirmed=true) Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_monitor_31000 Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_monitor_31000: ok (node=ha14a, call=170, rc=0, cib-update=108, confirmed=false) Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/108) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.26 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.27 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=27 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_30000" operation_key="p_drbd0_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:8;19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" call-id="154" rc-code="8" op-status="0" interval= Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7672, version=0.607.27) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.27 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.28 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=28 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_31000" operation_key="p_drbd0_monitor_31000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="170" rc-code="0" op-status="0" interval= Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/108, version=0.607.28) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.28 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.29 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=29 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @rc-code=0, @last-run=1491234658, @last-rc-change=14912 Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7673, version=0.607.29) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.29 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.30 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=30 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_start_0, @operation=start, @transition-key=83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=52 Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7674, version=0.607.30) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.30 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.31 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=31 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']: <lrm_rsc_op id="p_vip_clust19_monitor_30000" operation_key="p_vip_clust19_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="157" rc-code="0" op-st Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7675, version=0.607.31) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.31 2 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.32 (null) Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=32 Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_start_0, @operation=start, @transition-key=79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=123 Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7676, version=0.607.32) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.32 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.33 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=33 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_start_0, @operation=start, @transition-key=88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7677, version=0.607.33) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.33 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.34 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=34 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_start_0, @operation=start, @transition-key=96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1051 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7678, version=0.607.34) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.34 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.35 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=35 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_start_0, @operation=start, @transition-key=104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7679, version=0.607.35) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.35 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.36 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=36 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_start_0, @operation=start, @transition-key=90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7680, version=0.607.36) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.36 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.37 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=37 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=167, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1052 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7681, version=0.607.37) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.37 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.38 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=38 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_start_0, @operation=start, @transition-key=100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7682, version=0.607.38) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.38 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.39 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=39 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_start_0, @operation=start, @transition-key=102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7683, version=0.607.39) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.39 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.40 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=40 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_start_0, @operation=start, @transition-key=92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1056 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7684, version=0.607.40) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.40 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.41 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=41 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_start_0, @operation=start, @transition-key=98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=163, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7685, version=0.607.41) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.41 2 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.42 (null) Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=42 Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_start_0, @operation=start, @transition-key=94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=161, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1063 Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7686, version=0.607.42) Apr 03 08:51:05 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 3b1748e1e0487004594b1da9d8e47da6 for 0.607.42 (0x21e09d0 0) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.42 2 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.43 (null) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=43 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=168, @rc-code=1, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=10060 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="168" rc-code="1" op-status="0" in Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7687, version=0.607.43) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/99) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.43 2 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.44 (null) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=44 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/163, version=0.607.44) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.44 2 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.45 (null) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=45 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234669"/> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/165, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/167, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/169, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/171, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/173, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/175, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/99, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/177, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/101) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/103) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/105) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/101, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/103, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/107) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/105, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/109) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/107, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/109, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/111) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/111, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/113) Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/113, version=0.607.45) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.45 2 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.46 (null) Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=46 Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=169, @rc-code=0, @last-run=1491234669, @last-rc-change=1491234669, @exec-time=24 Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7691, version=0.607.46) Apr 03 08:51:14 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 5384604825dd17358b76d9b84f2be021 for 0.607.46 (0x21e09d0 0) -- Eric Robinson _______________________________________________ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org _______________________________________________ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org