hi: I was using pacemaker and drbd with sl linux 6.5/6.6. all are fine. now I am tesing sl linux 7.0 and I notice when I want to promote the drbd resource with "pcs resource meta my-ms-drbd master-max=2".
sometimes pengine find the change immediately, but sometimes it find the change after about a minute. I don't know if the delay is normal? I didn't notice the delay when I using sl linux 6.5/6.6. the "good" result. kvm-3-ms-drbd set master-max = 2 at 13:00:07 and pengine find it at 13:00:07 Feb 10 13:00:06 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2084.3) Feb 10 13:00:06 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2084.3) Feb 10 13:00:06 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2084.3) Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: --- 0.2084.3 Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: +++ 0.2085.1 206a58e68f4a9cd8e72c7ebb40bef026 Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: -- <nvpair value="1" id="kvm-3-ms-drbd-meta_attributes-master-max"/> Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: ++ <nvpair id="kvm-3-ms-drbd-meta_attributes-master-max" name="master-max" value="2"/> Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_replace operation for section configuration: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2085.1) Feb 10 13:00:07 [2898] love1-test.lhy.com.tw crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.2085.1) : Non-status change Feb 10 13:00:07 [2898] love1-test.lhy.com.tw crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/839, version=0.2085.1) Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-17.raw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: determine_online_status: Node love2-test.lhy.com.tw is online Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: determine_online_status: Node love1-test.lhy.com.tw is online Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Wrote version 0.2085.0 of the CIB to disk (digest: bfdd9b0a25cde05a4b2777b6fc670519) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Operation monitor found resource kvm-6-drbd:0 active in master mode on love2-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-6 active on love2-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-1-drbd:1 active on love1-test.lhy.com.tw Feb 10 13:00:07 [2893] love1-test.lhy.com.tw cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.2Mn5wa (digest: /var/lib/pacemaker/cib/cib.0nfve5) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-3-drbd:1 active on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Re-initiated expired calculated failure kvm-4_last_failure_0 (rc=7, magic=0:7;144:22:0:87034531-de2d-4395-b3c0-9bc0cecfc50e) on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-2-drbd:1 active on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-5 active on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Operation monitor found resource kvm-5-drbd:1 active in master mode on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-6-drbd:1 active on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-1-ms-drbd [kvm-1-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love1-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-1 (ocf::heartbeat:VirtualDomain): Started love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-2-ms-drbd [kvm-2-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love1-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-2 (ocf::heartbeat:VirtualDomain): Started love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-3-ms-drbd [kvm-3-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-3 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-4-ms-drbd [kvm-4-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-4 (ocf::heartbeat:VirtualDomain): Stopped Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-5-ms-drbd [kvm-5-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love1-test.lhy.com.tw love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-5 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-6-ms-drbd [kvm-6-drbd] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-6 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-1-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-1-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-2-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-2-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-3-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-3-drbd:1 (Slave love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-3-ms-drbd: Promoted 2 instances of a possible 2 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-4-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-4-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: native_color: Resource kvm-4 cannot run anywhere Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-5-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-5-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-5-ms-drbd: Promoted 2 instances of a possible 2 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-6-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-6-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Start recurring monitor (29s) for kvm-3-drbd:1 on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Cancelling action kvm-3-drbd:1_monitor_31000 (Slave vs. Master) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Start recurring monitor (29s) for kvm-3-drbd:1 on love1-test.lhy.com.tw Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Cancelling action kvm-3-drbd:1_monitor_31000 (Slave vs. Master) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1-drbd:0 (Slave love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1 (Started love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-2-drbd:0 (Slave love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-2-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-2 (Started love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-3-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: notice: LogActions: Promote kvm-3-drbd:1 (Slave -> Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-3 (Started love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-4-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-4-drbd:1 (Slave love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-4 (Stopped) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-5-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-5-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-5 (Started love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-6-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-6-drbd:1 (Slave love1-test.lhy.com.tw) Feb 10 13:00:07 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-6 (Started love2-test.lhy.com.tw) Feb 10 13:00:07 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] the "lazy" result: kvm-2-ms-drbd set master-max = 2 at 12:47:09. pengine find it at 12:47:52 pacemaker seems quite busy. so I also include the log before it notice "set master-max=2" Feb 10 12:46:59 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x1702230 for uid=0 gid=0 pid=26657 id=5c48e106-b6e0-471f-9862-ae2530333bc6 Feb 10 12:46:59 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2076.3) Feb 10 12:46:59 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:05 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x1702230 for uid=0 gid=0 pid=26768 id=f7d4e43f-43dc-444b-b723-184956716d24 Feb 10 12:47:05 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2076.3) Feb 10 12:47:05 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2076.3) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2076.3) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2076.3) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2076.3) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_replace_notify: Replaced: 0.2076.3 -> 0.2077.1 from love2-test.lhy.com.tw Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: --- 0.2076.3 Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: +++ 0.2077.1 b88b0f390464641de0dd02e91941212c Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: -- <rsc_location id="location-kvm-2-love2-test.lhy.com.tw-99" node="love2-test.lhy.com.tw" rsc="kvm-2" score="99"/> Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: ++ <cib admin_epoch="0" cib-last-written="Tue Feb 10 12:47:08 2015" crm_feature_set="3.0.7" dc-uuid="1" epoch="2077" have-quorum="1" num_updates="1" update-client="cibadmin" update-origin="love2-test.lhy.com.tw" validate-with="pacemaker-1.2"/> Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2077.1) Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.2077.1) : Non-status change Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ] Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: update_dc: Unset DC. Was love1-test.lhy.com.tw Feb 10 12:47:08 [2894] love1-test.lhy.com.tw stonith-ng: info: update_cib_stonith_devices: Updating device list from the cib: new location constraint Feb 10 12:47:08 [2894] love1-test.lhy.com.tw stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_ELECTION Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_dc_takeover: Taking over DC status for this partition Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/791, version=0.2077.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Wrote version 0.2077.0 of the CIB to disk (digest: e0b1003c5a419cf4297a8e9d85065af9) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.5eUwAY (digest: /var/lib/pacemaker/cib/cib.KicCQd) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/792, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/793, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/794, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/795, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: OK (rc=0, origin=local/crmd/796, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-5-drbd']: OK (rc=0, origin=local/attrd/817, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/797, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: OK (rc=0, origin=local/crmd/798, version=0.2077.2) Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: initialize_join: Node love2-test.lhy.com.tw[2] - join-12 phase 4 -> 0 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: initialize_join: Node love1-test.lhy.com.tw[1] - join-12 phase 4 -> 0 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: join_make_offer: join-12: Sending offer to love2-test.lhy.com.tw Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: join_make_offer: Node love2-test.lhy.com.tw[2] - join-12 phase 0 -> 1 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: join_make_offer: join-12: Sending offer to love1-test.lhy.com.tw Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: join_make_offer: Node love1-test.lhy.com.tw[1] - join-12 phase 0 -> 1 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_dc_join_offer_all: join-12: Waiting on 2 outstanding join acks Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: update_dc: Set DC to love1-test.lhy.com.tw (3.0.7) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/818, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-1-drbd']: OK (rc=0, origin=local/attrd/819, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/799, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/800, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/801, version=0.2077.2) Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node love1-test.lhy.com.tw[1] - join-12 phase 1 -> 2 Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/820, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-kvm-2']: No such device or address (rc=-6, origin=local/attrd/821, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-kvm-2']: No such device or address (rc=-6, origin=local/attrd/822, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='terminate']: No such device or address (rc=-6, origin=local/attrd/823, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-6-drbd']: OK (rc=0, origin=local/attrd/824, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/825, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='shutdown']: No such device or address (rc=-6, origin=local/attrd/826, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-kvm-4-drbd']: No such device or address (rc=-6, origin=local/attrd/827, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-kvm-4-drbd']: No such device or address (rc=-6, origin=local/attrd/828, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/829, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/830, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-4-drbd']: OK (rc=0, origin=local/attrd/831, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/832, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-3-drbd']: OK (rc=0, origin=local/attrd/833, version=0.2077.2) Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node love2-test.lhy.com.tw[2] - join-12 phase 1 -> 2 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crmd_join_phase_log: join-12: love2-test.lhy.com.tw=integrated Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crmd_join_phase_log: join-12: love1-test.lhy.com.tw=integrated Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: do_dc_join_finalize: join-12: Syncing our CIB to the rest of the cluster Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/802, version=0.2077.2) Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: finalize_join_for: Node love2-test.lhy.com.tw[2] - join-12 phase 2 -> 3 Feb 10 12:47:08 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: finalize_join_for: Node love1-test.lhy.com.tw[1] - join-12 phase 2 -> 3 Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/834, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-2-drbd']: OK (rc=0, origin=local/attrd/835, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/803, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/804, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/836, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //primitive[@id='kvm-2']: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2077.2) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_replace_notify: Replaced: 0.2077.2 -> 0.2078.1 from love2-test.lhy.com.tw Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: --- 0.2077.2 Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: Diff: +++ 0.2078.1 6fa754421695a978ce08373dffbb67de Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: -- <cib admin_epoch="0" epoch="2077" num_updates="2"/> Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: ++ <rsc_location id="location-kvm-2-love1-test.lhy.com.tw-99" node="love1-test.lhy.com.tw" rsc="kvm-2" score="99"/> Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2078.1) Feb 10 12:47:08 [2894] love1-test.lhy.com.tw stonith-ng: info: update_cib_stonith_devices: Updating device list from the cib: new location constraint Feb 10 12:47:08 [2894] love1-test.lhy.com.tw stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-5-drbd']: OK (rc=0, origin=local/attrd/837, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/838, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-1-drbd']: OK (rc=0, origin=local/attrd/839, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Wrote version 0.2078.0 of the CIB to disk (digest: 4a9ac4598ffeae5c1faac845cded1447) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.WI7bEl (digest: /var/lib/pacemaker/cib/cib.Q1fxBB) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/840, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-kvm-2']: No such device or address (rc=-6, origin=local/attrd/841, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-kvm-2']: No such device or address (rc=-6, origin=local/attrd/842, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='terminate']: No such device or address (rc=-6, origin=local/attrd/843, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-6-drbd']: OK (rc=0, origin=local/attrd/844, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/845, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='shutdown']: No such device or address (rc=-6, origin=local/attrd/846, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='fail-count-kvm-4-drbd']: No such device or address (rc=-6, origin=local/attrd/847, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='last-failure-kvm-4-drbd']: No such device or address (rc=-6, origin=local/attrd/848, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/849, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/850, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-4-drbd']: OK (rc=0, origin=local/attrd/851, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/852, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-3-drbd']: OK (rc=0, origin=local/attrd/853, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/854, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-2-drbd']: OK (rc=0, origin=local/attrd/855, version=0.2078.1) Feb 10 12:47:08 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/856, version=0.2078.1) Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: notice: log_cib_diff: cib:diff: Local-only Change: 0.2079.1 Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: -- <nvpair value="1" id="kvm-2-ms-drbd-meta_attributes-master-max"/> Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: notice: cib:diff: ++ <nvpair id="kvm-2-ms-drbd-meta_attributes-master-max" name="master-max" value="2"/> Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_replace operation for section configuration: OK (rc=0, origin=love2-test.lhy.com.tw/cibadmin/2, version=0.2079.1) Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-10.raw Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: info: write_cib_contents: Wrote version 0.2079.0 of the CIB to disk (digest: 426ac916d9809e7225206d967523b0d7) Feb 10 12:47:09 [2893] love1-test.lhy.com.tw cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.tsn1W3 (digest: /var/lib/pacemaker/cib/cib.Brz4Nl) Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=26876 id=58f20094-efe2-4574-9c7f-46f1691c75b9 Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=26920 id=41c0ee18-ff9c-4435-8cfc-df8e60a00d90 Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:11 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:12 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=26947 id=0364ce79-0c50-4421-a5f9-e3b2733f663a Feb 10 12:47:12 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:12 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=27021 id=66f58f1f-985e-40e9-b4bf-55599231045c Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_resource/2, version=0.2079.1) Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=27036 id=c4150508-7b6b-4663-a05b-2f7e664ce12a Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_resource/2, version=0.2079.1) Feb 10 12:47:15 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:16 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x17b5c90 for uid=0 gid=0 pid=27074 id=310b0278-edf8-4367-8d2f-9d3ca96dcf35 Feb 10 12:47:16 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:16 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: services_os_action_execute: Managed VirtualDomain_meta-data_0 process 26798 exited with rc=0 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ] Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: update_dc: Unset DC. Was love1-test.lhy.com.tw Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_log: FSA: Input I_JOIN_RESULT from route_message() received in state S_ELECTION Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/805, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/806, version=0.2079.1) Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_log: FSA: Input I_JOIN_RESULT from route_message() received in state S_ELECTION Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_ELECTION Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_dc_takeover: Taking over DC status for this partition Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/794, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/796, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/798, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/800, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/802, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/804, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/806, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/807, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/808, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: OK (rc=0, origin=local/crmd/809, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-5-drbd']: OK (rc=0, origin=local/attrd/857, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/810, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: OK (rc=0, origin=local/crmd/811, version=0.2079.1) Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: initialize_join: Node love2-test.lhy.com.tw[2] - join-13 phase 3 -> 0 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: initialize_join: Node love1-test.lhy.com.tw[1] - join-13 phase 3 -> 0 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: join_make_offer: join-13: Sending offer to love2-test.lhy.com.tw Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: join_make_offer: Node love2-test.lhy.com.tw[2] - join-13 phase 0 -> 1 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: join_make_offer: join-13: Sending offer to love1-test.lhy.com.tw Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: join_make_offer: Node love1-test.lhy.com.tw[1] - join-13 phase 0 -> 1 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_dc_join_offer_all: join-13: Waiting on 2 outstanding join acks Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: update_dc: Set DC to love1-test.lhy.com.tw (3.0.7) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/858, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-1-drbd']: OK (rc=0, origin=local/attrd/859, version=0.2079.1) Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node love2-test.lhy.com.tw[2] - join-13 phase 1 -> 2 Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/812, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/813, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/814, version=0.2079.1) Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node love1-test.lhy.com.tw[1] - join-13 phase 1 -> 2 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crmd_join_phase_log: join-13: love2-test.lhy.com.tw=integrated Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crmd_join_phase_log: join-13: love1-test.lhy.com.tw=integrated Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: do_dc_join_finalize: join-13: Syncing our CIB to the rest of the cluster Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/860, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-6-drbd']: OK (rc=0, origin=local/attrd/861, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/815, version=0.2079.1) Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: finalize_join_for: Node love2-test.lhy.com.tw[2] - join-13 phase 2 -> 3 Feb 10 12:47:29 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: finalize_join_for: Node love1-test.lhy.com.tw[1] - join-13 phase 2 -> 3 Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/862, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/863, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/816, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/817, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/864, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-4-drbd']: OK (rc=0, origin=local/attrd/865, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/866, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-3-drbd']: OK (rc=0, origin=local/attrd/867, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/868, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-2-drbd']: OK (rc=0, origin=local/attrd/869, version=0.2079.1) Feb 10 12:47:29 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/870, version=0.2079.1) Feb 10 12:47:30 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27288 id=a78a45ab-c7a8-4e0c-af97-a59ff14b578f Feb 10 12:47:30 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:30 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:35 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27382 id=f3301bb0-4f23-4e32-ad12-5338624a5120 Feb 10 12:47:35 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_resource/2, version=0.2079.1) Feb 10 12:47:35 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:35 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27403 id=6536dcf8-d6c6-450d-a412-35fe9fddc1a0 Feb 10 12:47:36 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_resource/2, version=0.2079.1) Feb 10 12:47:36 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:36 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27443 id=23612868-f8c7-4f66-83b4-4736f8d25aed Feb 10 12:47:36 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:36 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:40 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27526 id=378ab5df-dd08-439d-98cf-404edafe4b7d Feb 10 12:47:40 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:40 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:42 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27553 id=ea0e362c-af2f-4c8c-aae9-4e8147a3284a Feb 10 12:47:42 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:42 [2893] love1-test.lhy.com.tw cib: info: crm_client_destroy: Destroying 0 events Feb 10 12:47:43 [2893] love1-test.lhy.com.tw cib: info: crm_client_new: Connecting 0x16fc690 for uid=0 gid=0 pid=27608 id=ae24efd4-b675-42bd-bd57-bf8c0c5b0e9e Feb 10 12:47:43 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section nodes: OK (rc=0, origin=local/crm_attribute/2, version=0.2079.1) Feb 10 12:47:50 [2894] love1-test.lhy.com.tw stonith-ng: info: apply_xml_diff: Digest mis-match: expected f900b640712edb3268d3c77cd20af86d, calculated b2efd510d6e65fe0acf9adc01ce0aa29 Feb 10 12:47:50 [2894] love1-test.lhy.com.tw stonith-ng: warning: cib_process_diff: Diff 0.2079.1 -> 0.2079.2 from local not applied to 0.2079.1: Failed application of an update diff Feb 10 12:47:50 [2894] love1-test.lhy.com.tw stonith-ng: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/819, version=0.2079.3) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_sync_one operation for section 'all': OK (rc=0, origin=love2-test.lhy.com.tw/love2-test.lhy.com.tw/(null), version=0.2079.3) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/128, version=0.2079.3) Feb 10 12:47:50 [2894] love1-test.lhy.com.tw stonith-ng: info: cib_process_diff: Diff 0.2079.2 -> 0.2079.3 from local not applied to 0.2079.3: current "num_updates" is greater than required Feb 10 12:47:50 [2894] love1-test.lhy.com.tw stonith-ng: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/129, version=0.2079.3) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/808, version=0.2079.3) Feb 10 12:47:50 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/810, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/812, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/818, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/820, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/822, version=0.2079.3) Feb 10 12:47:51 [2898] love1-test.lhy.com.tw crmd: info: crm_update_peer_join: do_dc_join_ack: Node love2-test.lhy.com.tw[2] - join-13 phase 3 -> 4 Feb 10 12:47:51 [2898] love1-test.lhy.com.tw crmd: info: do_dc_join_ack: join-13: Updating node state to member for love2-test.lhy.com.tw Feb 10 12:47:51 [2898] love1-test.lhy.com.tw crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='love2-test.lhy.com.tw']/lrm Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/824, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/826, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/828, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/830, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/832, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/834, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/836, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/838, version=0.2079.3) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='love2-test.lhy.com.tw']/lrm: OK (rc=0, origin=local/crmd/820, version=0.2079.4) Feb 10 12:47:51 [2894] love1-test.lhy.com.tw stonith-ng: info: apply_xml_diff: Digest mis-match: expected dac587ccce6bb019a584f063ffbad0d0, calculated 1f84ba9b941fa174c57d078f075824c1 Feb 10 12:47:51 [2894] love1-test.lhy.com.tw stonith-ng: warning: cib_process_diff: Diff 0.2079.3 -> 0.2079.4 from local not applied to 0.2079.3: Failed application of an update diff Feb 10 12:47:51 [2894] love1-test.lhy.com.tw stonith-ng: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/821, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-5-drbd']: OK (rc=0, origin=local/attrd/871, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/130, version=0.2079.5) Feb 10 12:47:51 [2894] love1-test.lhy.com.tw stonith-ng: info: cib_process_diff: Diff 0.2079.4 -> 0.2079.5 from local not applied to 0.2079.5: current "num_updates" is greater than required Feb 10 12:47:51 [2894] love1-test.lhy.com.tw stonith-ng: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/872, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-1-drbd']: OK (rc=0, origin=local/attrd/873, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_sync_one operation for section 'all': OK (rc=0, origin=love2-test.lhy.com.tw/love2-test.lhy.com.tw/(null), version=0.2079.5) Feb 10 12:47:51 [2898] love1-test.lhy.com.tw crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 10 12:47:51 [2898] love1-test.lhy.com.tw crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/131, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/874, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-6-drbd']: OK (rc=0, origin=local/attrd/875, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/822, version=0.2079.5) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/823, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/824, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/825, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/876, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/877, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/878, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-4-drbd']: OK (rc=0, origin=local/attrd/879, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/880, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-3-drbd']: OK (rc=0, origin=local/attrd/881, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/882, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-2-drbd']: OK (rc=0, origin=local/attrd/883, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-5-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/884, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-5-drbd']: OK (rc=0, origin=local/attrd/885, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-1-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/886, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/840, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/842, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/844, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/850, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/852, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/854, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/856, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/858, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-1-drbd']: OK (rc=0, origin=local/attrd/887, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-6-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/888, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/860, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-6-drbd']: OK (rc=0, origin=local/attrd/889, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/890, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/862, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='probe_complete']: OK (rc=0, origin=local/attrd/891, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-4-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/892, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/864, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-4-drbd']: OK (rc=0, origin=local/attrd/893, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-3-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/894, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/866, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-3-drbd']: OK (rc=0, origin=local/attrd/895, version=0.2079.6) Feb 10 12:47:51 [2896] love1-test.lhy.com.tw attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: master-kvm-2-drbd (10000) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/896, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/868, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='1']//transient_attributes//nvpair[@name='master-kvm-2-drbd']: OK (rc=0, origin=local/attrd/897, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/898, version=0.2079.6) Feb 10 12:47:51 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=love2-test.lhy.com.tw/attrd/870, version=0.2079.6) Feb 10 12:47:52 [2893] love1-test.lhy.com.tw cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/826, version=0.2079.6) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: notice: unpack_config: On loss of CCM Quorum: Ignore Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: determine_online_status: Node love2-test.lhy.com.tw is online Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: determine_online_status: Node love1-test.lhy.com.tw is online Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Operation monitor found resource kvm-6-drbd:0 active in master mode on love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-6 active on love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-1-drbd:1 active on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-3-drbd:1 active on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Re-initiated expired calculated failure kvm-4_last_failure_0 (rc=7, magic=0:7;144:22:0:87034531-de2d-4395-b3c0-9bc0cecfc50e) on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-2-drbd:1 active on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-5 active on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: notice: unpack_rsc_op: Operation monitor found resource kvm-5-drbd:1 active in master mode on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: unpack_rsc_op: Operation monitor found resource kvm-6-drbd:1 active on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-1-ms-drbd [kvm-1-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love1-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-1 (ocf::heartbeat:VirtualDomain): Started love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-2-ms-drbd [kvm-2-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-2 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-3-ms-drbd [kvm-3-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-3 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-4-ms-drbd [kvm-4-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-4 (ocf::heartbeat:VirtualDomain): Stopped Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-5-ms-drbd [kvm-5-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love1-test.lhy.com.tw love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-5 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: clone_print: Master/Slave Set: kvm-6-ms-drbd [kvm-6-drbd] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Masters: [ love2-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: short_print: Slaves: [ love1-test.lhy.com.tw ] Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_print: kvm-6 (ocf::heartbeat:VirtualDomain): Started love2-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-1-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-1-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-2-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-2-drbd:1 (Slave love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-2-ms-drbd: Promoted 2 instances of a possible 2 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-3-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-3-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-4-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-4-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: native_color: Resource kvm-4 cannot run anywhere Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-5-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-5-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-5-ms-drbd: Promoted 2 instances of a possible 2 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: Promoting kvm-6-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: master_color: kvm-6-ms-drbd: Promoted 1 instances of a possible 1 to master Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Start recurring monitor (29s) for kvm-2-drbd:1 on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Cancelling action kvm-2-drbd:1_monitor_31000 (Slave vs. Master) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Start recurring monitor (29s) for kvm-2-drbd:1 on love1-test.lhy.com.tw Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: RecurringOp: Cancelling action kvm-2-drbd:1_monitor_31000 (Slave vs. Master) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1-drbd:0 (Slave love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1-drbd:1 (Master love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-1 (Started love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-2-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: notice: LogActions: Promote kvm-2-drbd:1 (Slave -> Master love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-2 (Started love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-3-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-3-drbd:1 (Slave love1-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-3 (Started love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-4-drbd:0 (Master love2-test.lhy.com.tw) Feb 10 12:47:52 [2897] love1-test.lhy.com.tw pengine: info: LogActions: Leave kvm-4-drbd:1 (Slave love1-test.lhy.com.tw) _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org