Yeah I had that problem earlier. I tried the config you had, and these
are what spits out in my error logs, xen1:
Jun 27 10:51:48 xen1 crmd: [3952]: info: update_dc: Set DC to
fa1.box.com (3.0.1)
Jun 27 10:51:49 xen1 crmd: [3952]: info: update_attrd: Connecting to attrd...
Jun 27 10:51:49 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for terminate
Jun 27 10:51:49 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for shutdown
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_state_transition: State
transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE
origin=do_cl_join_finalize_respond ]
Jun 27 10:51:49 xen1 attrd: [3950]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 27 10:51:49 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate (<null>)
Jun 27 10:51:49 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown (<null>)
Jun 27 10:51:49 xen1 attrd: [3950]: info: crm_new_peer: Node
xen2.box.com now has id: 50397194
Jun 27 10:51:49 xen1 attrd: [3950]: info: crm_new_peer: Node 50397194
is now known as xen2.box.com
Jun 27 10:51:49 xen1 crmd: [3952]: info: erase_xpath_callback:
Deletion of "//node_sta...@uname='xen1.box.com']/transient_attributes":
ok (rc=0)
Jun 27 10:51:49 xen1 attrd: [3950]: info: crm_new_peer: Node
fa1.box.com now has id: 16842762
Jun 27 10:51:49 xen1 attrd: [3950]: info: crm_new_peer: Node 16842762
is now known as fa1.box.com
Jun 27 10:51:49 xen1 cib: [3963]: info: write_cib_contents: Archived
previous version as /var/lib/heartbeat/crm/cib-62.raw
Jun 27 10:51:49 xen1 cib: [3963]: info: write_cib_contents: Wrote
version 0.79.0 of the CIB to disk (digest:
53669bbde5238efd1608e412ed8bdd37)
Jun 27 10:51:49 xen1 cib: [3963]: info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.FMAqCU (digest:
/var/lib/heartbeat/crm/cib.ecNEdF)
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=4:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_monitor_0 )
Jun 27 10:51:49 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:2: probe
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=5:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen2:1_monitor_0 )
Jun 27 10:51:49 xen1 lrmd: [3949]: info: rsc:drbd_xen2:1:3: probe
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=6:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6 op=xen1_fs_monitor_0 )
Jun 27 10:51:49 xen1 lrmd: [3949]: info: rsc:xen1_fs:4: probe
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=7:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6 op=xen2_fs_monitor_0 )
Jun 27 10:51:49 xen1 lrmd: [3949]: info: rsc:xen2_fs:5: probe
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=8:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6 op=vm1xen1_monitor_0 )
Jun 27 10:51:49 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:probe:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:49 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:probe:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:49 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:probe:stderr) 'xen2' not defined in your config.
Jun 27 10:51:49 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen2:1_monitor_0 (call=3, rc=7, cib-update=8,
confirmed=true) not running
Jun 27 10:51:49 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for master-drbd_xen2:0
Jun 27 10:51:49 xen1 Filesystem[3971]: WARNING: Couldn't find device
[/dev/drbd2]. Expected /dev/??? to exist
Jun 27 10:51:49 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen1_fs_monitor_0 (call=4, rc=7, cib-update=9,
confirmed=true) not running
Jun 27 10:51:49 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for master-drbd_xen1:0
Jun 27 10:51:49 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: master-drbd_xen1:0 (10000)
Jun 27 10:51:49 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 13: master-drbd_xen1:0=10000
Jun 27 10:51:49 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=9:0:7:e8d1e361-2956-49c9-9c49-9babc795edc6 op=vm1xen2_monitor_0 )
Jun 27 10:51:49 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_monitor_0 (call=2, rc=0, cib-update=10,
confirmed=true) ok
Jun 27 10:51:49 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen2_fs_monitor_0 (call=5, rc=7, cib-update=11,
confirmed=true) not running
Jun 27 10:51:49 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for master-drbd_xen1:1
Jun 27 10:51:54 xen1 lrmd: [3949]: info: rsc:vm1xen1:6: probe
Jun 27 10:51:54 xen1 lrmd: [3949]: info: rsc:vm1xen2:7: probe
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation vm1xen1_monitor_0 (call=6, rc=7, cib-update=12,
confirmed=true) not running
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation vm1xen2_monitor_0 (call=7, rc=7, cib-update=13,
confirmed=true) not running
Jun 27 10:51:55 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for probe_complete
Jun 27 10:51:55 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Jun 27 10:51:55 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 18: probe_complete=true
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=92:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:8: notify
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=8, rc=0, cib-update=14,
confirmed=true) ok
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=9:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_promote_0 )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:9: promote
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:promote:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:promote:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:55 xen1 kernel: block drbd1: role( Secondary -> Primary )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:promote:stdout)
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_promote_0 (call=9, rc=0, cib-update=15,
confirmed=true) ok
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=93:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:10: notify
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:51:55 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:notify:stdout)
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=10, rc=0, cib-update=16,
confirmed=true) ok
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=70:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=xen1_fs_start_0 )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: rsc:xen1_fs:11: start
Jun 27 10:51:55 xen1 Filesystem[4224]: INFO: Running start for
/dev/drbd1 on /xen1
Jun 27 10:51:55 xen1 kernel: kjournald starting. Commit interval 5 seconds
Jun 27 10:51:55 xen1 kernel: EXT3 FS on drbd1, internal journal
Jun 27 10:51:55 xen1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Jun 27 10:51:55 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen1_fs_start_0 (call=11, rc=0, cib-update=17,
confirmed=true) ok
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=71:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=xen1_fs_monitor_10000 )
Jun 27 10:51:55 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=74:2:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=vm1xen1_start_0 )
Jun 27 10:51:55 xen1 lrmd: [3949]: info: rsc:vm1xen1:13: start
Jun 27 10:51:56 xen1 lrmd: [3949]: info: RA output:
(vm1xen1:start:stderr) Error: Unable to open config file: /xen1/vm1
Jun 27 10:51:56 xen1 lrmd: [3949]: info: RA output:
(vm1xen1:start:stdout) Usage: xm create <ConfigFile> [options] [vars]
Create a domain based on <ConfigFile>. Options: -h, --help
Print this help. --help_config Print the available
configuration variables (vars) for the
configuration script. -q, --quiet Quiet. --path=PATH
Search path for configuration scripts. The value of
PATH is a colon-separated directory list. -f=FILE, --defconfig=FILE
Use the given Python configuration script.The
configuration script is loaded after arguments have
been processed. Each command-line option sets a
configuration variable named after its long option
name, and these variables are placed in the
environment of the script before it is loaded.
Variables for options that may be repeated have list
Jun 27 10:51:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation vm1xen1_start_0 (call=13, rc=1, cib-update=18,
confirmed=true) unknown error
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:51:56 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for fail-count-vm1xen1
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-vm1xen1 (INFINITY)
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 25: fail-count-vm1xen1=INFINITY
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:51:56 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for last-failure-vm1xen1
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-vm1xen1 (1277650316)
Jun 27 10:51:56 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 28: last-failure-vm1xen1=1277650316
Jun 27 10:52:25 xen1 lrmd: [3949]: info: rsc:xen1_fs:12: monitor
Jun 27 10:52:25 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen1_fs_monitor_10000 (call=12, rc=0, cib-update=19,
confirmed=false) ok
Jun 27 10:52:25 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=2:3:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=vm1xen1_stop_0 )
Jun 27 10:52:25 xen1 lrmd: [3949]: info: rsc:vm1xen1:14: stop
Jun 27 10:52:26 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=96:3:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:52:26 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:15: notify
Jun 27 10:52:26 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:26 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=15, rc=0, cib-update=20,
confirmed=true) ok
Jun 27 10:52:26 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for fail-count-vm1xen2
Jun 27 10:52:26 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for last-failure-vm1xen2
Jun 27 10:52:26 xen1 Xen[4363]: INFO: Xen domain vm1 already stopped.
Jun 27 10:52:26 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation vm1xen1_stop_0 (call=14, rc=0, cib-update=21,
confirmed=true) ok
Jun 27 10:52:56 xen1 lrmd: [3949]: info: cancel_op: operation
monitor[12] on ocf::Filesystem::xen1_fs for client 3952, its
parameters: CRM_meta_interval=[10000] directory=[/xen1] fstype=[ext3]
CRM_meta_start_delay=[30000] device=[/dev/drbd1]
CRM_meta_timeout=[50000] crm_feature_set=[3.0.1]
CRM_meta_name=[monitor] cancelled
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=73:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=xen1_fs_stop_0 )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:xen1_fs:16: stop
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=95:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:17: notify
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen1_fs_monitor_10000 (call=12, status=1, cib-update=0,
confirmed=true) Cancelled
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=17, rc=0, cib-update=22,
confirmed=true) ok
Jun 27 10:52:56 xen1 Filesystem[4537]: INFO: Running stop for
/dev/drbd1 on /xen1
Jun 27 10:52:56 xen1 Filesystem[4537]: INFO: Trying to unmount /xen1
Jun 27 10:52:56 xen1 Filesystem[4537]: INFO: unmounted /xen1 successfully
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation xen1_fs_stop_0 (call=16, rc=0, cib-update=23,
confirmed=true) ok
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=10:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_demote_0 )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:18: demote
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:demote:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:demote:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 kernel: block drbd1: role( Primary -> Secondary )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:demote:stdout)
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_demote_0 (call=18, rc=0, cib-update=24,
confirmed=true) ok
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=96:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:19: notify
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:notify:stdout)
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=19, rc=0, cib-update=25,
confirmed=true) ok
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=88:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen1:0_notify_0 )
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:20: notify
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_notify_0 (call=20, rc=0, cib-update=26,
confirmed=true) ok
Jun 27 10:52:56 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=11:4:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=drbd_xen1:0_stop_0
)
Jun 27 10:52:56 xen1 lrmd: [3949]: info: rsc:drbd_xen1:0:21: stop
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:stop:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:stop:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 kernel: block drbd1: peer( Secondary -> Unknown )
conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
Jun 27 10:52:56 xen1 kernel: block drbd1: short read expecting header
on sock: r=-512
Jun 27 10:52:56 xen1 kernel: block drbd1: asender terminated
Jun 27 10:52:56 xen1 kernel: block drbd1: Terminating asender thread
Jun 27 10:52:56 xen1 kernel: block drbd1: Connection closed
Jun 27 10:52:56 xen1 kernel: block drbd1: conn( Disconnecting -> StandAlone )
Jun 27 10:52:56 xen1 kernel: block drbd1: receiver terminated
Jun 27 10:52:56 xen1 kernel: block drbd1: Terminating receiver thread
Jun 27 10:52:56 xen1 kernel: block drbd1: disk( UpToDate -> Diskless )
Jun 27 10:52:56 xen1 kernel: block drbd1: drbd_bm_resize called with
capacity == 0
Jun 27 10:52:56 xen1 kernel: block drbd1: worker terminated
Jun 27 10:52:56 xen1 kernel: block drbd1: Terminating worker thread
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:stop:stdout)
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output:
(drbd_xen1:0:stop:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:56 xen1 kernel: block drbd1: State change failed: Disk
state is lower than outdated
Jun 27 10:52:56 xen1 kernel: block drbd1: state = { cs:StandAlone
ro:Secondary/Unknown ds:Diskless/DUnknown r--- }
Jun 27 10:52:56 xen1 kernel: block drbd1: wanted = { cs:StandAlone
ro:Secondary/Unknown ds:Outdated/DUnknown r--- }
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:stop:stdout)
Jun 27 10:52:56 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: master-drbd_xen1:0 (-INFINITY)
Jun 27 10:52:56 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 32: master-drbd_xen1:0=-INFINITY
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:stop:stdout)
Jun 27 10:52:56 xen1 crm_attribute: [4746]: info: Invoked:
crm_attribute -N xen1.box.com -n master-drbd_xen1:0 -l reboot -D
Jun 27 10:52:56 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: master-drbd_xen1:0 (<null>)
Jun 27 10:52:56 xen1 attrd: [3950]: info: attrd_perform_update: Sent
delete 34: node=xen1.box.com, attr=master-drbd_xen1:0, id=<n/a>,
set=(null), section=status
Jun 27 10:52:56 xen1 attrd: [3950]: info: attrd_perform_update: Sent
delete 36: node=xen1.box.com, attr=master-drbd_xen1:0, id=<n/a>,
set=(null), section=status
Jun 27 10:52:56 xen1 lrmd: [3949]: info: RA output: (drbd_xen1:0:stop:stdout)
Jun 27 10:52:56 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen1:0_stop_0 (call=21, rc=0, cib-update=27,
confirmed=true) ok
Jun 27 10:52:57 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for fail-count-drbd_xen1:0
Jun 27 10:52:57 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for last-failure-drbd_xen1:0
Jun 27 10:52:58 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=41:7:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=drbd_xen2:1_start_0
)
Jun 27 10:52:58 xen1 lrmd: [3949]: info: rsc:drbd_xen2:1:22: start
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:start:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:start:stderr) 'xen2' not defined in your config.
Jun 27 10:52:58 xen1 drbd[4747]: ERROR: DRBD resource xen2 not found
in configuration file /etc/drbd.conf.
Jun 27 10:52:58 xen1 crm_attribute: [4774]: info: Invoked:
crm_attribute -N xen1.box.com -n master-drbd_xen2:1 -l reboot -D
Jun 27 10:52:58 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for master-drbd_xen2:1
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output: (drbd_xen2:1:start:stdout)
Jun 27 10:52:58 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen2:1_start_0 (call=22, rc=5, cib-update=28,
confirmed=true) not installed
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:52:58 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for fail-count-drbd_xen2:1
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-drbd_xen2:1 (INFINITY)
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 45: fail-count-drbd_xen2:1=INFINITY
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:52:58 xen1 attrd: [3950]: info: find_hash_entry: Creating
hash entry for last-failure-drbd_xen2:1
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-drbd_xen2:1
(1277650378)
Jun 27 10:52:58 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=97:7:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen2:1_notify_0 )
Jun 27 10:52:58 xen1 lrmd: [3949]: info: rsc:drbd_xen2:1:23: notify
Jun 27 10:52:58 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 48: last-failure-drbd_xen2:1=1277650378
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:notify:stderr) 'xen2' not defined in your config.
Jun 27 10:52:58 xen1 drbd[4781]: ERROR: DRBD resource xen2 not found
in configuration file /etc/drbd.conf.
Jun 27 10:52:58 xen1 crm_attribute: [4808]: info: Invoked:
crm_attribute -N xen1.box.com -n master-drbd_xen2:1 -l reboot -D
Jun 27 10:52:58 xen1 lrmd: [3949]: info: RA output: (drbd_xen2:1:notify:stdout)
Jun 27 10:52:58 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen2:1_notify_0 (call=23, rc=0, cib-update=29,
confirmed=true) ok
Jun 27 10:53:00 xen1 cib: [4809]: info: write_cib_contents: Archived
previous version as /var/lib/heartbeat/crm/cib-63.raw
Jun 27 10:53:00 xen1 cib: [4809]: info: write_cib_contents: Wrote
version 0.80.0 of the CIB to disk (digest:
4103d849359e1962585c9d84e56ce3f0)
Jun 27 10:53:00 xen1 cib: [4809]: info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.bteVor (digest:
/var/lib/heartbeat/crm/cib.td8DMI)
Jun 27 10:53:00 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=100:8:0:e8d1e361-2956-49c9-9c49-9babc795edc6
op=drbd_xen2:1_notify_0 )
Jun 27 10:53:00 xen1 lrmd: [3949]: info: rsc:drbd_xen2:1:24: notify
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:notify:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:notify:stderr) 'xen2' not defined in your config.
Jun 27 10:53:00 xen1 drbd[4810]: ERROR: DRBD resource xen2 not found
in configuration file /etc/drbd.conf.
Jun 27 10:53:00 xen1 crm_attribute: [4837]: info: Invoked:
crm_attribute -N xen1.box.com -n master-drbd_xen2:1 -l reboot -D
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output: (drbd_xen2:1:notify:stdout)
Jun 27 10:53:00 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen2:1_notify_0 (call=24, rc=0, cib-update=30,
confirmed=true) ok
Jun 27 10:53:00 xen1 crmd: [3952]: info: do_lrm_rsc_op: Performing
key=1:8:0:e8d1e361-2956-49c9-9c49-9babc795edc6 op=drbd_xen2:1_stop_0 )
Jun 27 10:53:00 xen1 lrmd: [3949]: info: rsc:drbd_xen2:1:25: stop
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:stop:stderr) DRBD module version: 8.3.8 userland
version: 8.3.6 you should upgrade your drbd tools!
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output:
(drbd_xen2:1:stop:stderr) 'xen2' not defined in your config.
Jun 27 10:53:00 xen1 drbd[4838]: ERROR: DRBD resource xen2 not found
in configuration file /etc/drbd.conf.
Jun 27 10:53:00 xen1 crm_attribute: [4865]: info: Invoked:
crm_attribute -N xen1.box.com -n master-drbd_xen2:1 -l reboot -D
Jun 27 10:53:00 xen1 lrmd: [3949]: info: RA output: (drbd_xen2:1:stop:stdout)
Jun 27 10:53:00 xen1 crmd: [3952]: info: process_lrm_event: LRM
operation drbd_xen2:1_stop_0 (call=25, rc=5, cib-update=31,
confirmed=true) not installed
Jun 27 10:53:00 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:53:00 xen1 attrd: [3950]: info: attrd_ais_dispatch: Update
relayed from fa1.box.com
Jun 27 10:53:00 xen1 attrd: [3950]: info: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-drbd_xen2:1
(1277650381)
Jun 27 10:53:00 xen1 attrd: [3950]: info: attrd_perform_update: Sent
update 51: last-failure-drbd_xen2:1=1277650381
Jun 27 10:53:10 xen1 ntpd[2996]: synchronized to LOCAL(0), stratum 10
Jun 27 10:53:10 xen1 ntpd[2996]: kernel time sync enabled 0001
New config is:
node fa1.box.com
node xen1.box.com
node xen2.box.com
primitive drbd_xen1 ocf:linbit:drbd \
params drbd_resource="xen1" \
op monitor interval="15s"
primitive drbd_xen2 ocf:linbit:drbd \
params drbd_resource="xen2" \
op monitor interval="15s"
primitive vm1xen1 ocf:heartbeat:Xen \
params xmfile="/xen1/vm1" \
op monitor interval="0s" timeout="60s" start-delay="5s"
depth="0" target-role="Stoppedop" start \
op stop interval="0s" timeout="120s" \
meta is-managed="true" \
meta target-role="Started"
primitive vm1xen2 ocf:heartbeat:Xen \
params xmfile="/xen2/vm1" \
op monitor interval="0s" timeout="60s" start-delay="5s"
depth="0" target-role="Stoppedop" start \
op stop interval="0s" timeout="120s" \
meta is-managed="true" \
meta target-role="Started"
primitive xen1_fs ocf:heartbeat:Filesystem \
params device="/dev/drbd1" directory="/xen1" fstype="ext3" \
op monitor start-delay="30s" interval="10s"
primitive xen2_fs ocf:heartbeat:Filesystem \
params device="/dev/drbd2" directory="/xen2" fstype="ext3" \
op monitor start-delay="30s" interval="10s"
ms ms_drbd_xen1 drbd_xen1 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true"
ms ms_drbd_xen2 drbd_xen2 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true"
location drbd-fence-by-handler-ms_drbd_xen1 ms_drbd_xen1 \
rule $id="drbd-fence-by-handler-rule-ms_drbd_xen1"
$role="Master" -inf: #uname ne fa1.box.com
location drbd-fence-by-handler-ms_drbd_xen2 ms_drbd_xen2 \
rule $id="drbd-fence-by-handler-rule-ms_drbd_xen2"
$role="Master" -inf: #uname ne xen2.box.com
colocation fs_on_drbd1 inf: xen1_fs ms_drbd_xen1:Master
colocation fs_on_drbd2 inf: xen2_fs ms_drbd_xen2:Master
colocation vm1xen1-with-xen1_fs inf: vm1xen1 xen1_fs
colocation vm1xen2-with-xen2_fs inf: vm1xen2 xen2_fs
order fs_after_drbd1 inf: ms_drbd_xen1:promote xen1_fs:start
order fs_after_drbd2 inf: ms_drbd_xen2:promote xen2_fs:start
order vm1xen1-after-xen1_fs inf: xen1_fs:start vm1xen1:start
order vm1xen2-after-xen2_fs inf: xen2_fs:start vm1xen2:start
property $id="cib-bootstrap-options" \
dc-version="1.0.9-89bd754939df5150de7cd76835f98fe90851b677" \
cluster-infrastructure="openais" \
expected-quorum-votes="3" \
no-quorum-policy="ignore" \
stonith-enabled="false" \
default-resource-stickiness="1000" \
last-lrm-refresh="1277632456"
Although before it used to it very quickly , but now it kinda acts
differently, just I don't like it that when it mounts the partition as
you can see from the logs it starts the VPS very very quickly, right
after the mount, that is why it can't see the config file, if
pacemaker could've given it 3-4 seconds it might have worked.
Kinda confused.
Thanks
Joe
On Sun, Jun 27, 2010 at 7:34 AM, Greg Woods <[email protected]> wrote:
> On Sun, 2010-06-27 at 03:02 -0700, Joe Shang wrote:
>
>> Failed actions:
>> drbd_xen2:1_start_0 (node=xen1.box.com, call=10, rc=5,
>> status=complete): not installed
>
> This is one of the things that I don't like about heartbeat/pacemaker. A
> minor error (misconfiguring a single resource) can cause major problems
> (like a stonith death match that brings down the entire cluster).
>
> One thing I have seen with Xen VMs is that the default timeouts are too
> short. That may not be your particular problem, but you probably need to
> increase them anyway. This is an example of what I have:
>
>
>
> primitive VM-ldap ocf:heartbeat:Xen \
> params xmfile="/etc/xen/ldap" \
> op monitor interval="10" timeout="120" depth="0"
> target-role="Stopped" \
> op start interval="0" timeout="60s" \
> op stop interval="0" timeout="120s" \
> meta is-managed="true" target-role="Started"
>
> Before I added the explicit "op start" and "op stop" timeouts, I woulod
> get failed stop or start operations and any attempt to fail over would
> start a death match.
>
> --Greg
>
>
> _______________________________________________
> Linux-HA mailing list
> [email protected]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
