----- Original Message ----- > From: "Tim Small" <t...@seoss.co.uk> > To: pacemaker@oss.clusterlabs.org > Sent: Friday, April 27, 2012 5:31:23 AM > Subject: [Pacemaker] ERROR: te_graph_trigger: Transition failed: terminated > pacemaker's problem or mine? > > Hi, > > I'm trying to get to the bottom of a problem I'm seeing with a > cluster. > At this stage I'm unclear as to whether the issue is with the config > or > not - the generated error messages seem unclear. So I'm not sure > whether I should be staring at the config or the source code at this > point, and would appreciate a clue! > > I'm running with some of the (live) resources in an unmanaged state > whilst testing fail-over with other (non-dependant) resources. >
I'm betting this transition error is a result of how the un-managed resources are used in the colocation and order constraints with the managed resources. Can you produce a hb_report/crm_report for this. It isn't obvious (at least to me) what is causing this by looking at the logs. -- Vossel > The managed resources are a number of OpenVZ virtual machines (each > comprising 3 primitives - file-system + OpenVZ VE + SendArp). The > filesystems are on LVM volume groups, and the single LVM PV for each > volume group resides on a DRBD volume. There are n virtual machines > per > DRBD volume. > > I'm running pacemaker 1.0.9.1+hg15626-1 on Debian 6.0. Here are some > of > the messages (configuration follows at the end of the email): > > > Apr 27 11:06:35 fig crmd: [395]: info: crm_timer_popped: PEngine > Recheck > Timer (I_PE_CALC) just popped! > Apr 27 11:06:35 fig crmd: [395]: info: do_state_transition: State > transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC > cause=C_TIMER_POPPED origin=crm_timer_popped ] > Apr 27 11:06:35 fig crmd: [395]: info: do_state_transition: > Progressed > to state S_POLICY_ENGINE after C_TIMER_POPPED > Apr 27 11:06:35 fig crmd: [395]: info: do_state_transition: All 2 > cluster nodes are eligible to run resources. > Apr 27 11:06:35 fig crmd: [395]: info: do_pe_invoke: Query 985: > Requesting the current CIB: S_POLICY_ENGINE > Apr 27 11:06:35 fig crmd: [395]: info: do_pe_invoke_callback: > Invoking > the PE: query=985, ref=pe_calc-dc-1335521195-1437, seq=184, quorate=1 > Apr 27 11:06:35 fig pengine: [394]: notice: unpack_config: On loss of > CCM Quorum: Ignore > Apr 27 11:06:35 fig pengine: [394]: info: unpack_config: Node scores: > 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 > Apr 27 11:06:35 fig pengine: [394]: info: determine_online_status: > Node > fig is online > Apr 27 11:06:35 fig pengine: [394]: info: determine_online_status: > Node > hazel is online > Apr 27 11:06:35 fig pengine: [394]: info: native_add_running: > resource > essex03-LVM isnt managed > Apr 27 11:06:35 fig pengine: [394]: info: native_add_running: > resource > calypso-FS isnt managed > Apr 27 11:06:35 fig pengine: [394]: info: native_add_running: > resource > calypso-VE isnt managed > Apr 27 11:06:35 fig pengine: [394]: notice: unpack_rsc_op: Operation > essex03-DRBD:0_monitor_0 found resource essex03-DRBD:0 active in > master > mode on hazel > Apr 27 11:06:35 fig pengine: [394]: info: native_add_running: > resource > essex03-DRBD:0 isnt managed > Apr 27 11:06:35 fig pengine: [394]: notice: unpack_rsc_op: Operation > essex02-DRBD:1_monitor_0 found resource essex02-DRBD:1 active in > master > mode on hazel > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > artemis-FS#011(ocf::heartbeat:Filesystem):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > artemis-SendArp#011(ocf::heartbeat:SendArp):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > artemis-VE#011(ocf::heartbeat:ManageVE):#011Stopped (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > athena-FS#011(ocf::heartbeat:Filesystem):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > athena-SendArp#011(ocf::heartbeat:SendArp):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > athena-VE#011(ocf::heartbeat:ManageVE):#011Stopped (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > calypso-FS#011(ocf::heartbeat:Filesystem):#011Started hazel > (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > calypso-SendArp#011(ocf::heartbeat:SendArp):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > calypso-VE#011(ocf::heartbeat:ManageVE):#011Started hazel (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > epione-FS#011(ocf::heartbeat:Filesystem):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > epione-SendArp#011(ocf::heartbeat:SendArp):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > epione-VE#011(ocf::heartbeat:ManageVE):#011Stopped (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essex02-LVM#011(ocf::heartbeat:LVM):#011Started hazel > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essex03-LVM#011(ocf::heartbeat:LVM):#011Started hazel (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essextest-FS#011(ocf::heartbeat:Filesystem):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essextest-SendArp#011(ocf::heartbeat:SendArp):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essextest-VE#011(ocf::heartbeat:ManageVE):#011Stopped > Apr 27 11:06:35 fig pengine: [394]: notice: clone_print: > Master/Slave > Set: ms-drbd-essex02 > Apr 27 11:06:35 fig pengine: [394]: notice: short_print: > Masters: [ > hazel ] > Apr 27 11:06:35 fig pengine: [394]: notice: short_print: Slaves: > [ > fig ] > Apr 27 11:06:35 fig pengine: [394]: notice: clone_print: > Master/Slave > Set: ms-drbd-essex03 (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: native_print: > essex03-DRBD:0#011(ocf::linbit:drbd):#011Master hazel (unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: short_print: > Stopped: [ > essex03-DRBD:1 ] > Apr 27 11:06:35 fig pengine: [394]: info: get_failcount: > ms-drbd-essex03 > has failed 1 times on hazel > Apr 27 11:06:35 fig pengine: [394]: notice: common_apply_stickiness: > ms-drbd-essex03 can fail 999999 more times on hazel before being > forced off > Apr 27 11:06:35 fig pengine: [394]: info: get_failcount: > ms-drbd-essex03 > has failed 1 times on hazel > Apr 27 11:06:35 fig pengine: [394]: notice: common_apply_stickiness: > ms-drbd-essex03 can fail 999999 more times on hazel before being > forced off > Apr 27 11:06:35 fig pengine: [394]: info: master_color: Promoting > essex02-DRBD:0 (Slave fig) > Apr 27 11:06:35 fig pengine: [394]: info: master_color: > ms-drbd-essex02: > Promoted 1 instances of a possible 1 to master > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource artemis-VE allocated to 'nowhere': inactive > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Resource > artemis-SendArp cannot run anywhere > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource athena-VE allocated to 'nowhere': inactive > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Resource > athena-SendArp cannot run anywhere > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource essex03-DRBD:0 allocated to hazel: master > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource essex03-DRBD:1 allocated to 'nowhere': inactive > Apr 27 11:06:35 fig pengine: [394]: notice: can_be_master: Forcing > unmanaged master essex03-DRBD:0 to remain promoted on hazel > Apr 27 11:06:35 fig pengine: [394]: info: master_color: Promoting > essex03-DRBD:0 (Master hazel) > Apr 27 11:06:35 fig pengine: [394]: info: master_color: > ms-drbd-essex03: > Promoted 1 instances of a possible 1 to master > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource essex03-LVM allocated to hazel: active > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource calypso-FS allocated to hazel: active > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource calypso-VE allocated to hazel: active > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Unmanaged > resource epione-VE allocated to 'nowhere': inactive > Apr 27 11:06:35 fig pengine: [394]: info: native_color: Resource > epione-SendArp cannot run anywhere > Apr 27 11:06:35 fig pengine: [394]: info: master_color: Promoting > essex02-DRBD:0 (Slave fig) > Apr 27 11:06:35 fig pengine: [394]: info: master_color: > ms-drbd-essex02: > Promoted 1 instances of a possible 1 to master > Apr 27 11:06:35 fig pengine: [394]: notice: can_be_master: Forcing > unmanaged master essex03-DRBD:0 to remain promoted on hazel > Apr 27 11:06:35 fig pengine: [394]: info: master_color: Promoting > essex03-DRBD:0 (Master hazel) > Apr 27 11:06:35 fig pengine: [394]: info: master_color: > ms-drbd-essex03: > Promoted 1 instances of a possible 1 to master > Apr 27 11:06:35 fig pengine: [394]: notice: RecurringOp: Start > recurring monitor (29s) for essex02-DRBD:0 on fig > Apr 27 11:06:35 fig pengine: [394]: notice: RecurringOp: Start > recurring monitor (31s) for essex02-DRBD:1 on hazel > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex02 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex02 > Apr 27 11:06:35 fig pengine: [394]: notice: RecurringOp: Start > recurring monitor (29s) for essex02-DRBD:0 on fig > Apr 27 11:06:35 fig pengine: [394]: notice: RecurringOp: Start > recurring monitor (31s) for essex02-DRBD:1 on hazel > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex02 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex02 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex03 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex03 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex03 > Apr 27 11:06:35 fig pengine: [394]: ERROR: > create_notification_boundaries: Creating boundaries for > ms-drbd-essex03 > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > artemis-FS#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > artemis-SendArp#011(Stopped) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > artemis-VE#011(Stopped unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > athena-FS#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > athena-SendArp#011(Stopped) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > athena-VE#011(Stopped unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > calypso-FS#011(Started unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > calypso-SendArp#011(hazel) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > calypso-VE#011(Started unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > epione-FS#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > epione-SendArp#011(Stopped) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > epione-VE#011(Stopped unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Move resource > essex02-LVM#011(Started hazel -> fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > essex03-LVM#011(Started unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > essextest-FS#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > essextest-SendArp#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Start > essextest-VE#011(fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Promote > essex02-DRBD:0#011(Slave -> Master fig) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Demote > essex02-DRBD:1#011(Master -> Slave hazel) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > essex03-DRBD:0#011(Master unmanaged) > Apr 27 11:06:35 fig pengine: [394]: notice: LogActions: Leave > resource > essex03-DRBD:1#011(Stopped unmanaged) > Apr 27 11:06:35 fig crmd: [395]: info: do_state_transition: State > transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ > input=I_PE_SUCCESS > cause=C_IPC_MESSAGE origin=handle_response ] > Apr 27 11:06:35 fig crmd: [395]: info: unpack_graph: Unpacked > transition > 121: 34 actions in 34 synapses > Apr 27 11:06:35 fig crmd: [395]: info: do_te_invoke: Processing graph > 121 (ref=pe_calc-dc-1335521195-1437) derived from > /var/lib/pengine/pe-input-82.bz2 > Apr 27 11:06:35 fig crmd: [395]: info: te_pseudo_action: Pseudo > action > 53 fired and confirmed > Apr 27 11:06:35 fig pengine: [394]: info: process_pe_message: > Transition > 121: PEngine Input stored in: /var/lib/pengine/pe-input-82.bz2 > Apr 27 11:06:35 fig crmd: [395]: info: te_rsc_command: Initiating > action > 119: notify essex02-DRBD:0_pre_notify_demote_0 on fig (local) > Apr 27 11:06:35 fig crmd: [395]: info: do_lrm_rsc_op: Performing > key=119:121:0:3dcf0e40-239f-4eed-9764-9c29003fa305 > op=essex02-DRBD:0_notify_0 ) > Apr 27 11:06:35 fig lrmd: [392]: info: rsc:essex02-DRBD:0:594: notify > Apr 27 11:06:35 fig crmd: [395]: info: te_rsc_command: Initiating > action > 121: notify essex02-DRBD:1_pre_notify_demote_0 on hazel > Apr 27 11:06:36 fig crmd: [395]: info: match_graph_event: Action > essex02-DRBD:1_pre_notify_demote_0 (121) confirmed on hazel (rc=0) > Apr 27 11:06:36 fig crmd: [395]: info: process_lrm_event: LRM > operation > essex02-DRBD:0_notify_0 (call=594, rc=0, cib-update=986, > confirmed=true) ok > Apr 27 11:06:36 fig crmd: [395]: info: match_graph_event: Action > essex02-DRBD:0_pre_notify_demote_0 (119) confirmed on fig (rc=0) > Apr 27 11:06:36 fig crmd: [395]: info: te_pseudo_action: Pseudo > action > 54 fired and confirmed > Apr 27 11:06:36 fig crmd: [395]: notice: run_graph: > ==================================================== > Apr 27 11:06:36 fig crmd: [395]: WARN: run_graph: Transition 121 > (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=30, > Source=/var/lib/pengine/pe-input-82.bz2): Terminated > Apr 27 11:06:36 fig crmd: [395]: ERROR: te_graph_trigger: Transition > failed: terminated > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Graph 121 (34 > actions in 34 synapses): batch-limit=30 jobs, network-delay=60000ms > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 0 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 6]: > Pending (id: artemis-FS_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 18]: > Pending (id: essex02-LVM_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 1 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 8]: > Pending (id: athena-FS_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 18]: > Pending (id: essex02-LVM_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 2 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 12]: > Pending (id: calypso-SendArp_start_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 14]: > Pending (id: calypso-VE_start_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 3 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 15]: > Pending (id: epione-FS_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 18]: > Pending (id: essex02-LVM_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 4 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 17]: > Pending (id: essex02-LVM_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 10]: > Pending (id: calypso-FS_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 5 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 18]: > Pending (id: essex02-LVM_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 17]: > Pending (id: essex02-LVM_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 27]: > Pending (id: essex02-DRBD:0_promote_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 46]: > Pending (id: ms-drbd-essex02_promoted_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 50]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_promoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 6 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 21]: > Pending (id: essextest-FS_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 18]: > Pending (id: essex02-LVM_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 7 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 22]: > Pending (id: essextest-SendArp_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 23]: > Pending (id: essextest-VE_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 8 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 23]: > Pending (id: essextest-VE_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 21]: > Pending (id: essextest-FS_start_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 9 is > pending > (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 27]: > Pending (id: essex02-DRBD:0_promote_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 45]: > Pending (id: ms-drbd-essex02_promote_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 10 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 28]: > Pending (id: essex02-DRBD:0_monitor_29000, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 27]: > Pending (id: essex02-DRBD:0_promote_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 50]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_promoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 56]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_demoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 11 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 115]: > Pending (id: essex02-DRBD:0_pre_notify_promote_0, loc: fig, priority: > 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 47]: > Pending (id: ms-drbd-essex02_pre_notify_promote_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 12 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 116]: > Pending (id: essex02-DRBD:0_post_notify_promote_0, loc: fig, > priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 49]: > Pending (id: ms-drbd-essex02_post_notify_promoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 13 was > confirmed (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 14 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 120]: > Pending (id: essex02-DRBD:0_post_notify_demote_0, loc: fig, priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 55]: > Pending (id: ms-drbd-essex02_post_notify_demoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 15 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 31]: > Pending (id: essex02-DRBD:1_demote_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 17]: > Pending (id: essex02-LVM_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 51]: > Pending (id: ms-drbd-essex02_demote_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 16 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 32]: > Pending (id: essex02-DRBD:1_monitor_31000, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 31]: > Pending (id: essex02-DRBD:1_demote_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 50]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_promoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 56]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_demoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 17 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 117]: > Pending (id: essex02-DRBD:1_pre_notify_promote_0, loc: hazel, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 47]: > Pending (id: ms-drbd-essex02_pre_notify_promote_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 18 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 118]: > Pending (id: essex02-DRBD:1_post_notify_promote_0, loc: hazel, > priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 49]: > Pending (id: ms-drbd-essex02_post_notify_promoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 19 was > confirmed (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 20 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 122]: > Pending (id: essex02-DRBD:1_post_notify_demote_0, loc: hazel, > priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 55]: > Pending (id: ms-drbd-essex02_post_notify_demoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 21 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 45]: > Pending (id: ms-drbd-essex02_promote_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 48]: > Pending (id: ms-drbd-essex02_confirmed-pre_notify_promote_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 52]: > Pending (id: ms-drbd-essex02_demoted_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 22 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 46]: > Pending (id: ms-drbd-essex02_promoted_0, type: pseduo, priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 27]: > Pending (id: essex02-DRBD:0_promote_0, loc: fig, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 23 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 47]: > Pending (id: ms-drbd-essex02_pre_notify_promote_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 56]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_demoted_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 24 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 48]: > Pending (id: ms-drbd-essex02_confirmed-pre_notify_promote_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 47]: > Pending (id: ms-drbd-essex02_pre_notify_promote_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 115]: > Pending (id: essex02-DRBD:0_pre_notify_promote_0, loc: fig, priority: > 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 117]: > Pending (id: essex02-DRBD:1_pre_notify_promote_0, loc: hazel, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 25 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 49]: > Pending (id: ms-drbd-essex02_post_notify_promoted_0, type: pseduo, > priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 46]: > Pending (id: ms-drbd-essex02_promoted_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 48]: > Pending (id: ms-drbd-essex02_confirmed-pre_notify_promote_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 26 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 50]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_promoted_0, type: > pseduo, priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 49]: > Pending (id: ms-drbd-essex02_post_notify_promoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 116]: > Pending (id: essex02-DRBD:0_post_notify_promote_0, loc: fig, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 118]: > Pending (id: essex02-DRBD:1_post_notify_promote_0, loc: hazel, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 27 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 51]: > Pending (id: ms-drbd-essex02_demote_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 17]: > Pending (id: essex02-LVM_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 54]: > Completed (id: ms-drbd-essex02_confirmed-pre_notify_demote_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 28 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 52]: > Pending (id: ms-drbd-essex02_demoted_0, type: pseduo, priority: > 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 31]: > Pending (id: essex02-DRBD:1_demote_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 51]: > Pending (id: ms-drbd-essex02_demote_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 29 was > confirmed (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 30 was > confirmed (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 31 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 55]: > Pending (id: ms-drbd-essex02_post_notify_demoted_0, type: pseduo, > priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 52]: > Pending (id: ms-drbd-essex02_demoted_0, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 54]: > Completed (id: ms-drbd-essex02_confirmed-pre_notify_demote_0, type: > pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 32 is > pending (priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 56]: > Pending (id: ms-drbd-essex02_confirmed-post_notify_demoted_0, type: > pseduo, priority: 1000000) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 55]: > Pending (id: ms-drbd-essex02_post_notify_demoted_0, type: pseduo, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 120]: > Pending (id: essex02-DRBD:0_post_notify_demote_0, loc: fig, priority: > 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input > 122]: > Pending (id: essex02-DRBD:1_post_notify_demote_0, loc: hazel, > priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_graph: Synapse 33 is > pending (priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: [Action 2]: > Pending (id: all_stopped, type: pseduo, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: WARN: print_elem: * [Input 17]: > Pending (id: essex02-LVM_stop_0, loc: hazel, priority: 0) > Apr 27 11:06:36 fig crmd: [395]: info: te_graph_trigger: Transition > 121 > is now complete > Apr 27 11:06:36 fig crmd: [395]: info: notify_crmd: Transition 121 > status: done - <null> > Apr 27 11:06:36 fig crmd: [395]: info: do_state_transition: State > transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS > cause=C_FSA_INTERNAL origin=notify_crmd ] > Apr 27 11:06:36 fig crmd: [395]: info: do_state_transition: Starting > PEngine Recheck Timer > > > > The configuration is as follows: > > > node fig > node hazel > primitive artemis-FS ocf:heartbeat:Filesystem \ > params device="/dev/essex02/artemis" > directory="/containers/artemis" fstype="ext4" > options="relatime,discard" \ > op start interval="0" timeout="60s" \ > op start interval="0" timeout="60s" > primitive artemis-SendArp ocf:heartbeat:SendArp \ > params ip="XYZ.XY.126.102" nic="eth0" > primitive artemis-VE ocf:heartbeat:ManageVE \ > params veid="2501" \ > op start interval="0" timeout="75s" \ > op start interval="0" timeout="75s" \ > meta target-role="Stopped" is-managed="false" > primitive athena-FS ocf:heartbeat:Filesystem \ > params device="/dev/essex02/athena" > directory="/containers/athena" fstype="ext4" > options="relatime,discard" \ > op start interval="0" timeout="60s" \ > op start interval="0" timeout="60s" \ > meta is-managed="true" target-role="Started" > primitive athena-SendArp ocf:heartbeat:SendArp \ > params ip="XYZ.XY.118.23" nic="eth0" > primitive athena-VE ocf:heartbeat:ManageVE \ > params veid="2504" \ > op start interval="0" timeout="75s" \ > op start interval="0" timeout="75s" \ > meta target-role="Stopped" is-managed="false" > primitive calypso-FS ocf:heartbeat:Filesystem \ > params device="/dev/essex03/calypso" > directory="/containers/calypso" fstype="ext4" > options="relatime,discard" \ > op start interval="0" timeout="60s" \ > op start interval="0" timeout="60s" \ > meta target-role="Started" is-managed="false" > primitive calypso-SendArp ocf:heartbeat:SendArp \ > params ip="XYZ.XY.126.110" nic="eth0" > primitive calypso-VE ocf:heartbeat:ManageVE \ > params veid="2000" \ > op start interval="0" timeout="75s" \ > op start interval="0" timeout="75s" \ > meta target-role="Stopped" is-managed="false" > primitive epione-FS ocf:heartbeat:Filesystem \ > params device="/dev/essex02/epione" > directory="/containers/epione" fstype="ext4" > options="relatime,discard" \ > op start interval="0" timeout="60s" \ > op start interval="0" timeout="60s" \ > meta target-role="Started" > primitive epione-SendArp ocf:heartbeat:SendArp \ > params ip="XYZ.XY.118.21" nic="eth0" > primitive epione-VE ocf:heartbeat:ManageVE \ > params veid="2502" \ > op start interval="0" timeout="75s" \ > op start interval="0" timeout="75s" \ > meta target-role="Stopped" is-managed="false" > primitive essex02-DRBD ocf:linbit:drbd \ > params drbd_resource="essex02" \ > op monitor interval="29s" role="Master" \ > op monitor interval="31s" role="Slave" \ > op start interval="0" timeout="240s" \ > op start interval="0" timeout="100s" > primitive essex02-LVM ocf:heartbeat:LVM \ > params volgrpname="essex02" \ > op start interval="0" timeout="240s" \ > op start interval="0" timeout="100s" \ > meta target-role="Started" is-managed="true" > primitive essex03-DRBD ocf:linbit:drbd \ > params drbd_resource="essex03" \ > op monitor interval="29s" role="Master" \ > op monitor interval="31s" role="Slave" \ > op start interval="0" timeout="240s" \ > meta target-role="Started" is-managed="false" > primitive essex03-LVM ocf:heartbeat:LVM \ > params volgrpname="essex03" \ > op start interval="0" timeout="240s" \ > op start interval="0" timeout="100s" \ > meta target-role="Started" is-managed="false" > primitive essextest-FS ocf:heartbeat:Filesystem \ > params device="/dev/essex02/essextest" > directory="/containers/essextest" fstype="ext4" > options="relatime,discard" \ > op start interval="0" timeout="60s" \ > op start interval="0" timeout="60s" \ > meta target-role="Started" > primitive essextest-SendArp ocf:heartbeat:SendArp \ > params ip="XYZ.XY.118.25" nic="eth0" \ > meta target-role="Started" > primitive essextest-VE ocf:heartbeat:ManageVE \ > params veid="2599" \ > op start interval="0" timeout="75s" \ > op start interval="0" timeout="75s" \ > meta target-role="Started" > ms ms-drbd-essex02 essex02-DRBD \ > meta master-max="1" notify="true" globally-unique="false" > target-role="Started" is-managed="true" > ms ms-drbd-essex03 essex03-DRBD \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" globally-unique="false" > target-role="Started" is-managed="false" > location ms-drbd-essex02-on-fig ms-drbd-essex02 \ > rule $id="ms-drbd-essex02-on-fig-rule" $role="master" 100: > #uname eq fig > location ms-drbd-essex03-on-hazel ms-drbd-essex03 \ > rule $id="ms-drbd-essex03-on-hazel-rule" $role="master" 100: > #uname eq hazel > colocation artemis-FS-with-essex02-LVM inf: artemis-FS essex02-LVM > colocation artemis-SendArp-with-artemis-VE inf: artemis-SendArp > artemis-VE > colocation artemis-VE-with-artemis-FS inf: artemis-VE artemis-FS > colocation athena-FS-with-essex02-LVM inf: athena-FS essex02-LVM > colocation athena-SendArp-with-athena-VE inf: athena-SendArp > athena-VE > colocation athena-VE-with-athena-FS inf: athena-VE athena-FS > colocation calypso-FS-on-essex03-LVM inf: calypso-FS essex03-LVM > colocation calypso-SendArp-with-calypso-VE inf: calypso-SendArp > calypso-VE > colocation calypso-VE-with-calypso-FS inf: calypso-VE calypso-FS > colocation epione-FS-on-essex02-LVM inf: epione-FS essex02-LVM > colocation epione-FS-with-essex02-LVM inf: epione-FS essex02-LVM > colocation epione-SendArp-with-epione-VE inf: epione-SendArp > epione-VE > colocation epione-VE-with-epione-FS inf: epione-VE epione-FS > colocation essex02-LVM-with-essex02-DRBD-Master inf: essex02-LVM > ms-drbd-essex02:Master > colocation essex03LVM-on-ms-drbd-essex03 inf: essex03-LVM > ms-drbd-essex03:Master > colocation essextest-FS-with-essex02-LVM inf: essextest-FS > essex02-LVM > colocation essextest-SendArp-with-essextest-VE inf: essextest-SendArp > essextest-VE > colocation essextest-VE-with-essextest-FS inf: essextest-VE > essextest-FS > order artemis-FS-before-artemis-VE inf: artemis-FS artemis-VE > order artemis-VE-before-artemis-SendArp inf: artemis-VE > artemis-SendArp > order athena-FS-before-athena-VE inf: athena-FS athena-VE > order athena-VE-before-athena-SendArp inf: athena-VE athena-SendArp > order calypso-FS-before-calypso-VE inf: calypso-FS calypso-VE > order calypso-VE-before-calypso-SendArp inf: calypso-VE > calypso-SendArp > order epione-FS-before-epione-VE inf: epione-FS epione-VE > order epione-VE-before-epione-SendArp inf: epione-VE epione-SendArp > order essex02-lvm-before-artemis-FS inf: essex02-LVM artemis-FS > order essex02-lvm-before-athena-FS inf: essex02-LVM athena-FS > order essex02-lvm-before-calypso-FS inf: essex02-LVM calypso-FS > order essex02-lvm-before-epione-FS inf: essex02-LVM epione-FS > order essex02-lvm-before-essextest-FS inf: essex02-LVM essextest-FS > order essextest-FS-before-essextest-VE inf: essextest-FS essextest-VE > order essextest-VE-before-essextest-SendArp inf: essextest-VE > essextest-SendArp > order ms-drbd-essex02-before-lvm inf: ms-drbd-essex02:promote > essex02-LVM:start > order ms-drbd-essex03-before-lvm inf: ms-drbd-essex03:promote > essex03-LVM:start > property $id="cib-bootstrap-options" \ > dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="2" \ > no-quorum-policy="ignore" \ > stonith-enabled="false" \ > last-lrm-refresh="1335487560" > > > # crm configure verify > WARNING: artemis-FS: default timeout 20s for stop is smaller than the > advised 60 > WARNING: artemis-VE: default timeout 20s for stop is smaller than the > advised 75 > WARNING: athena-FS: default timeout 20s for stop is smaller than the > advised 60 > WARNING: athena-VE: default timeout 20s for stop is smaller than the > advised 75 > WARNING: calypso-FS: default timeout 20s for stop is smaller than the > advised 60 > WARNING: calypso-VE: default timeout 20s for stop is smaller than the > advised 75 > WARNING: epione-FS: default timeout 20s for stop is smaller than the > advised 60 > WARNING: epione-VE: default timeout 20s for stop is smaller than the > advised 75 > WARNING: essex02-LVM: default timeout 20s for stop is smaller than > the > advised 30 > WARNING: essex03-LVM: default timeout 20s for stop is smaller than > the > advised 30 > WARNING: essextest-FS: default timeout 20s for stop is smaller than > the > advised 60 > WARNING: essextest-VE: default timeout 20s for stop is smaller than > the > advised 75 > WARNING: essex02-DRBD: specified timeout 100s for start is smaller > than > the advised 240 > WARNING: essex02-DRBD: default timeout 20s for stop is smaller than > the > advised 100 > WARNING: essex03-DRBD: default timeout 20s for stop is smaller than > the > advised 100 > > > > > > > -- > South East Open Source Solutions Limited > Registered in England and Wales with company number 06134732. > Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ > VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309 > > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org