Hi! I have this strange heartbeat problem that is complicating my life Im running heartbeat 2.0.8 on debian 2.6 Im using crm Ve4 starting heartbeat on both nodes, i have cib.xml well configured using the python script. After i start heartbeat, node 2 starts normally but i have problems with node 1: ACTUALLY IF I CHECK cib.xml ON NODE1 after starting heartbeat, I FIND IT EMPTY ... ALTHOUGH IT WAS GOOD BE4 STARTING HEARBEAT
In addition, always talking about node1, although the right permissions are set to /var/run/heartbeat/ccm/ccm and var/run/heartbeat/ccm/crm as 777 and hacluster:haclient, when i try to access /var/run/heartbeat/ccm/ccm i get a "permission denied" in "vi" on node2.(P.S: on node 1 that is starting normally i have acess to /var/run/heartbeat/ccm/ccm) i dunno if this has any relation to my problem but thought to mention it below u can find my ha.cf, and some logs from ha-log I hope you can help Thanks for your time Joe Abdo ha.cf ucast eth1 192.168.1.62 keepalive 1 deadtime 15 warntime 5 initdead 120 # depend on your hardware udpport 694 ping 192.168.1.4 auto_failback off node DATADOMAIN-BDC node DATADOMAIN-PDC use_logd yes compression bz2 compression_threshold 2 crm yes ha-log logd[24090]: 2007/10/26_11:10:27 info: logd started with /etc/logd.cf. logd[24094]: 2007/10/26_11:10:27 info: G_main_add_SignalHandler: Added signal handler for signal 15 logd[24090]: 2007/10/26_11:10:27 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[24111]: 2007/10/26_11:10:27 info: Enabling logging daemon heartbeat[24111]: 2007/10/26_11:10:27 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) heartbeat[24111]: 2007/10/26_11:10:27 WARN: File /etc/ha.d/haresources exists. heartbeat[24111]: 2007/10/26_11:10:27 WARN: This file is not used because crm is enabled heartbeat[24111]: 2007/10/26_11:10:27 WARN: logd is enabled but logfile/debugfile is still configured in ha.cf heartbeat[24111]: 2007/10/26_11:10:27 info: ************************** heartbeat[24111]: 2007/10/26_11:10:27 info: Configuration validated. Starting heartbeat 2.0.7 heartbeat[24112]: 2007/10/26_11:10:27 info: heartbeat: version 2.0.7 heartbeat[24112]: 2007/10/26_11:10:27 info: Heartbeat generation: 9 heartbeat[24112]: 2007/10/26_11:10:27 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[24112]: 2007/10/26_11:10:27 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[24112]: 2007/10/26_11:10:27 info: Removing /var/run/heartbeat/rsctmp failed, recreating. heartbeat[24112]: 2007/10/26_11:10:27 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth1 heartbeat[24112]: 2007/10/26_11:10:27 info: glib: ucast: bound send socket to device: eth1 heartbeat[24112]: 2007/10/26_11:10:27 info: glib: ucast: bound receive socket to device: eth1 heartbeat[24112]: 2007/10/26_11:10:27 info: glib: ucast: started on port 694 interface eth1 to 192.168.1.62 heartbeat[24112]: 2007/10/26_11:10:27 info: glib: ping heartbeat started. heartbeat[24112]: 2007/10/26_11:10:27 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[24112]: 2007/10/26_11:10:27 info: Local status now set to: 'up' heartbeat[24112]: 2007/10/26_11:10:28 info: Link 192.168.1.4:192.168.1.4 up. heartbeat[24112]: 2007/10/26_11:10:28 info: Status update for node 192.168.1.4: status ping heartbeat[24112]: 2007/10/26_11:10:39 info: Link datadomain-bdc:eth1 up. heartbeat[24112]: 2007/10/26_11:10:39 info: Status update for node datadomain-bdc: status up heartbeat[24112]: 2007/10/26_11:10:40 info: Comm_now_up(): updating status to active heartbeat[24112]: 2007/10/26_11:10:40 info: Local status now set to: 'active' heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/ccm" (106,110) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/cib" (106,110) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/lrmd" (0,0) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/stonithd" (0,0) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/attrd" (106,110) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/crmd" (106,110) heartbeat[24112]: 2007/10/26_11:10:40 info: Starting child client "/usr/lib/heartbeat/mgmtd -v" (0,0) heartbeat[24122]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/ccm" as uid 106 gid 110 (pid 24122) heartbeat[24123]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/cib" as uid 106 gid 110 (pid 24123) heartbeat[24124]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/lrmd" as uid 0 gid 0 (pid 24124) heartbeat[24125]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 24125) heartbeat[24126]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/attrd" as uid 106 gid 110 (pid 24126) heartbeat[24127]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/crmd" as uid 106 gid 110 (pid 24127) heartbeat[24128]: 2007/10/26_11:10:40 info: Starting "/usr/lib/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 24128) lrmd[24124]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 lrmd[24124]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[24124]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 10 lrmd[24124]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 12 lrmd[24124]: 2007/10/26_11:10:40 info: Started. mgmtd[24128]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[24128]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 10 mgmtd[24128]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 12 heartbeat[24112]: 2007/10/26_11:10:40 info: Status update for node datadomain-bdc: status active attrd[24126]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 ccm[24122]: 2007/10/26_11:10:40 info: Hostname: datadomain-pdc attrd[24126]: 2007/10/26_11:10:40 info: register_with_ha:attrd.c Hostname: datadomain-pdc cib[24123]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[24123]: 2007/10/26_11:10:40 info: G_main_add_TriggerHandler: Added signal manual handler cib[24123]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[24123]: 2007/10/26_11:10:40 info: main:main.c Retrieval of a per-action CIB: disabled cib[24123]: 2007/10/26_11:10:40 info: cib_register_ha:main.c Signing in with Heartbeat cib[24123]: 2007/10/26_11:10:40 info: cib_register_ha:main.c FSA Hostname: datadomain-pdc cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile:io.c Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml cib[24123]: 2007/10/26_11:10:40 WARN: validate_cib_digest:io.c No on-disk digest present cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: readCibXmlFile: [on-disk] cib[24123]: 2007/10/26_11:10:40 info: activateCibXml:io.c CIB size is 44912 bytes (was 0) cib[24123]: 2007/10/26_11:10:40 info: startCib:main.c CIB Initialization completed successfully cib[24123]: 2007/10/26_11:10:40 WARN: init_start:main.c CCM Activation failed cib[24123]: 2007/10/26_11:10:40 WARN: init_start:main.c CCM Connection failed 1 times (30 max) attrd[24126]: 2007/10/26_11:10:40 info: register_with_ha:attrd.c UUID: fe7e083d-b165-495e-bcd9-97f394f5bff2 stonithd[24125]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 10 stonithd[24125]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 12 stonithd[24125]: 2007/10/26_11:10:40 info: Signing in with heartbeat. crmd[24127]: 2007/10/26_11:10:40 info: init_start:main.c Starting crmd crmd[24127]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[24127]: 2007/10/26_11:10:40 info: G_main_add_TriggerHandler: Added signal manual handler crmd[24127]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 17 mgmtd[24128]: 2007/10/26_11:10:40 info: init_crm stonithd[24125]: 2007/10/26_11:10:40 notice: /usr/lib/heartbeat/stonithd start up successfully. stonithd[24125]: 2007/10/26_11:10:40 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[24123]: 2007/10/26_11:10:41 WARN: init_start:main.c CCM Activation failed cib[24123]: 2007/10/26_11:10:41 WARN: init_start:main.c CCM Connection failed 2 times (30 max) cib[24123]: 2007/10/26_11:10:42 WARN: init_start:main.c CCM Activation failed cib[24123]: 2007/10/26_11:10:42 WARN: init_start:main.c CCM Connection failed 3 times (30 max) ccm[24122]: 2007/10/26_11:10:43 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[24123]: 2007/10/26_11:10:43 info: init_start:main.c Starting cib mainloop cib[24129]: 2007/10/26_11:10:43 WARN: validate_cib_digest:io.c No on-disk digest present cib[24129]: 2007/10/26_11:10:43 info: write_cib_contents:io.c Wrote version 0.0.0 of the CIB to disk (digest: 41480864a95aa900ca2f7b570e67a99c) cib[24123]: 2007/10/26_11:10:43 info: cib_client_status_callback:callbacks.c Status update: Client datadomain-pdc/cib now has status [join] cib[24123]: 2007/10/26_11:10:43 info: cib_client_status_callback:callbacks.c Status update: Client datadomain-pdc/cib now has status [online] crmd[24127]: 2007/10/26_11:10:43 info: do_cib_control:cib.c CIB connection established crmd[24127]: 2007/10/26_11:10:43 info: register_with_ha:control.c Hostname: datadomain-pdc cib[24123]: 2007/10/26_11:10:43 info: cib_null_callback:callbacks.c Setting cib_refresh_notify callbacks for crmd: on cib[24123]: 2007/10/26_11:10:43 info: cib_null_callback:callbacks.c Setting cib_diff_notify callbacks for mgmtd: on cib[24123]: 2007/10/26_11:10:44 info: cib_client_status_callback:callbacks.c Status update: Client datadomain-bdc/cib now has status [online] crmd[24127]: 2007/10/26_11:10:44 info: register_with_ha:control.c UUID: fe7e083d-b165-495e-bcd9-97f394f5bff2 crmd[24127]: 2007/10/26_11:10:45 info: populate_cib_nodes:control.c Requesting the list of configured nodes heartbeat[24112]: 2007/10/26_11:10:45 WARN: 1 lost packet(s) for [datadomain-bdc] [19:21] heartbeat[24112]: 2007/10/26_11:10:45 info: No pkts missing from datadomain-bdc! mgmtd[24128]: 2007/10/26_11:10:45 info: Started. crmd[24127]: 2007/10/26_11:10:45 notice: populate_cib_nodes:control.c Node: datadomain-pdc (uuid: fe7e083d-b165-495e-bcd9-97f394f5bff2) ccm[24122]: 2007/10/26_11:10:45 info: Break tie for 2 nodes cluster cib[24123]: 2007/10/26_11:10:45 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm cib[24123]: 2007/10/26_11:10:45 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 cib[24123]: 2007/10/26_11:10:45 info: cib_ccm_msg_callback:callbacks.c PEER: datadomain-pdc heartbeat[24112]: 2007/10/26_11:10:46 WARN: 1 lost packet(s) for [datadomain-bdc] [25:27] heartbeat[24112]: 2007/10/26_11:10:46 info: No pkts missing from datadomain-bdc! crmd[24127]: 2007/10/26_11:10:46 notice: populate_cib_nodes:control.c Node: datadomain-bdc (uuid: ede73bcf-bf29-4a84-a560-554211996f59) crmd[24127]: 2007/10/26_11:10:46 info: do_ha_control:control.c Connected to Heartbeat cib[24123]: 2007/10/26_11:10:46 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm cib[24123]: 2007/10/26_11:10:46 info: mem_handle_event: no mbr_track info cib[24123]: 2007/10/26_11:10:46 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm cib[24123]: 2007/10/26_11:10:46 info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4 cib[24123]: 2007/10/26_11:10:46 info: cib_ccm_msg_callback:callbacks.c PEER: datadomain-pdc cib[24123]: 2007/10/26_11:10:46 info: cib_ccm_msg_callback:callbacks.c PEER: datadomain-bdc cib[24123]: 2007/10/26_11:10:46 info: activateCibXml:io.c CIB size is 47912 bytes (was 45120) cib[24123]: 2007/10/26_11:10:46 info: cib_diff_notify:notify.c Local-only Change (client:24127, call: 3): 0.0.0 (ok) cib[24130]: 2007/10/26_11:10:46 WARN: file2xml:xml.c File contained no XML cib[24130]: 2007/10/26_11:10:46 ERROR: validate_cib_digest:io.c Digest comparision failed: vs. (null) cib[24130]: 2007/10/26_11:10:46 ERROR: write_cib_contents:io.c /var/lib/heartbeat/crm/cib.xml was manually modified while Heartbeat was active! cib[24123]: 2007/10/26_11:10:46 ERROR: cib_diskwrite_complete:main.c Disk write failed: status=256, signo=0, exitcode=1 cib[24123]: 2007/10/26_11:10:46 ERROR: cib_diskwrite_complete:main.c Disabling disk writes after write failure crmd[24127]: 2007/10/26_11:10:46 info: do_ccm_control:ccm.c CCM connection established... waiting for first callback crmd[24127]: 2007/10/26_11:10:46 info: do_started:control.c Delaying start, CCM (0000000000100000) not connected crmd[24127]: 2007/10/26_11:10:46 info: init_start:main.c Starting crmd's mainloop crmd[24127]: 2007/10/26_11:10:46 notice: crmd_client_status_callback:callbacks.c Status update: Client datadomain-pdc/crmd now has status [online] crmd[24127]: 2007/10/26_11:10:46 info: crmd_client_status_callback:callbacks.c Uncaching UUID for datadomain-pdc crmd[24127]: 2007/10/26_11:10:47 notice: crmd_client_status_callback:callbacks.c Status update: Client datadomain-bdc/crmd now has status [online] crmd[24127]: 2007/10/26_11:10:47 info: crmd_client_status_callback:callbacks.c Uncaching UUID for datadomain-bdc cib[24123]: 2007/10/26_11:10:47 info: cib_diff_notify:notify.c Local-only Change (client:24127, call: 5): 0.0.0 (ok) crmd[24127]: 2007/10/26_11:10:47 notice: crmd_client_status_callback:callbacks.c Status update: Client datadomain-pdc/crmd now has status [online] crmd[24127]: 2007/10/26_11:10:47 info: crmd_client_status_callback:callbacks.c Uncaching UUID for datadomain-pdc cib[24123]: 2007/10/26_11:10:47 info: cib_diff_notify:notify.c Local-only Change (client:24127, call: 6): 0.0.0 (ok) crmd[24127]: 2007/10/26_11:10:47 notice: crmd_client_status_callback:callbacks.c Status update: Client datadomain-bdc/crmd now has status [online] crmd[24127]: 2007/10/26_11:10:47 info: crmd_client_status_callback:callbacks.c Uncaching UUID for datadomain-bdc crmd[24127]: 2007/10/26_11:10:48 info: do_started:control.c Delaying start, CCM (0000000000100000) not connected crmd[24127]: 2007/10/26_11:10:48 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm crmd[24127]: 2007/10/26_11:10:48 info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4 crmd[24127]: 2007/10/26_11:10:48 info: crmd_ccm_msg_callback:callbacks.c Quorum (re)attained after event=NEW MEMBERSHIP (id=2) crmd[24127]: 2007/10/26_11:10:48 info: ccm_event_detail:ccm.c NEW MEMBERSHIP: trans=2, nodes=2, new=2, lost=0 n_idx=0, new_idx=0, old_idx=4 crmd[24127]: 2007/10/26_11:10:48 info: ccm_event_detail:ccm.c CURRENT: datadomain-pdc [nodeid=1, born=1] crmd[24127]: 2007/10/26_11:10:48 info: ccm_event_detail:ccm.c CURRENT: datadomain-bdc [nodeid=0, born=2] crmd[24127]: 2007/10/26_11:10:48 info: ccm_event_detail:ccm.c NEW: datadomain-pdc [nodeid=1, born=1] crmd[24127]: 2007/10/26_11:10:48 info: ccm_event_detail:ccm.c NEW: datadomain-bdc [nodeid=0, born=2] crmd[24127]: 2007/10/26_11:10:48 info: do_started:control.c The local CRM is operational crmd[24127]: 2007/10/26_11:10:48 info: do_state_transition:fsa.c datadomain-pdc: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_CCM_CALLBACK origin=do_started ] crmd[24127]: 2007/10/26_11:10:48 info: update_dc:utils.c Set DC to () cib[24123]: 2007/10/26_11:10:48 info: cib_diff_notify:notify.c Local-only Change (client:24127, call: 9): 0.0.0 (ok) attrd[24126]: 2007/10/26_11:10:48 info: main:attrd.c Starting mainloop... crmd[24127]: 2007/10/26_11:11:49 info: crm_timer_popped:utils.c Election Trigger (I_DC_TIMEOUT) just popped! crmd[24127]: 2007/10/26_11:11:49 WARN: do_log:misc.c [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING) crmd[24127]: 2007/10/26_11:11:49 info: do_state_transition:fsa.c datadomain-pdc: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] crmd[24127]: 2007/10/26_11:11:49 info: update_dc:utils.c Set DC to () crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Election check: vote from datadomain-bdc crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Election won over datadomain-bdc crmd[24127]: 2007/10/26_11:11:49 info: do_election_check:election.c Still waiting on 2 non-votes (2 total) crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Updated voted hash for datadomain-pdc to vote crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Election ignore: our vote (datadomain-pdc) crmd[24127]: 2007/10/26_11:11:49 info: do_election_check:election.c Still waiting on 1 non-votes (2 total) crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Updated voted hash for datadomain-pdc to vote crmd[24127]: 2007/10/26_11:11:49 info: do_election_count_vote:election.c Election ignore: our vote (datadomain-pdc) crmd[24127]: 2007/10/26_11:11:49 info: do_election_check:election.c Still waiting on 1 non-votes (2 total) crmd[24127]: 2007/10/26_11:11:50 info: do_election_count_vote:election.c Updated voted hash for datadomain-bdc to no-vote crmd[24127]: 2007/10/26_11:11:50 info: do_election_count_vote:election.c Election ignore: no-vote from datadomain-bdc crmd[24127]: 2007/10/26_11:11:50 info: do_election_check:election.c Still waiting on 1 non-votes (2 total) crmd[24127]: 2007/10/26_11:11:50 info: do_state_transition:fsa.c datadomain-pdc: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] crmd[24127]: 2007/10/26_11:11:50 info: start_subsystem:subsystems.c Starting sub-system "tengine" crmd[24127]: 2007/10/26_11:11:50 info: start_subsystem:subsystems.c Starting sub-system "pengine" tengine[24151]: 2007/10/26_11:11:50 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[24127]: 2007/10/26_11:11:50 info: do_dc_takeover:election.c Taking over DC status for this partition tengine[24151]: 2007/10/26_11:11:50 info: G_main_add_TriggerHandler: Added signal manual handler crmd[24127]: 2007/10/26_11:11:50 info: update_dc:utils.c Set DC to () crmd[24127]: 2007/10/26_11:11:50 info: do_dc_join_offer_all:join_dc.c join-1: Waiting on 2 outstanding join acks cib[24123]: 2007/10/26_11:11:50 info: cib_process_readwrite:messages.c We are now in R/W mode cib[24123]: 2007/10/26_11:11:50 info: revision_check:messages.c Updating CIB revision to 1.3 cib[24123]: 2007/10/26_11:11:50 info: cib_diff_notify:notify.c Update (client: 24127, call:13): 0.0.0 -> 0.0.1 (ok) crmd[24127]: 2007/10/26_11:11:50 info: update_dc:utils.c Set DC to datadomain-pdc (1.0.6) cib[24123]: 2007/10/26_11:11:50 info: cib_null_callback:callbacks.c Setting cib_diff_notify callbacks for tengine: on tengine[24151]: 2007/10/26_11:11:50 info: init_start:main.c Registering TE UUID: df9db025-8ad3-4239-8417-ea52c343fbeb tengine[24151]: 2007/10/26_11:11:50 info: set_graph_functions:utils.c Setting custom graph functions tengine[24151]: 2007/10/26_11:11:50 info: unpack_graph:unpack.c Unpacked transition -1: 0 actions in 0 synapses tengine[24151]: 2007/10/26_11:11:50 info: init_start:main.c Starting tengine pengine[24152]: 2007/10/26_11:11:50 info: G_main_add_SignalHandler: Added signal handler for signal 15 pengine[24152]: 2007/10/26_11:11:50 info: init_start:main.c Starting pengine crmd[24127]: 2007/10/26_11:11:51 info: do_state_transition:fsa.c datadomain-pdc: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[24127]: 2007/10/26_11:11:51 info: do_state_transition:fsa.c All 2 cluster nodes responded to the join offer. crmd[24127]: 2007/10/26_11:11:51 info: update_attrd:join_dc.c Connecting to attrd... attrd[24126]: 2007/10/26_11:11:51 info: attrd_local_callback:attrd.c Sending full refresh cib[24123]: 2007/10/26_11:11:51 info: sync_our_cib:messages.c Syncing CIB to all peers cib[24123]: 2007/10/26_11:11:51 info: cib_diff_notify:notify.c Update (client: 24127, call:16): 0.0.1 -> 0.0.2 (ok) tengine[24151]: 2007/10/26_11:11:51 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.0.1 -> 0.0.2 cib[24123]: 2007/10/26_11:11:51 info: cib_diff_notify:notify.c Update (client: 24127, call:17): 0.0.2 -> 0.1.3 (ok) tengine[24151]: 2007/10/26_11:11:51 info: te_update_diff:callbacks.c Processing diff (cib_bump): 0.0.2 -> 0.1.3 cib[24123]: 2007/10/26_11:11:52 info: cib_diff_notify:notify.c Update (client: 24127, call:18): 0.1.3 -> 0.1.4 (ok) tengine[24151]: 2007/10/26_11:11:52 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.3 -> 0.1.4 cib[24123]: 2007/10/26_11:11:52 info: cib_diff_notify:notify.c Update (client: 24127, call:19): 0.1.4 -> 0.1.5 (ok) tengine[24151]: 2007/10/26_11:11:52 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.4 -> 0.1.5 crmd[24127]: 2007/10/26_11:11:52 info: update_dc:utils.c Set DC to datadomain-pdc (1.0.6) crmd[24127]: 2007/10/26_11:11:52 info: do_dc_join_ack:join_dc.c join-1: Updating node state to member for datadomain-pdc) cib[24123]: 2007/10/26_11:11:52 info: cib_diff_notify:notify.c Update (client: 24127, call:20): 0.1.5 -> 0.1.6 (ok) tengine[24151]: 2007/10/26_11:11:52 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.5 -> 0.1.6 crmd[24127]: 2007/10/26_11:11:53 info: do_dc_join_ack:join_dc.c join-1: Updating node state to member for datadomain-bdc) cib[24123]: 2007/10/26_11:11:54 info: cib_diff_notify:notify.c Update (client: 24127, call:21): 0.1.6 -> 0.1.7 (ok) crmd[24127]: 2007/10/26_11:11:54 info: do_state_transition:fsa.c datadomain-pdc: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[24127]: 2007/10/26_11:11:54 info: do_state_transition:fsa.c All 2 cluster nodes are eligable to run resources. tengine[24151]: 2007/10/26_11:11:54 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.6 -> 0.1.7 tengine[24151]: 2007/10/26_11:11:54 WARN: process_graph_event:events.c Event not found. tengine[24151]: 2007/10/26_11:11:54 info: process_graph_event: match:not-found tengine[24151]: 2007/10/26_11:11:54 info: update_abort_priority:utils.c Abort priority upgraded to 1000000 tengine[24151]: 2007/10/26_11:11:54 WARN: process_graph_event:events.c Event not found. tengine[24151]: 2007/10/26_11:11:54 info: process_graph_event: match:not-found pengine[24152]: 2007/10/26_11:11:56 info: get_last_sequence:utils.c /var/lib/heartbeat/pengine/pe-input.last was not valid pengine[24152]: 2007/10/26_11:11:56 ERROR: write_xml_file:xml.c bzWriteClose() failed: -6 pengine[24152]: 2007/10/26_11:11:56 ERROR: Cannot write output to /var/lib/heartbeat/pengine/pe-input-0.bz2: No space left on device pengine[24152]: 2007/10/26_11:11:56 info: process_pe_message:pengine.c Transition 1: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-0.bz2 crmd[24127]: 2007/10/26_11:11:56 info: process_lrm_event:lrm.c LRM operation (3) monitor_0 on exim4_2 Error: (7) not running cib[24123]: 2007/10/26_11:11:56 info: cib_diff_notify:notify.c Update (client: 24127, call:53): 0.1.7 -> 0.1.8 (ok) tengine[24151]: 2007/10/26_11:11:56 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.7 -> 0.1.8 tengine[24151]: 2007/10/26_11:11:56 info: match_graph_event:events.c Action exim4_2_monitor_0 (4) confirmed crmd[24127]: 2007/10/26_11:11:56 info: process_lrm_event:lrm.c LRM operation (2) monitor_0 on IPaddr_192_168_1_11 Error: (7) not running cib[24123]: 2007/10/26_11:11:56 info: cib_diff_notify:notify.c Update (client: 24127, call:54): 0.1.8 -> 0.1.9 (ok) tengine[24151]: 2007/10/26_11:11:56 info: te_update_diff:callbacks.c Processing diff (cib_update): 0.1.8 -> 0.1.9 tengine[24151]: 2007/10/26_11:11:56 info: match_graph_event:events.c Action IPaddr_192_168_1_11_monitor_0 (3) confirmed tengine[24151]: 2007/10/26_11:11:56 info: send_rsc_command:actions.c Initiating action 2: probe_complete on datadomain-pdc tengine[24151]: 2007/10/26_11:11:56 info: te_pseudo_action:actions.c Pseudo action 1 confirmed tengine[24151]: 2007/10/26_11:11:56 info: te_pseudo_action:actions.c Pseudo action 10 confirmed _________________________________________________________________ Discover the new Windows Vista http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
