Hi Linux-HA list :) I'm running CentOS 5 (Linux 2.6.18-53.1.6.el5 #1 SMP Wed Jan 23 11:28:47 EST 2008 x86_64 GNU/Linux), and Heartbeat from the packages:
heartbeat.x86_64 2.1.3-3.el5.centos heartbeat-pils.x86_64 2.1.3-3.el5.centos heartbeat-stonith.x86_64 2.1.3-3.el5.centos I'm trying to set up my second cluster, and it's giving me some headaches already. It should have two machines, and will support a DRBD filesystem and an IP address as resources. No resources are configured at the moment, I just got the two CentOS machines and I'm trying to start heartbeat. When I issue "service heartbeat start", just after successful packages installation, I got those error messages multiple times, and then heartbeat (or some of it's components) just decide that rebooting the machine is the right thing to do. :( (Full logfiles for a single session, from startup to the first error messages is attached) /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed /var/log/ha-log:ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed /var/log/ha-debug:ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed I just ran out of alternatives, and the error messages aren't descriptive enough to point me to the next clue. Does someone have suggestions? Thank you all in advance for your help. Kindest regards. -- Luis Motta Campos (a.k.a. Monsieur Champs) is a software engineer, Perl fanatic evangelist, and amateur {cook, photographer}
logd[3763]: 2008/03/03_13:00:07 info: logd started with /etc/logd.cf. logd[3771]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15 logd[3763]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[3784]: 2008/03/03_13:00:07 info: Enabling logging daemon heartbeat[3784]: 2008/03/03_13:00:07 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) heartbeat[3784]: 2008/03/03_13:00:07 info: Version 2 support: on heartbeat[3784]: 2008/03/03_13:00:07 info: ************************** heartbeat[3784]: 2008/03/03_13:00:07 info: Configuration validated. Starting heartbeat 2.1.3 heartbeat[3785]: 2008/03/03_13:00:07 info: heartbeat: version 2.1.3 heartbeat[3785]: 2008/03/03_13:00:07 info: Heartbeat generation: 1203522985 heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2 heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1 heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[3785]: 2008/03/03_13:00:07 info: Local status now set to: 'up' heartbeat[3785]: 2008/03/03_13:00:38 WARN: node db-mysql-test4-ha.ripe.net: is dead heartbeat[3785]: 2008/03/03_13:00:38 info: Comm_now_up(): updating status to active heartbeat[3785]: 2008/03/03_13:00:38 info: Local status now set to: 'active' heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/ccm" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/cib" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/lrmd -r" (0,0) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/stonithd" (0,0) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/attrd" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/crmd" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/mgmtd -v" (0,0) heartbeat[3925]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/ccm" as uid 498 gid 496 (pid 3925) heartbeat[3926]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/cib" as uid 498 gid 496 (pid 3926) heartbeat[3927]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3927) heartbeat[3928]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/stonithd" as uid 0 gid 0 (pid 3928) stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 heartbeat[3929]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/attrd" as uid 498 gid 496 (pid 3929) heartbeat[3930]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/crmd" as uid 498 gid 496 (pid 3930) heartbeat[3931]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3931) mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[3931]: 2008/03/03_13:00:38 debug: Enabling coredumps mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 ccm[3925]: 2008/03/03_13:00:38 info: Hostname: db-mysql-test3-ha.ripe.net mgmtd[3931]: 2008/03/03_13:00:38 WARN: lrm_signon: can not initiate connection mgmtd[3931]: 2008/03/03_13:00:38 info: login to lrm: 0, ret:0 stonithd[3928]: 2008/03/03_13:00:38 info: Signing in with heartbeat. cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3926]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[3926]: 2008/03/03_13:00:38 info: main: Retrieval of a per-action CIB: disabled cib[3926]: 2008/03/03_13:00:38 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="false" admin_epoch="0" epoch="1" num_updates="1" have_quorum="false" ignore_dtd="false" num_peers="0" cib_feature_revision="2.0" cib-last-written="Mon Mar 3 12:30:41 2008"> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <configuration> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <crm_config> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cluster_property_set id="cib-bootstrap-options"> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <attributes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </attributes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cluster_property_set> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </crm_config> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nodes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <node id="0f24958a-37b1-495a-84fc-911fe61720de" uname="db-mysql-test3-ha.ripe.net" type="normal"/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </nodes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <resources/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <constraints/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </configuration> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <status/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cib> cib[3926]: 2008/03/03_13:00:38 info: startCib: CIB Initialization completed successfully cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: Signing in with Heartbeat cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: FSA Hostname: db-mysql-test3-ha.ripe.net cib[3926]: 2008/03/03_13:00:38 info: ccm_connect: Registering with CCM... cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Activation failed cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Connection failed 1 times (30 max) lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 lrmd[3927]: 2008/03/03_13:00:38 info: Started. attrd[3929]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net crmd[3930]: 2008/03/03_13:00:38 info: main: CRM Hg Version: node: 552305612591183b1628baa5bc6e903e0f1e26a3 crmd[3930]: 2008/03/03_13:00:38 info: crmd_init: Starting crmd crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 stonithd[3928]: 2008/03/03_13:00:38 notice: /usr/lib64/heartbeat/stonithd start up successfully. stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de mgmtd[3931]: 2008/03/03_13:00:39 info: init_crm ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed ccm[3925]: 2008/03/03_13:00:39 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[3785]: 2008/03/03_13:00:40 info: killing /usr/lib64/heartbeat/mgmtd -v process group 3931 with signal 15 cib[3926]: 2008/03/03_13:00:41 info: ccm_connect: Registering with CCM... cib[3926]: 2008/03/03_13:00:41 info: cib_init: Starting cib mainloop cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 cib[3926]: 2008/03/03_13:00:41 info: cib_ccm_msg_callback: PEER: db-mysql-test3-ha.ripe.net cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3926]: 2008/03/03_13:00:42 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [join] cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [online] mgmtd[3931]: 2008/03/03_13:00:43 debug: main: run the loop... mgmtd[3931]: 2008/03/03_13:00:43 info: Started. mgmtd[3931]: 2008/03/03_13:00:43 info: mgmtd is shutting down cib[3926]: 2008/03/03_13:00:43 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on crmd[3930]: 2008/03/03_13:00:43 info: do_cib_control: CIB connection established cib[3926]: 2008/03/03_13:00:43 ERROR: generic plugin load failed cib[3926]: 2008/03/03_13:00:43 ERROR: cl_compress_field: loading compression module failed cib[3926]: 2008/03/03_13:00:43 ERROR: uncompress2compress: compressing 6th field failed mgmtd[3931]: 2008/03/03_13:00:43 debug: [mgmtd] stopped heartbeat[3785]: 2008/03/03_13:00:43 info: killing /usr/lib64/heartbeat/crmd process group 3930 with signal 15 crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net cib[3952]: 2008/03/03_13:00:43 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 330f8d72be5851c99d7ce2e9a8b976a4) cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de crmd[3930]: 2008/03/03_13:00:44 info: populate_cib_nodes: Requesting the list of configured nodes crmd[3930]: 2008/03/03_13:00:44 WARN: get_uuid: Could not calculate UUID for db-mysql-test4-ha.ripe.net crmd[3930]: 2008/03/03_13:00:44 WARN: populate_cib_nodes: Node db-mysql-test4-ha.ripe.net: no uuid found crmd[3930]: 2008/03/03_13:00:45 notice: populate_cib_nodes: Node: db-mysql-test3-ha.ripe.net (uuid: 0f24958a-37b1-495a-84fc-911fe61720de) crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Connected to Heartbeat cib[3926]: 2008/03/03_13:00:45 ERROR: generic plugin load failed cib[3926]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed cib[3926]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 info: do_ccm_control: CCM connection established... waiting for first callback crmd[3930]: 2008/03/03_13:00:45 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3930]: 2008/03/03_13:00:45 info: crmd_init: Starting crmd's mainloop crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Requesting shutdown crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Using default shutdown escalation: 1200000ms crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online] crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online] crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 WARN: do_log: [[FSA]] Input I_SHUTDOWN from crm_shutdown() received in state (S_STARTING) crmd[3930]: 2008/03/03_13:00:45 info: do_state_transition: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] crmd[3930]: 2008/03/03_13:00:45 info: do_shutdown: All subsystems stopped, continuing crmd[3930]: 2008/03/03_13:00:45 info: do_lrm_control: Disconnected from the LRM crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Disconnected from Heartbeat crmd[3930]: 2008/03/03_13:00:45 info: do_cib_control: Disconnecting CIB crmd[3930]: 2008/03/03_13:00:45 info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd crmd[3930]: 2008/03/03_13:00:45 info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ] crmd[3930]: 2008/03/03_13:00:45 info: do_exit: [crmd] stopped (0) ccm[3925]: 2008/03/03_13:00:45 info: client (pid=3930) removed from ccm heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/attrd process group 3929 with signal 15 cib[3926]: 2008/03/03_13:00:45 WARN: send_via_callback_channel: Cannot find client for token a2d999f1-a039-4587-93db-746028050fec cib[3926]: 2008/03/03_13:00:45 WARN: do_local_notify: A-Sync reply to 3930 failed: client left before we could send reply attrd[3929]: 2008/03/03_13:00:45 info: main: Starting mainloop... attrd[3929]: 2008/03/03_13:00:45 info: attrd_shutdown: Exiting attrd[3929]: 2008/03/03_13:00:45 info: main: Exiting... attrd[3929]: 2008/03/03_13:00:45 info: attrd_cib_connection_destroy: Connection to the CIB terminated... heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/stonithd process group 3928 with signal 15 stonithd[3928]: 2008/03/03_13:00:45 notice: /usr/lib64/heartbeat/stonithd normally quit. heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/lrmd -r process group 3927 with signal 15 lrmd[3927]: 2008/03/03_13:00:45 info: lrmd is shutting down heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/cib process group 3926 with signal 15 cib[3926]: 2008/03/03_13:00:45 info: cib_shutdown: Disconnected 0 clients cib[3926]: 2008/03/03_13:00:45 info: cib_process_disconnect: All clients disconnected... cib[3926]: 2008/03/03_13:00:45 info: terminate_ha_connection: initiate_exit: Disconnecting heartbeat cib[3926]: 2008/03/03_13:00:45 info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting cib[3926]: 2008/03/03_13:00:45 info: main: Done ccm[3925]: 2008/03/03_13:00:46 info: client (pid=3926) removed from ccm heartbeat[3785]: 2008/03/03_13:00:46 info: killing /usr/lib64/heartbeat/ccm process group 3925 with signal 15 ccm[3925]: 2008/03/03_13:00:46 info: received SIGTERM, going to shut down heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBFIFO process 3790 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBWRITE process 3791 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBREAD process 3792 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3792 exited. 3 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3790 exited. 2 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3791 exited. 1 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: db-mysql-test3-ha.ripe.net Heartbeat shutdown complete. logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: received SIGTERM logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: waiting for 0 messages to be read by write process logd[3763]: 2008/03/03_13:00:47 debug: logd_term_action: sending SIGTERM to write process logd[3771]: 2008/03/03_13:00:47 info: logd_term_write_action: received SIGTERM logd[3771]: 2008/03/03_13:00:47 debug: Writing out 0 messages then quitting logd[3771]: 2008/03/03_13:00:47 info: Exiting write process
logd[3763]: 2008/03/03_13:00:07 info: logd started with /etc/logd.cf. logd[3771]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15 logd[3763]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[3784]: 2008/03/03_13:00:07 info: Enabling logging daemon heartbeat[3784]: 2008/03/03_13:00:07 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) heartbeat[3784]: 2008/03/03_13:00:07 info: Version 2 support: on heartbeat[3784]: 2008/03/03_13:00:07 info: ************************** heartbeat[3784]: 2008/03/03_13:00:07 info: Configuration validated. Starting heartbeat 2.1.3 heartbeat[3785]: 2008/03/03_13:00:07 info: heartbeat: version 2.1.3 heartbeat[3785]: 2008/03/03_13:00:07 info: Heartbeat generation: 1203522985 heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2 heartbeat[3785]: 2008/03/03_13:00:07 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1 heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3785]: 2008/03/03_13:00:07 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[3785]: 2008/03/03_13:00:07 info: Local status now set to: 'up' heartbeat[3785]: 2008/03/03_13:00:38 WARN: node db-mysql-test4-ha.ripe.net: is dead heartbeat[3785]: 2008/03/03_13:00:38 info: Comm_now_up(): updating status to active heartbeat[3785]: 2008/03/03_13:00:38 info: Local status now set to: 'active' heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/ccm" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/cib" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/lrmd -r" (0,0) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/stonithd" (0,0) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/attrd" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/crmd" (498,496) heartbeat[3785]: 2008/03/03_13:00:38 info: Starting child client "/usr/lib64/heartbeat/mgmtd -v" (0,0) heartbeat[3925]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/ccm" as uid 498 gid 496 (pid 3925) heartbeat[3926]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/cib" as uid 498 gid 496 (pid 3926) heartbeat[3927]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3927) heartbeat[3928]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/stonithd" as uid 0 gid 0 (pid 3928) stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 heartbeat[3929]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/attrd" as uid 498 gid 496 (pid 3929) heartbeat[3930]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/crmd" as uid 498 gid 496 (pid 3930) heartbeat[3931]: 2008/03/03_13:00:38 info: Starting "/usr/lib64/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3931) mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 mgmtd[3931]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 ccm[3925]: 2008/03/03_13:00:38 info: Hostname: db-mysql-test3-ha.ripe.net mgmtd[3931]: 2008/03/03_13:00:38 WARN: lrm_signon: can not initiate connection mgmtd[3931]: 2008/03/03_13:00:38 info: login to lrm: 0, ret:0 stonithd[3928]: 2008/03/03_13:00:38 info: Signing in with heartbeat. cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3926]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler cib[3926]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[3926]: 2008/03/03_13:00:38 info: main: Retrieval of a per-action CIB: disabled cib[3926]: 2008/03/03_13:00:38 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="false" admin_epoch="0" epoch="1" num_updates="1" have_quorum="false" ignore_dtd="false" num_peers="0" cib_feature_revision="2.0" cib-last-written="Mon Mar 3 12:30:41 2008"> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <configuration> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <crm_config> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <cluster_property_set id="cib-bootstrap-options"> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <attributes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </attributes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cluster_property_set> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </crm_config> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <nodes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <node id="0f24958a-37b1-495a-84fc-911fe61720de" uname="db-mysql-test3-ha.ripe.net" type="normal"/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </nodes> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <resources/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <constraints/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </configuration> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] <status/> cib[3926]: 2008/03/03_13:00:38 info: log_data_element: readCibXmlFile: [on-disk] </cib> cib[3926]: 2008/03/03_13:00:38 info: startCib: CIB Initialization completed successfully cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: Signing in with Heartbeat cib[3926]: 2008/03/03_13:00:38 info: cib_register_ha: FSA Hostname: db-mysql-test3-ha.ripe.net cib[3926]: 2008/03/03_13:00:38 info: ccm_connect: Registering with CCM... cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Activation failed cib[3926]: 2008/03/03_13:00:38 WARN: ccm_connect: CCM Connection failed 1 times (30 max) lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 lrmd[3927]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 lrmd[3927]: 2008/03/03_13:00:38 info: Started. attrd[3929]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net crmd[3930]: 2008/03/03_13:00:38 info: main: CRM Hg Version: node: 552305612591183b1628baa5bc6e903e0f1e26a3 crmd[3930]: 2008/03/03_13:00:38 info: crmd_init: Starting crmd crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_TriggerHandler: Added signal manual handler crmd[3930]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 stonithd[3928]: 2008/03/03_13:00:38 notice: /usr/lib64/heartbeat/stonithd start up successfully. stonithd[3928]: 2008/03/03_13:00:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 attrd[3929]: 2008/03/03_13:00:38 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de mgmtd[3931]: 2008/03/03_13:00:39 info: init_crm ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin majority failed ccm[3925]: 2008/03/03_13:00:39 ERROR: generic plugin load failed ccm[3925]: 2008/03/03_13:00:39 ERROR: cl_load_plugin: init plugin universe failed ccm[3925]: 2008/03/03_13:00:39 ERROR: ccm_calculate_quorum: loading plugin twonodes failed ccm[3925]: 2008/03/03_13:00:39 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[3785]: 2008/03/03_13:00:40 info: killing /usr/lib64/heartbeat/mgmtd -v process group 3931 with signal 15 cib[3926]: 2008/03/03_13:00:41 info: ccm_connect: Registering with CCM... cib[3926]: 2008/03/03_13:00:41 info: cib_init: Starting cib mainloop cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm cib[3926]: 2008/03/03_13:00:41 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 cib[3926]: 2008/03/03_13:00:41 info: cib_ccm_msg_callback: PEER: db-mysql-test3-ha.ripe.net cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:41 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3926]: 2008/03/03_13:00:42 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [join] cib[3926]: 2008/03/03_13:00:42 info: cib_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/cib now has status [online] mgmtd[3931]: 2008/03/03_13:00:43 info: Started. mgmtd[3931]: 2008/03/03_13:00:43 info: mgmtd is shutting down cib[3926]: 2008/03/03_13:00:43 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on crmd[3930]: 2008/03/03_13:00:43 info: do_cib_control: CIB connection established cib[3926]: 2008/03/03_13:00:43 ERROR: generic plugin load failed cib[3926]: 2008/03/03_13:00:43 ERROR: cl_compress_field: loading compression module failed cib[3926]: 2008/03/03_13:00:43 ERROR: uncompress2compress: compressing 6th field failed heartbeat[3785]: 2008/03/03_13:00:43 info: killing /usr/lib64/heartbeat/crmd process group 3930 with signal 15 crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: Hostname: db-mysql-test3-ha.ripe.net cib[3952]: 2008/03/03_13:00:43 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 330f8d72be5851c99d7ce2e9a8b976a4) cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3952]: 2008/03/03_13:00:43 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3930]: 2008/03/03_13:00:43 info: register_with_ha: UUID: 0f24958a-37b1-495a-84fc-911fe61720de crmd[3930]: 2008/03/03_13:00:44 info: populate_cib_nodes: Requesting the list of configured nodes crmd[3930]: 2008/03/03_13:00:44 WARN: get_uuid: Could not calculate UUID for db-mysql-test4-ha.ripe.net crmd[3930]: 2008/03/03_13:00:44 WARN: populate_cib_nodes: Node db-mysql-test4-ha.ripe.net: no uuid found crmd[3930]: 2008/03/03_13:00:45 notice: populate_cib_nodes: Node: db-mysql-test3-ha.ripe.net (uuid: 0f24958a-37b1-495a-84fc-911fe61720de) crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Connected to Heartbeat cib[3926]: 2008/03/03_13:00:45 ERROR: generic plugin load failed cib[3926]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed cib[3926]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 info: do_ccm_control: CCM connection established... waiting for first callback crmd[3930]: 2008/03/03_13:00:45 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3930]: 2008/03/03_13:00:45 info: crmd_init: Starting crmd's mainloop crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Requesting shutdown crmd[3930]: 2008/03/03_13:00:45 info: crm_shutdown: Using default shutdown escalation: 1200000ms crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online] crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 notice: crmd_client_status_callback: Status update: Client db-mysql-test3-ha.ripe.net/crmd now has status [online] crmd[3930]: 2008/03/03_13:00:45 ERROR: generic plugin load failed crmd[3930]: 2008/03/03_13:00:45 ERROR: cl_compress_field: loading compression module failed crmd[3930]: 2008/03/03_13:00:45 ERROR: uncompress2compress: compressing 6th field failed crmd[3930]: 2008/03/03_13:00:45 WARN: do_log: [[FSA]] Input I_SHUTDOWN from crm_shutdown() received in state (S_STARTING) crmd[3930]: 2008/03/03_13:00:45 info: do_state_transition: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] crmd[3930]: 2008/03/03_13:00:45 info: do_shutdown: All subsystems stopped, continuing crmd[3930]: 2008/03/03_13:00:45 info: do_lrm_control: Disconnected from the LRM crmd[3930]: 2008/03/03_13:00:45 info: do_ha_control: Disconnected from Heartbeat crmd[3930]: 2008/03/03_13:00:45 info: do_cib_control: Disconnecting CIB crmd[3930]: 2008/03/03_13:00:45 info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd crmd[3930]: 2008/03/03_13:00:45 info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ] crmd[3930]: 2008/03/03_13:00:45 info: do_exit: [crmd] stopped (0) ccm[3925]: 2008/03/03_13:00:45 info: client (pid=3930) removed from ccm heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/attrd process group 3929 with signal 15 cib[3926]: 2008/03/03_13:00:45 WARN: send_via_callback_channel: Cannot find client for token a2d999f1-a039-4587-93db-746028050fec cib[3926]: 2008/03/03_13:00:45 WARN: do_local_notify: A-Sync reply to 3930 failed: client left before we could send reply attrd[3929]: 2008/03/03_13:00:45 info: main: Starting mainloop... attrd[3929]: 2008/03/03_13:00:45 info: attrd_shutdown: Exiting attrd[3929]: 2008/03/03_13:00:45 info: main: Exiting... attrd[3929]: 2008/03/03_13:00:45 info: attrd_cib_connection_destroy: Connection to the CIB terminated... heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/stonithd process group 3928 with signal 15 stonithd[3928]: 2008/03/03_13:00:45 notice: /usr/lib64/heartbeat/stonithd normally quit. heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/lrmd -r process group 3927 with signal 15 lrmd[3927]: 2008/03/03_13:00:45 info: lrmd is shutting down heartbeat[3785]: 2008/03/03_13:00:45 info: killing /usr/lib64/heartbeat/cib process group 3926 with signal 15 cib[3926]: 2008/03/03_13:00:45 info: cib_shutdown: Disconnected 0 clients cib[3926]: 2008/03/03_13:00:45 info: cib_process_disconnect: All clients disconnected... cib[3926]: 2008/03/03_13:00:45 info: terminate_ha_connection: initiate_exit: Disconnecting heartbeat cib[3926]: 2008/03/03_13:00:45 info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting cib[3926]: 2008/03/03_13:00:45 info: main: Done ccm[3925]: 2008/03/03_13:00:46 info: client (pid=3926) removed from ccm heartbeat[3785]: 2008/03/03_13:00:46 info: killing /usr/lib64/heartbeat/ccm process group 3925 with signal 15 ccm[3925]: 2008/03/03_13:00:46 info: received SIGTERM, going to shut down heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBFIFO process 3790 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBWRITE process 3791 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: killing HBREAD process 3792 with signal 15 heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3792 exited. 3 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3790 exited. 2 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: Core process 3791 exited. 1 remaining heartbeat[3785]: 2008/03/03_13:00:47 info: db-mysql-test3-ha.ripe.net Heartbeat shutdown complete. logd[3771]: 2008/03/03_13:00:47 info: logd_term_write_action: received SIGTERM logd[3771]: 2008/03/03_13:00:47 info: Exiting write process
_______________________________________________ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems