Hi, Firstly thank you for such a great tool.
When testing pacemaker I encountered a start error, which seems to be related to reported libqb segmentation fault. - cluster started and acquired quorum - some nodes failed to connect to CIB, and lost membership as a result - restart solved the problem Segmentation fault reports libqb library in version 0.17.1, a standard package provided for CentOS.6. Please let me know if the problem is known, and if there is a remedy (e.g. using the latest libqb). Logs are below. Thank you in advance, Logs from /var/log/messages: Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Configured corosync to accept connections from group 498: Library error (2) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Starting Pacemaker 1.1.13-1.el6 (Build: 577898d): generated-manpages agent-manpages ncurses libqb-logging libqb-ipc upstart nagios corosync-native atomic-attrd acls Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Quorum acquired Apr 22 15:46:41 (...) pacemakerd[111190]: notice: pcmk_quorum_notification: Node (...)[3] - state is now member (was (null)) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: pcmk_quorum_notification: Node (...)[4] - state is now member (was (null)) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: pcmk_quorum_notification: Node (...)[2] - state is now member (was (null)) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: pcmk_quorum_notification: Node (...)[1] - state is now member (was (null)) Apr 22 15:46:41 (...) lrmd[111194]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) stonith-ng[111193]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) cib[111192]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) attrd[111195]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) stonith-ng[111193]: notice: Connecting to cluster infrastructure: corosync Apr 22 15:46:41 (...) pengine[111196]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) attrd[111195]: notice: Connecting to cluster infrastructure: corosync Apr 22 15:46:41 (...) crmd[111197]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) crmd[111197]: notice: CRM Git Version: 1.1.13-1.el6 (577898d) Apr 22 15:46:41 (...) attrd[111195]: error: Could not connect to the Cluster Process Group API: 11 Apr 22 15:46:41 (...) attrd[111195]: error: Cluster connection failed Apr 22 15:46:41 (...) attrd[111195]: notice: Cleaning up before exit Apr 22 15:46:41 (...) stonith-ng[111193]: notice: crm_update_peer_proc: Node (...)[3] - state is now member (was (null)) Apr 22 15:46:41 (...) pacemakerd[111190]: error: Managed process 111195 (attrd) dumped core Apr 22 15:46:41 (...) pacemakerd[111190]: error: The attrd process (111195) terminated with signal 11 (core=1) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Respawning failed child process: attrd Apr 22 15:46:41 (...) cib[111192]: notice: Connecting to cluster infrastructure: corosync Apr 22 15:46:41 (...) cib[111192]: error: Could not connect to the Cluster Process Group API: 11 Apr 22 15:46:41 (...) cib[111192]: crit: Cannot sign in to the cluster... terminating Apr 22 15:46:41 (...) kernel: [17169.112132] attrd[111195]: segfault at 1b8 ip 00007f6fc9dc3181 sp 00007ffd7cf668f0 error 4 in libqb.so.0.17.1[7f6fc9db4000+21000] Apr 22 15:46:41 (...) pacemakerd[111190]: warning: The cib process (111192) can no longer be respawned, shutting the cluster down. Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Shutting down Pacemaker Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Stopping crmd: Sent -15 to process 111197 Apr 22 15:46:41 (...) attrd[111198]: notice: Additional logging available in /var/log/pacemaker.log Apr 22 15:46:41 (...) crmd[111197]: warning: Couldn't complete CIB registration 1 times... pause and retry Apr 22 15:46:41 (...) crmd[111197]: notice: Invoking handler for signal 15: Terminated Apr 22 15:46:41 (...) crmd[111197]: notice: Requesting shutdown, upper limit is 1200000ms Apr 22 15:46:41 (...) crmd[111197]: warning: FSA: Input I_SHUTDOWN from crm_shutdown() received in state S_STARTING Apr 22 15:46:41 (...) crmd[111197]: notice: State transition S_STARTING -> S_STOPPING [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] Apr 22 15:46:41 (...) crmd[111197]: notice: Disconnecting from Corosync Apr 22 15:46:41 (...) attrd[111198]: notice: Connecting to cluster infrastructure: corosync Apr 22 15:46:41 (...) attrd[111198]: error: Could not connect to the Cluster Process Group API: 11 Apr 22 15:46:41 (...) attrd[111198]: error: Cluster connection failed Apr 22 15:46:41 (...) attrd[111198]: notice: Cleaning up before exit Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Stopping pengine: Sent -15 to process 111196 Apr 22 15:46:41 (...) pengine[111196]: notice: Invoking handler for signal 15: Terminated Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Stopping attrd: Sent -15 to process 111198 Apr 22 15:46:41 (...) pacemakerd[111190]: error: Managed process 111198 (attrd) dumped core Apr 22 15:46:41 (...) pacemakerd[111190]: error: The attrd process (111198) terminated with signal 11 (core=1) Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Stopping lrmd: Sent -15 to process 111194 Apr 22 15:46:41 (...) lrmd[111194]: notice: Invoking handler for signal 15: Terminated Apr 22 15:46:41 (...) pacemakerd[111190]: notice: Stopping stonith-ng: Sent -15 to process 111193 Apr 22 15:46:41 (...) kernel: [17169.121628] attrd[111198]: segfault at 1b8 ip 00007f3a98f66181 sp 00007ffe33407380 error 4 in libqb.so.0.17.1[7f3a98f57000+21000] Apr 22 15:46:50 (...) stonith-ng[111193]: error: Could not connect to the CIB service: Transport endpoint is not connected (-107) Apr 22 15:46:50 (...) stonith-ng[111193]: notice: Invoking handler for signal 15: Terminated Apr 22 15:46:50 (...) pacemakerd[111190]: notice: Shutdown complete Apr 22 15:46:50 (...) pacemakerd[111190]: notice: Attempting to inhibit respawning after fatal error Logs from corosync log: Apr 22 15:46:22 [93582] (...) corosync notice [MAIN ] Corosync Cluster Engine exiting normally Apr 22 15:46:40 [111147] (...) corosync notice [MAIN ] Corosync Cluster Engine ('2.3.5.12-a71e'): started and ready to provide service. Apr 22 15:46:40 [111147] (...) corosync info [MAIN ] Corosync built-in features: dbus pie relro bindnow Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] Initializing transport (UDP/IP Unicast). Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] Initializing transmit/receive security (NSS) crypto: none hash: none Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] The network interface [(...)] is now up. Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync configuration map access [0] Apr 22 15:46:40 [111147] (...) corosync info [QB ] server name: cmap Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync configuration service [1] Apr 22 15:46:40 [111147] (...) corosync info [QB ] server name: cfg Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2] Apr 22 15:46:40 [111147] (...) corosync info [QB ] server name: cpg Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync profile loading service [4] Apr 22 15:46:40 [111147] (...) corosync notice [QUORUM] Using quorum provider corosync_votequorum Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5] Apr 22 15:46:40 [111147] (...) corosync info [QB ] server name: votequorum Apr 22 15:46:40 [111147] (...) corosync notice [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3] Apr 22 15:46:40 [111147] (...) corosync info [QB ] server name: quorum Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] adding new UDPU member {(...)} Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] adding new UDPU member {(...)} Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] adding new UDPU member {(...)} Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] adding new UDPU member {(...)} Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] A new membership ((...):660) was formed. Members joined: 3 Apr 22 15:46:40 [111147] (...) corosync notice [QUORUM] Members[1]: 3 Apr 22 15:46:40 [111147] (...) corosync notice [MAIN ] Completed service synchronization, ready to provide service. Apr 22 15:46:40 [111147] (...) corosync notice [TOTEM ] A new membership ((...):664) was formed. Members joined: 4 2 1 Apr 22 15:46:40 [111147] (...) corosync notice [QUORUM] This node is within the primary component and will provide service. Apr 22 15:46:40 [111147] (...) corosync notice [QUORUM] Members[4]: 3 4 2 1 Apr 22 15:46:40 [111147] (...) corosync notice [MAIN ] Completed service synchronization, ready to provide service. Apr 22 15:46:41 [111147] (...) corosync error [MAIN ] Denied connection attempt from 498:498 Apr 22 15:46:41 [111147] (...) corosync error [QB ] Invalid IPC credentials (111148-111195-2). Apr 22 15:46:41 [111147] (...) corosync error [MAIN ] Denied connection attempt from 498:498 Apr 22 15:46:41 [111147] (...) corosync error [QB ] Invalid IPC credentials (111148-111192-2). Apr 22 15:46:41 [111147] (...) corosync error [MAIN ] Denied connection attempt from 498:498 Apr 22 15:46:41 [111147] (...) corosync error [QB ] Invalid IPC credentials (111148-111198-2). -- Best Regards, Radoslaw Garbacz XtremeData Incorporation
_______________________________________________ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org