On 07/06/18 12:00, Prasad Nagaraj wrote: > And here is how startup log looks on one of the nodes. > Its pretty much same on the other nodes as well >
That all looks fine to me (from a corosync POV anyway). Have you checked for any iptables rules getting in the way? Chrissie > Jun 07 10:41:37 corosync [MAIN ] Corosync Cluster Engine ('UNKNOWN'): > started and ready to provide service. > Jun 07 10:41:37 corosync [MAIN ] Corosync built-in features: nss > Jun 07 10:41:37 corosync [MAIN ] Successfully read main configuration > file '/etc/corosync/corosync.conf'. > Jun 07 10:41:38 corosync [TOTEM ] Initializing transport (UDP/IP Unicast). > Jun 07 10:41:38 corosync [TOTEM ] Initializing transmit/receive > security: libtomcrypt SOBER128/SHA1HMAC (mode 0). > Jun 07 10:41:38 corosync [TOTEM ] the ip i gave is [0.0.0.0] > Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log > Jun 07 10:41:38 corosync [TOTEM ] The network interface [172.22.0.4] is > now up. > Jun 07 10:41:38 corosync [pcmk ] info: process_ais_conf: Reading configure > Jun 07 10:41:38 corosync [pcmk ] ERROR: process_ais_conf: You have > configured a cluster using the Pacemaker plugin for Corosync. The plugin > is not supported in this environment and will be removed very soon. > Jun 07 10:41:38 corosync [pcmk ] ERROR: process_ais_conf: Please see > Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) > for details on using Pacemaker with CMAN > Jun 07 10:41:38 corosync [pcmk ] info: config_find_init: Local handle: > 2730409743423111169 for logging > Jun 07 10:41:38 corosync [pcmk ] info: config_find_next: Processing > additional logging options... > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Defaulting to > 'on' for option: debug > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Found 'yes' for > option: to_logfile > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Found > '/var/log/cluster/corosync.log' for option: logfile > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Found 'no' for > option: to_syslog > Jun 07 10:41:38 corosync [pcmk ] info: process_ais_conf: User > configured file based logging and explicitly disabled syslog. > Jun 07 10:41:38 corosync [pcmk ] info: config_find_init: Local handle: > 5880381755227111426 for quorum > Jun 07 10:41:38 corosync [pcmk ] info: config_find_next: No additional > configuration supplied for: quorum > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: No default for > option: provider > Jun 07 10:41:38 corosync [pcmk ] info: config_find_init: Local handle: > 4835695805891346435 for service > Jun 07 10:41:38 corosync [pcmk ] info: config_find_next: Processing > additional service options... > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Found '1' for > option: ver > Jun 07 10:41:38 corosync [pcmk ] info: process_ais_conf: Enabling MCP > mode: Use the Pacemaker init script to complete Pacemaker startup > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Defaulting to > 'pcmk' for option: clustername > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_logd > Jun 07 10:41:38 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_mgmtd > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_startup: CRM: Initialized > Jun 07 10:41:38 corosync [pcmk ] Logging: Initialized pcmk_startup > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_startup: Maximum core file > size is: 18446744073709551615 > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_startup: Service: 9 > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_startup: Local hostname: > vme6c95166f0 > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_update_nodeid: Local node > id: 67114668 > Jun 07 10:41:38 corosync [pcmk ] info: update_member: Creating entry > for node 67114668 born on 0 > Jun 07 10:41:38 corosync [pcmk ] info: update_member: 0x1ffdf20 Node > 67114668 now known as vme6c95166f0 (was: (null)) > Jun 07 10:41:38 corosync [pcmk ] info: update_member: Node vme6c95166f0 > now has 1 quorum votes (was 0) > Jun 07 10:41:38 corosync [pcmk ] info: update_member: Node > 67114668/vme6c95166f0 is now: member > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: Pacemaker > Cluster Manager 1.1.14-8.el6_8.1 > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > extended virtual synchrony service > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > configuration service > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > cluster closed process group service v1.01 > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > cluster config database access v1.01 > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > profile loading service > Jun 07 10:41:38 corosync [SERV ] Service engine loaded: corosync > cluster quorum service v0.1 > Jun 07 10:41:38 corosync [MAIN ] Compatibility mode set to whitetank. > Using V1 and V2 of the synchronization engine. > Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.4} > Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.11} > Jun 07 10:41:38 corosync [TOTEM ] adding new UDPU member {172.22.0.13} > Jun 07 10:41:38 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 77856: memb=0, new=0, lost=0 > Jun 07 10:41:38 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 77856: memb=1, new=1, lost=0 > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_peer_update: NEW: > vme6c95166f0 67114668 > Jun 07 10:41:38 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme6c95166f0 67114668 > Jun 07 10:41:38 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:41:38 corosync [CPG ] chosen downlist: sender r(0) > ip(172.22.0.4) ; members(old:0 left:0) > Jun 07 10:41:38 corosync [MAIN ] Completed service synchronization, > ready to provide service. > Jun 07 10:42:21 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 77868: memb=1, new=0, lost=0 > Jun 07 10:42:21 corosync [pcmk ] info: pcmk_peer_update: memb: > vme6c95166f0 67114668 > Jun 07 10:42:21 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 77868: memb=1, new=0, lost=0 > Jun 07 10:42:21 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme6c95166f0 67114668 > Jun 07 10:42:21 corosync [pcmk ] info: update_member: 0x1ffdf20 Node > 67114668 ((null)) born on: 77856 > Jun 07 10:42:21 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:42:21 corosync [CPG ] chosen downlist: sender r(0) > ip(172.22.0.4) ; members(old:1 left:0) > Jun 07 10:42:21 corosync [MAIN ] Completed service synchronization, > ready to provide service. > Jun 07 10:42:34 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 77876: memb=1, new=0, lost=0 > Jun 07 10:42:34 corosync [pcmk ] info: pcmk_peer_update: memb: > vme6c95166f0 67114668 > Jun 07 10:42:34 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 77876: memb=1, new=0, lost=0 > Jun 07 10:42:34 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme6c95166f0 67114668 > Jun 07 10:42:34 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Jun 07 10:49:47 corosync [MAIN ] Corosync Cluster Engine ('UNKNOWN'): > started and ready to provide service. > Jun 07 10:49:47 corosync [MAIN ] Corosync built-in features: nss > Jun 07 10:49:47 corosync [MAIN ] Successfully read main configuration > file '/etc/corosync/corosync.conf'. > Jun 07 10:49:47 corosync [TOTEM ] Initializing transport (UDP/IP Unicast). > Jun 07 10:49:47 corosync [TOTEM ] Initializing transmit/receive > security: libtomcrypt SOBER128/SHA1HMAC (mode 0). > Jun 07 10:49:47 corosync [TOTEM ] the ip i gave is [0.0.0.0] > Jun 07 10:49:47 corosync [TOTEM ] The network interface [172.22.0.13] is > now up. > Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log > Jun 07 10:49:47 corosync [pcmk ] info: process_ais_conf: Reading configure > Jun 07 10:49:47 corosync [pcmk ] ERROR: process_ais_conf: You have > configured a cluster using the Pacemaker plugin for Corosync. The plugin > is not supported in this environment and will be removed very soon. > Jun 07 10:49:47 corosync [pcmk ] ERROR: process_ais_conf: Please see > Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) > for details on using Pacemaker with CMAN > Jun 07 10:49:47 corosync [pcmk ] info: config_find_init: Local handle: > 2730409743423111169 for logging > Jun 07 10:49:47 corosync [pcmk ] info: config_find_next: Processing > additional logging options... > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Defaulting to > 'on' for option: debug > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Found 'yes' for > option: to_logfile > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Found > '/var/log/cluster/corosync.log' for option: logfile > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Found 'no' for > option: to_syslog > Jun 07 10:49:47 corosync [pcmk ] info: process_ais_conf: User > configured file based logging and explicitly disabled syslog. > Jun 07 10:49:47 corosync [pcmk ] info: config_find_init: Local handle: > 5880381755227111426 for quorum > Jun 07 10:49:47 corosync [pcmk ] info: config_find_next: No additional > configuration supplied for: quorum > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: No default for > option: provider > Jun 07 10:49:47 corosync [pcmk ] info: config_find_init: Local handle: > 4835695805891346435 for service > Jun 07 10:49:47 corosync [pcmk ] info: config_find_next: Processing > additional service options... > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Found '1' for > option: ver > Jun 07 10:49:47 corosync [pcmk ] info: process_ais_conf: Enabling MCP > mode: Use the Pacemaker init script to complete Pacemaker startup > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Defaulting to > 'pcmk' for option: clustername > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_logd > Jun 07 10:49:47 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_mgmtd > Jun 07 10:49:47 corosync [pcmk ] info: pcmk_startup: CRM: Initialized > Jun 07 10:49:47 corosync [pcmk ] Logging: Initialized pcmk_startup > Jun 07 10:49:47 corosync [pcmk ] info: pcmk_startup: Maximum core file > size is: 18446744073709551615 > Jun 07 10:49:47 corosync [pcmk ] info: pcmk_startup: Service: 9 > Jun 07 10:49:47 corosync [pcmk ] info: pcmk_startup: Local hostname: > vme0f7614459 > Jun 07 10:49:47 corosync [pcmk ] info: pcmk_update_nodeid: Local node > id: 218109612 > Jun 07 10:49:47 corosync [pcmk ] info: update_member: Creating entry > for node 218109612 born on 0 > Jun 07 10:49:47 corosync [pcmk ] info: update_member: 0x228ef20 Node > 218109612 now known as vme0f7614459 (was: (null)) > Jun 07 10:49:47 corosync [pcmk ] info: update_member: Node vme0f7614459 > now has 1 quorum votes (was 0) > Jun 07 10:49:47 corosync [pcmk ] info: update_member: Node > 218109612/vme0f7614459 is now: member > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: Pacemaker > Cluster Manager 1.1.14-8.el6_8.1 > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > extended virtual synchrony service > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > configuration service > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > cluster closed process group service v1.01 > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > cluster config database access v1.01 > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > profile loading service > Jun 07 10:49:47 corosync [SERV ] Service engine loaded: corosync > cluster quorum service v0.1 > Jun 07 10:49:47 corosync [MAIN ] Compatibility mode set to whitetank. > Using V1 and V2 of the synchronization engine. > Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.4} > Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.11} > Jun 07 10:49:47 corosync [TOTEM ] adding new UDPU member {172.22.0.13} > Jun 07 10:50:17 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78176: memb=0, new=0, lost=0 > Jun 07 10:50:17 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78176: memb=1, new=1, lost=0 > Jun 07 10:50:17 corosync [pcmk ] info: pcmk_peer_update: NEW: > vme0f7614459 218109612 > Jun 07 10:50:17 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme0f7614459 218109612 > Jun 07 10:50:17 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:50:17 corosync [CPG ] chosen downlist: sender r(0) > ip(172.22.0.13) ; members(old:0 left:0) > Jun 07 10:50:17 corosync [MAIN ] Completed service synchronization, > ready to provide service. > Jun 07 10:50:29 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78188: memb=1, new=0, lost=0 > Jun 07 10:50:29 corosync [pcmk ] info: pcmk_peer_update: memb: > vme0f7614459 218109612 > Jun 07 10:50:29 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78188: memb=1, new=0, lost=0 > Jun 07 10:50:29 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme0f7614459 218109612 > Jun 07 10:50:29 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:50:29 corosync [CPG ] chosen downlist: sender r(0) > ip(172.22.0.13) ; members(old:1 left:0) > Jun 07 10:50:29 corosync [MAIN ] Completed service synchronization, > ready to provide service. > Jun 07 10:50:41 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78200: memb=1, new=0, lost=0 > Jun 07 10:50:41 corosync [pcmk ] info: pcmk_peer_update: memb: > vme0f7614459 218109612 > Jun 07 10:50:41 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78200: memb=1, new=0, lost=0 > Jun 07 10:50:41 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vme0f7614459 218109612 > Jun 07 10:50:41 corosync [pcmk ] info: update_member: 0x228ef20 Node > 218109612 ((null)) born on: 78176 > Jun 07 10:50:41 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Jun 07 10:48:49 corosync [MAIN ] Corosync Cluster Engine ('UNKNOWN'): > started and ready to provide service. > Jun 07 10:48:49 corosync [MAIN ] Corosync built-in features: nss > Jun 07 10:48:49 corosync [MAIN ] Successfully read main configuration > file '/etc/corosync/corosync.conf'. > Jun 07 10:48:49 corosync [TOTEM ] Initializing transport (UDP/IP Unicast). > Jun 07 10:48:49 corosync [TOTEM ] Initializing transmit/receive > security: libtomcrypt SOBER128/SHA1HMAC (mode 0). > Jun 07 10:48:49 corosync [TOTEM ] the ip i gave is [0.0.0.0] > Jun 07 10:48:49 corosync [TOTEM ] The network interface [172.22.0.11] is > now up. > Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log > Jun 07 10:48:49 corosync [pcmk ] info: process_ais_conf: Reading configure > Jun 07 10:48:49 corosync [pcmk ] ERROR: process_ais_conf: You have > configured a cluster using the Pacemaker plugin for Corosync. The plugin > is not supported in this environment and will be removed very soon. > Jun 07 10:48:49 corosync [pcmk ] ERROR: process_ais_conf: Please see > Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) > for details on using Pacemaker with CMAN > Jun 07 10:48:49 corosync [pcmk ] info: config_find_init: Local handle: > 2730409743423111169 for logging > Jun 07 10:48:49 corosync [pcmk ] info: config_find_next: Processing > additional logging options... > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Defaulting to > 'on' for option: debug > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Found 'yes' for > option: to_logfile > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Found > '/var/log/cluster/corosync.log' for option: logfile > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Found 'no' for > option: to_syslog > Jun 07 10:48:49 corosync [pcmk ] info: process_ais_conf: User > configured file based logging and explicitly disabled syslog. > Jun 07 10:48:49 corosync [pcmk ] info: config_find_init: Local handle: > 5880381755227111426 for quorum > Jun 07 10:48:49 corosync [pcmk ] info: config_find_next: No additional > configuration supplied for: quorum > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: No default for > option: provider > Jun 07 10:48:49 corosync [pcmk ] info: config_find_init: Local handle: > 4835695805891346435 for service > Jun 07 10:48:49 corosync [pcmk ] info: config_find_next: Processing > additional service options... > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Found '1' for > option: ver > Jun 07 10:48:49 corosync [pcmk ] info: process_ais_conf: Enabling MCP > mode: Use the Pacemaker init script to complete Pacemaker startup > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Defaulting to > 'pcmk' for option: clustername > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_logd > Jun 07 10:48:49 corosync [pcmk ] info: get_config_opt: Defaulting to > 'no' for option: use_mgmtd > Jun 07 10:48:49 corosync [pcmk ] info: pcmk_startup: CRM: Initialized > Jun 07 10:48:49 corosync [pcmk ] Logging: Initialized pcmk_startup > Jun 07 10:48:49 corosync [pcmk ] info: pcmk_startup: Maximum core file > size is: 18446744073709551615 > Jun 07 10:48:49 corosync [pcmk ] info: pcmk_startup: Service: 9 > Jun 07 10:48:49 corosync [pcmk ] info: pcmk_startup: Local hostname: > vm2883711991 > Jun 07 10:48:49 corosync [pcmk ] info: pcmk_update_nodeid: Local node > id: 184555180 > Jun 07 10:48:49 corosync [pcmk ] info: update_member: Creating entry > for node 184555180 born on 0 > Jun 07 10:48:49 corosync [pcmk ] info: update_member: 0x1576f20 Node > 184555180 now known as vm2883711991 (was: (null)) > Jun 07 10:48:49 corosync [pcmk ] info: update_member: Node vm2883711991 > now has 1 quorum votes (was 0) > Jun 07 10:48:49 corosync [pcmk ] info: update_member: Node > 184555180/vm2883711991 is now: member > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: Pacemaker > Cluster Manager 1.1.14-8.el6_8.1 > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > extended virtual synchrony service > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > configuration service > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > cluster closed process group service v1.01 > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > cluster config database access v1.01 > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > profile loading service > Jun 07 10:48:49 corosync [SERV ] Service engine loaded: corosync > cluster quorum service v0.1 > Jun 07 10:48:49 corosync [MAIN ] Compatibility mode set to whitetank. > Using V1 and V2 of the synchronization engine. > Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.4} > Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.11} > Jun 07 10:48:49 corosync [TOTEM ] adding new UDPU member {172.22.0.13} > Jun 07 10:49:16 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78140: memb=0, new=0, lost=0 > Jun 07 10:49:16 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78140: memb=1, new=1, lost=0 > Jun 07 10:49:16 corosync [pcmk ] info: pcmk_peer_update: NEW: > vm2883711991 184555180 > Jun 07 10:49:16 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vm2883711991 184555180 > Jun 07 10:49:16 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:49:28 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78148: memb=1, new=0, lost=0 > Jun 07 10:49:28 corosync [pcmk ] info: pcmk_peer_update: memb: > vm2883711991 184555180 > Jun 07 10:49:28 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78148: memb=1, new=0, lost=0 > Jun 07 10:49:28 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vm2883711991 184555180 > Jun 07 10:49:28 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:49:40 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78152: memb=1, new=0, lost=0 > Jun 07 10:49:40 corosync [pcmk ] info: pcmk_peer_update: memb: > vm2883711991 184555180 > Jun 07 10:49:40 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78152: memb=1, new=0, lost=0 > Jun 07 10:49:40 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vm2883711991 184555180 > Jun 07 10:49:40 corosync [pcmk ] info: update_member: 0x1576f20 Node > 184555180 ((null)) born on: 78140 > Jun 07 10:49:40 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > Jun 07 10:49:40 corosync [CPG ] chosen downlist: sender r(0) > ip(172.22.0.11) ; members(old:0 left:0) > Jun 07 10:49:40 corosync [MAIN ] Completed service synchronization, > ready to provide service. > Jun 07 10:49:52 corosync [pcmk ] notice: pcmk_peer_update: Transitional > membership event on ring 78160: memb=1, new=0, lost=0 > Jun 07 10:49:52 corosync [pcmk ] info: pcmk_peer_update: memb: > vm2883711991 184555180 > Jun 07 10:49:52 corosync [pcmk ] notice: pcmk_peer_update: Stable > membership event on ring 78160: memb=1, new=0, lost=0 > Jun 07 10:49:52 corosync [pcmk ] info: pcmk_peer_update: MEMB: > vm2883711991 184555180 > Jun 07 10:49:52 corosync [TOTEM ] A processor joined or left the > membership and a new membership was formed. > > > > On Thu, Jun 7, 2018 at 4:01 PM, Prasad Nagaraj > <prasad.nagara...@gmail.com <mailto:prasad.nagara...@gmail.com>> wrote: > > Hi Christine - > > Thanks for looking into this and here are the details. > All the nodes are pingable from each other and actively exchanging > corosync packets from each other as seen from tcpdump > > Here is the ifconfig out from each of the node > # ifconfig > eth0 Link encap:Ethernet HWaddr 00:0D:3A:03:35:64 > inet addr:172.22.0.4 Bcast:172.22.0.255 Mask:255.255.255.0 > inet6 addr: fe80::20d:3aff:fe03:3564/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:3721169 errors:0 dropped:0 overruns:0 frame:0 > TX packets:2780455 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1229505889 (1.1 GiB) TX bytes:982021535 (936.5 MiB) > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:1367018 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1367018 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:459591075 (438.3 MiB) TX bytes:459591075 (438.3 MiB) > > ----------------------------------------------------------------------------------------------------- > # ifconfig > eth0 Link encap:Ethernet HWaddr 00:0D:3A:03:38:D7 > inet addr:172.22.0.11 Bcast:172.22.0.255 Mask:255.255.255.0 > inet6 addr: fe80::20d:3aff:fe03:38d7/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4052226 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3744671 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1930027786 (1.7 GiB) TX bytes:1180930029 (1.0 GiB) > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:1394930 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1394930 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:508170210 (484.6 MiB) TX bytes:508170210 (484.6 MiB) > > -------------------------------------------------------------------------------------------------- > > # ifconfig > eth0 Link encap:Ethernet HWaddr 00:0D:3A:04:06:F6 > inet addr:172.22.0.13 Bcast:172.22.0.255 Mask:255.255.255.0 > inet6 addr: fe80::20d:3aff:fe04:6f6/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:3974698 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3891546 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:1903617077 (1.7 GiB) TX bytes:1234961001 (1.1 GiB) > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:1503643 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1503643 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:541177718 (516.1 MiB) TX bytes:541177718 (516.1 MiB) > > --------------------------------------------------------------------------------------------------------------- > tcpdump snippet from each node is as below > > ------------------------------------------------------------------------------------------------------------- > > ------------------------------------------------------------------------------------------------------------ > > ]# tcpdump udp port 5405 > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes > 10:21:23.077694 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.084548 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.095320 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.095334 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:21:23.144459 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.144477 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.155382 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.155398 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:21:23.204543 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.204563 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.205786 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:21:23.206175 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.264298 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.264322 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.264954 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.265077 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:21:23.323183 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.323202 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.324683 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:21:23.324696 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.382816 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:21:23.382834 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:21:23.383349 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:21:23.383802 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > > ------------------------------------------------------------------------------------------------------------------ > > tcpdump udp port 5405 > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes > 10:23:16.891854 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:16.891871 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:16.891927 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:23:16.891939 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:23:16.913586 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, > length 156 > 10:23:16.945218 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:16.945236 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:16.945305 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:23:16.945326 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:23:16.972761 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, > length 156 > 10:23:17.004903 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:23:17.004919 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.004928 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.004938 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:23:17.031846 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, > length 156 > 10:23:17.059219 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.059306 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:23:17.059317 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:23:17.059941 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.091094 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, > length 156 > 10:23:17.116600 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.116662 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:23:17.116672 IP 172.22.0.4.40755 > 172.22.0.13.netsupport: UDP, > length 376 > 10:23:17.117347 IP 172.22.0.13.50468 > 172.22.0.4.netsupport: UDP, > length 332 > 10:23:17.140960 IP 172.22.0.8.50438 > 172.22.0.4.netsupport: UDP, > length 82 > 10:23:17.141319 IP 172.22.0.6.38535 > 172.22.0.4.netsupport: UDP, > length 156 > 10:23:17.176250 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > > --------------------------------------------------------------------------------------------------------------------- > # tcpdump udp port 5405 > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes > 10:25:30.562549 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.562576 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.588697 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.588728 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.619383 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.624817 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.647937 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.647968 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.672207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.684604 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.707733 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.707760 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.731354 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.744345 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.767456 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.767483 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.791532 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.804432 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.827539 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.827563 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.850832 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.863531 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.886664 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.886691 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.910820 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.923403 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:30.946507 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:30.946531 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:30.970931 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > 10:25:30.983055 IP 172.22.0.13.57332 > 172.22.0.11.netsupport: UDP, > length 332 > 10:25:31.006306 IP 172.22.0.11.54545 > 172.22.0.4.netsupport: UDP, > length 332 > 10:25:31.006339 IP 172.22.0.11.44864 > 172.22.0.13.netsupport: UDP, > length 332 > 10:25:31.030207 IP 172.22.0.4.34060 > 172.22.0.11.netsupport: UDP, > length 376 > > > ---------------------------------------------------------------------------------------------------------------- > And here is the lsof output for each node. > lsof -i | grep corosync > corosync 47873 root 10u IPv4 1193147 0t0 UDP > 172.22.0.4:netsupport > corosync 47873 root 13u IPv4 1193151 0t0 UDP > 172.22.0.4:45846 <http://172.22.0.4:45846> > corosync 47873 root 14u IPv4 1193152 0t0 UDP > 172.22.0.4:34060 <http://172.22.0.4:34060> > corosync 47873 root 15u IPv4 1193153 0t0 UDP > 172.22.0.4:40755 <http://172.22.0.4:40755> > > lsof -i | grep corosync > corosync 11039 root 10u IPv4 54862 0t0 UDP > 172.22.0.13:netsupport > corosync 11039 root 13u IPv4 54869 0t0 UDP > 172.22.0.13:50468 <http://172.22.0.13:50468> > corosync 11039 root 14u IPv4 54870 0t0 UDP > 172.22.0.13:57332 <http://172.22.0.13:57332> > corosync 11039 root 15u IPv4 54871 0t0 UDP > 172.22.0.13:46460 <http://172.22.0.13:46460> > > lsof -i | grep corosync > corosync 75188 root 10u IPv4 1582737 0t0 UDP > 172.22.0.11:netsupport > corosync 75188 root 13u IPv4 1582741 0t0 UDP > 172.22.0.11:54545 <http://172.22.0.11:54545> > corosync 75188 root 14u IPv4 1582742 0t0 UDP > 172.22.0.11:53213 <http://172.22.0.11:53213> > corosync 75188 root 15u IPv4 1582743 0t0 UDP > 172.22.0.11:44864 <http://172.22.0.11:44864> > > > Thanks! > > > > On Thu, Jun 7, 2018 at 3:33 PM, Christine Caulfield > <ccaul...@redhat.com <mailto:ccaul...@redhat.com>> wrote: > > On 07/06/18 09:21, Prasad Nagaraj wrote: > > Hi - I am running corosync on 3 nodes of CentOS release 6.9 > (Final). > > Corosync version is corosync-1.4.7. > > The nodes are not seeing each other and not able to form > memberships. > > What I see is continuous message about " A processor joined or left > the > > membership and a new membership was formed." > > For example:on node: vm2883711991 > > > > I can't draw any conclusions from the logs, we'd need to see what > corosync though it was binding to and the IP addresses of the hosts. > > Have a look at the start of the logs and see if they match what > you'd > expect (ie are similar to the ones on the working clusters), > Also check > using lsof, to see what addresses corosync is bound to. tcpdump > on port > 5405 will show you if traffic is leaving the nodes and being > received. > > Also check firewall settings and make sure the nodes can ping > each other. > > If you're still stumped them feel free to post more info here > for us to > look at, though if you have that configuration working on other > nodes it > might be something in your environment > > Chrissie > > > > > > Jun 07 07:54:52 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:54:52 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:54:52 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:54:52 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:04 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71084: memb=1, new=0, lost=0 > > Jun 07 07:55:04 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:55:04 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71084: memb=1, new=0, lost=0 > > Jun 07 07:55:04 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:55:04 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:04 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:55:04 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71096: memb=1, new=0, lost=0 > > Jun 07 07:55:16 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:55:16 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71096: memb=1, new=0, lost=0 > > Jun 07 07:55:16 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:55:16 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:16 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:55:16 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:28 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71108: memb=1, new=0, lost=0 > > Jun 07 07:55:28 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:55:28 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71108: memb=1, new=0, lost=0 > > Jun 07 07:55:28 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:55:28 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:28 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:55:28 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:40 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71120: memb=1, new=0, lost=0 > > Jun 07 07:55:40 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:55:40 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71120: memb=1, new=0, lost=0 > > Jun 07 07:55:40 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:55:40 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:40 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:55:40 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:52 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71132: memb=1, new=0, lost=0 > > Jun 07 07:55:52 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:55:52 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71132: memb=1, new=0, lost=0 > > Jun 07 07:55:52 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:55:52 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:52 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:55:52 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:56:04 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71144: memb=1, new=0, lost=0 > > Jun 07 07:56:04 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:56:04 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71144: memb=1, new=0, lost=0 > > Jun 07 07:56:04 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:56:04 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:17 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71156: memb=1, new=0, lost=0 > > Jun 07 07:56:17 corosync [pcmk ] info: pcmk_peer_update: memb: > > vm2883711991 184555180 > > Jun 07 07:56:17 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71156: memb=1, new=0, lost=0 > > Jun 07 07:56:17 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vm2883711991 184555180 > > Jun 07 07:56:17 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:17 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.11) ; members(old:1 left:0) > > Jun 07 07:56:17 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > > > Similarly on node: vme6c95166f0 > > > > Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71088: memb=1, new=0, lost=0 > > Jun 07 07:55:12 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:55:12 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71088: memb=1, new=0, lost=0 > > Jun 07 07:55:12 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:55:12 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:12 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.4) ; members(old:1 left:0) > > Jun 07 07:55:12 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71100: memb=1, new=0, lost=0 > > Jun 07 07:55:24 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:55:24 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71100: memb=1, new=0, lost=0 > > Jun 07 07:55:24 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:55:24 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:24 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.4) ; members(old:1 left:0) > > Jun 07 07:55:24 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:55:37 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71112: memb=1, new=0, lost=0 > > Jun 07 07:55:37 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:55:37 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71112: memb=1, new=0, lost=0 > > Jun 07 07:55:37 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:55:37 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:49 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71124: memb=1, new=0, lost=0 > > Jun 07 07:55:49 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:55:49 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71124: memb=1, new=0, lost=0 > > Jun 07 07:55:49 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:55:49 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:55:49 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.4) ; members(old:1 left:0) > > Jun 07 07:55:49 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:56:02 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71136: memb=1, new=0, lost=0 > > Jun 07 07:56:02 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:56:02 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71136: memb=1, new=0, lost=0 > > Jun 07 07:56:02 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:56:02 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:02 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.4) ; members(old:1 left:0) > > Jun 07 07:56:02 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > Jun 07 07:56:14 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71148: memb=1, new=0, lost=0 > > Jun 07 07:56:14 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:56:14 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71148: memb=1, new=0, lost=0 > > Jun 07 07:56:14 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:56:14 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:27 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71160: memb=1, new=0, lost=0 > > Jun 07 07:56:27 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:56:27 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71160: memb=1, new=0, lost=0 > > Jun 07 07:56:27 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:56:27 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:39 corosync [pcmk ] notice: pcmk_peer_update: > Transitional > > membership event on ring 71172: memb=1, new=0, lost=0 > > Jun 07 07:56:39 corosync [pcmk ] info: pcmk_peer_update: memb: > > vme6c95166f0 67114668 > > Jun 07 07:56:39 corosync [pcmk ] notice: pcmk_peer_update: Stable > > membership event on ring 71172: memb=1, new=0, lost=0 > > Jun 07 07:56:39 corosync [pcmk ] info: pcmk_peer_update: MEMB: > > vme6c95166f0 67114668 > > Jun 07 07:56:39 corosync [TOTEM ] A processor joined or left the > > membership and a new membership was formed. > > Jun 07 07:56:39 corosync [CPG ] chosen downlist: sender r(0) > > ip(172.22.0.4) ; members(old:1 left:0) > > Jun 07 07:56:39 corosync [MAIN ] Completed service > synchronization, > > ready to provide service. > > > > and similar message on the third node as well. > > > > my corosync.conf is as follows: > > compatibility: whitetank > > totem { > > version: 2 > > secauth: off > > threads: 0 > > interface { > > member { > > memberaddr: 172.22.0.4 > > } > > member { > > memberaddr: 172.22.0.11 > > } > > member { > > memberaddr: 172.22.0.13 > > } > > > > bindnetaddr: 172.22.0.4 > > > > ringnumber: 0 > > mcastport: 5405 > > ttl: 1 > > } > > transport: udpu > > token: 10000 > > token_retransmits_before_loss_const: 10 > > } > > > > logging { > > fileline: off > > to_stderr: yes > > to_logfile: yes > > to_syslog: no > > logfile: /var/log/cluster/corosync.log > > timestamp: on > > logger_subsys { > > subsys: AMF > > debug: on > > } > > } > > service { > > name: pacemaker > > ver: 1 > > } > > amf { > > mode: disabled > > } > > > > In general , this configuration has worked for me in other > clusters I > > have, but this particular setup is running into this issue. > Request help > > on how to debug and resolve this condition. > > > > Thanks > > > > > > > > _______________________________________________ > > Users mailing list: Users@clusterlabs.org > <mailto:Users@clusterlabs.org> > > https://lists.clusterlabs.org/mailman/listinfo/users > <https://lists.clusterlabs.org/mailman/listinfo/users> > > > > Project Home: http://www.clusterlabs.org > > Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf> > > Bugs: http://bugs.clusterlabs.org > > > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > <mailto:Users@clusterlabs.org> > https://lists.clusterlabs.org/mailman/listinfo/users > <https://lists.clusterlabs.org/mailman/listinfo/users> > > Project Home: http://www.clusterlabs.org > Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf> > Bugs: http://bugs.clusterlabs.org > > > > > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org