Hi Digimer,
Yes, I just did. Looks like they are failing. I'm not sure why that is.
Please see the attachment for all servers log.
By the way, I do appreciated all the helps I can get.
Vinh
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Digimer
Sent: Wednesday, January 07, 2015 4:33 PM
To: linux clustering
Subject: Re: [Linux-cluster] needs helps GFS2 on 5 nodes cluster
Quorum is enabled by default. I need to see the entire logs from all five
nodes, as I mentioned in the first email. Please disable cman from starting on
boot, configure fencing properly and then reboot all nodes cleanly. Start the
'tail -f -n 0 /var/log/messages' on all five nodes, then in another window,
start cman on all five nodes. When things settle down, copy/paste all the log
output please.
On 07/01/15 04:29 PM, Cao, Vinh wrote:
> Hi Digimer,
>
> Here is from the logs:
> [root@ustlvcmsp1954 ~]# tail -f /var/log/messages
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> loaded: corosync profile loading service
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [QUORUM] Using quorum
> provider quorum_cman
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> loaded: corosync cluster quorum service v0.1
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [MAIN ] Compatibility mode
> set to whitetank. Using V1 and V2 of the synchronization engine.
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [TOTEM ] A processor joined
> or left the membership and a new membership was formed.
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [QUORUM] Members[1]: 1
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [QUORUM] Members[1]: 1
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [CPG ] chosen downlist:
> sender r(0) ip(10.30.197.108) ; members(old:0 left:0)
> Jan 7 16:14:01 ustlvcmsp1954 corosync[8182]: [MAIN ] Completed service
> synchronization, ready to provide service.
> Jan 7 16:14:01 ustlvcmsp1954 rgmanager[8099]: Waiting for quorum to form
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Unloading all
> Corosync service engines.
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync extended virtual synchrony service
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync configuration service
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync cluster closed process group service v1.01
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync cluster config database access v1.01
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync profile loading service
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: openais checkpoint service B.01.01
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync CMAN membership service 2.90
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [SERV ] Service engine
> unloaded: corosync cluster quorum service v0.1
> Jan 7 16:15:06 ustlvcmsp1954 corosync[8182]: [MAIN ] Corosync Cluster
> Engine exiting with status 0 at main.c:2055.
> Jan 7 16:15:06 ustlvcmsp1954 rgmanager[8099]: Quorum formed
>
> Then it die at:
> Starting cman... [ OK ]
> Waiting for quorum... Timed-out waiting for cluster
> [FAILED]
>
> Yes, I did made changes with: <fence_daemon post_join_delay="30"/> the
> problem is still there. One thing I don't know why cluster is looking for
> quorum?
> I did have any disk quorum setup in cluster.conf file.
>
> Any helps can I get appreciated.
>
> Vinh
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Digimer
> Sent: Wednesday, January 07, 2015 3:59 PM
> To: linux clustering
> Subject: Re: [Linux-cluster] needs helps GFS2 on 5 nodes cluster
>
> On 07/01/15 03:39 PM, Cao, Vinh wrote:
>> Hello Digimer,
>>
>> Yes, I would agrre with you RHEL6.4 is old. We patched monthly, but I'm not
>> sure why these servers are still at 6.4. Most of our system are 6.6.
>>
>> Here is my cluster config. All I want is using cluster to have BGFS2 mount
>> via /etc/fstab.
>> root@ustlvcmsp1955 ~]# cat /etc/cluster/cluster.conf <?xml
>> version="1.0"?> <cluster config_version="15" name="p1954_to_p1958">
>> <clusternodes>
>> <clusternode name="ustlvcmsp1954" nodeid="1"/>
>> <clusternode name="ustlvcmsp1955" nodeid="2"/>
>> <clusternode name="ustlvcmsp1956" nodeid="3"/>
>> <clusternode name="ustlvcmsp1957" nodeid="4"/>
>> <clusternode name="ustlvcmsp1958" nodeid="5"/>
>> </clusternodes>
>
> You don't configure the fencing for the nodes... If anything causes a fence,
> the cluster will lock up (by design).
>
>> <fencedevices>
>> <fencedevice agent="fence_vmware_soap"
>> ipaddr="10.30.197.108" login="rhfence" name="p1954" passwd="xxxxxxxx"/>
>> <fencedevice agent="fence_vmware_soap"
>> ipaddr="10.30.197.109" login="rhfence" name="p1955" passwd=" xxxxxxxx "/>
>> <fencedevice agent="fence_vmware_soap"
>> ipaddr="10.30.197.110" login="rhfence" name="p1956" passwd=" xxxxxxxx "/>
>> <fencedevice agent="fence_vmware_soap"
>> ipaddr="10.30.197.111" login="rhfence" name="p1957" passwd=" xxxxxxxx "/>
>> <fencedevice agent="fence_vmware_soap"
>> ipaddr="10.30.197.112" login="rhfence" name="p1958" passwd=" xxxxxxxx "/>
>> </fencedevices>
>> </cluster>
>>
>> clustat show:
>>
>> Cluster Status for p1954_to_p1958 @ Wed Jan 7 15:38:00 2015 Member
>> Status: Quorate
>>
>> Member Name ID
>> Status
>> ------ ---- ----
>> ------
>> ustlvcmsp1954 1
>> Offline
>> ustlvcmsp1955 2
>> Online, Local
>> ustlvcmsp1956 3
>> Online
>> ustlvcmsp1957 4
>> Offline
>> ustlvcmsp1958 5
>> Online
>>
>> I need to make them all online, so I can use fencing for mounting shared
>> disk.
>>
>> Thanks,
>> Vinh
>
> What about the log entries from the start-up? Did you try the post_join_delay
> config?
>
>
>> -----Original Message-----
>> From: [email protected]
>> [mailto:[email protected]] On Behalf Of Digimer
>> Sent: Wednesday, January 07, 2015 3:16 PM
>> To: linux clustering
>> Subject: Re: [Linux-cluster] needs helps GFS2 on 5 nodes cluster
>>
>> My first though would be to set <fence_daemon post_join_delay="30" /> in
>> cluster.conf.
>>
>> If that doesn't work, please share your configuration file. Then, with all
>> nodes offline, open a terminal to each node and run 'tail -f -n 0
>> /var/log/messages'. With that running, start all the nodes and wait for
>> things to settle down, then paste the five nodes' output as well.
>>
>> Also, 6.4 is pretty old, why not upgrade to 6.6?
>>
>> digimer
>>
>> On 07/01/15 03:10 PM, Cao, Vinh wrote:
>>> Hello Cluster guru,
>>>
>>> I'm trying to setup Redhat 6.4 OS cluster with 5 nodes. With two
>>> nodes I don't have any issue.
>>>
>>> But with 5 nodes, when I ran clustat I got 3 nodes online and the
>>> other two off line.
>>>
>>> When I start the one that are off line. Service cman start. I got:
>>>
>>> [root@ustlvcmspxxx ~]# service cman status
>>>
>>> corosync is stopped
>>>
>>> [root@ustlvcmsp1954 ~]# service cman start
>>>
>>> Starting cluster:
>>>
>>> Checking if cluster has been disabled at boot... [ OK ]
>>>
>>> Checking Network Manager... [ OK ]
>>>
>>> Global setup... [ OK ]
>>>
>>> Loading kernel modules... [ OK ]
>>>
>>> Mounting configfs... [ OK ]
>>>
>>> Starting cman... [ OK ]
>>>
>>> Waiting for quorum... Timed-out waiting for cluster
>>>
>>>
>>> [FAILED]
>>>
>>> Stopping cluster:
>>>
>>> Leaving fence domain... [ OK ]
>>>
>>> Stopping gfs_controld... [ OK ]
>>>
>>> Stopping dlm_controld... [ OK ]
>>>
>>> Stopping fenced... [ OK ]
>>>
>>> Stopping cman... [ OK ]
>>>
>>> Waiting for corosync to shutdown: [ OK ]
>>>
>>> Unloading kernel modules... [ OK ]
>>>
>>> Unmounting configfs... [ OK ]
>>>
>>> Can you help?
>>>
>>> Thank you,
>>>
>>> Vinh
>>>
>>>
>>>
>>
>>
>> --
>> Digimer
>> Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is
>> trapped in the mind of a person without access to education?
>>
>> --
>> Linux-cluster mailing list
>> [email protected]
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is
trapped in the mind of a person without access to education?
--
Linux-cluster mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-cluster
[root@ustlvcmsp1954 ~]# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum...
Timed-out waiting for cluster
[FAILED]
===logs bellow.
Jan 7 17:16:08 ustlvcmsp1954 kernel: SCTP: Hash tables configured (established
65536 bind 65536)
Jan 7 17:16:08 ustlvcmsp1954 kernel: DLM (built Dec 12 2014 16:06:44) installed
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Corosync built-in
features: nss dbus rdma snmp
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Successfully read
config from /etc/cluster/cluster.conf
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Successfully parsed
cman config
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [TOTEM ] Initializing transport
(UDP/IP Multicast).
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [TOTEM ] The network interface
[10.30.197.108] is now up.
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [CMAN ] CMAN 3.0.12.1 (built
Jul 3 2014 11:37:43) started
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync CMAN membership service 2.90
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
openais checkpoint service B.01.01
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync extended virtual synchrony service
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync configuration service
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync cluster closed process group service v1.01
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync cluster config database access v1.01
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync profile loading service
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Compatibility mode set
to whitetank. Using V1 and V2 of the synchronization engine.
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [QUORUM] Members[1]: 1
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [QUORUM] Members[1]: 1
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.108) ; members(old:0 left:0)
Jan 7 17:16:08 ustlvcmsp1954 corosync[4376]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Unloading all Corosync
service engines.
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync extended virtual synchrony service
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync configuration service
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync cluster closed process group service v1.01
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync cluster config database access v1.01
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync profile loading service
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: openais checkpoint service B.01.01
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync CMAN membership service 2.90
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [SERV ] Service engine
unloaded: corosync cluster quorum service v0.1
Jan 7 17:17:13 ustlvcmsp1954 corosync[4376]: [MAIN ] Corosync Cluster
Engine exiting with status 0 at main.c:2055.
==================p1955
[root@ustlvcmsp1955 ~]# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum...
Timed-out waiting for cluster
[FAILED]
Stopping cluster:
Leaving fence domain... [ OK ]
Stopping gfs_controld... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown: [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
---logs
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [MAIN ] Corosync built-in
features: nss dbus rdma snmp
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [MAIN ] Successfully read
config from /etc/cluster/cluster.conf
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [MAIN ] Successfully parsed
cman config
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [TOTEM ] Initializing transport
(UDP/IP Multicast).
Jan 7 17:19:09 ustlvcmsp1955 corosync[4273]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [TOTEM ] The network interface
[10.30.197.109] is now up.
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [CMAN ] CMAN 3.0.12.1 (built
Jul 3 2014 11:37:43) started
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync CMAN membership service 2.90
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
openais checkpoint service B.01.01
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync extended virtual synchrony service
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync configuration service
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync cluster closed process group service v1.01
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync cluster config database access v1.01
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync profile loading service
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [MAIN ] Compatibility mode set
to whitetank. Using V1 and V2 of the synchronization engine.
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [QUORUM] Members[1]: 2
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [QUORUM] Members[1]: 2
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.109) ; members(old:0 left:0)
Jan 7 17:19:10 ustlvcmsp1955 corosync[4273]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:19:10 ustlvcmsp1955 rgmanager[3342]: Waiting for quorum to form
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Unloading all Corosync
service engines.
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync extended virtual synchrony service
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync configuration service
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync cluster closed process group service v1.01
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync cluster config database access v1.01
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync profile loading service
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: openais checkpoint service B.01.01
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync CMAN membership service 2.90
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [SERV ] Service engine
unloaded: corosync cluster quorum service v0.1
Jan 7 17:19:59 ustlvcmsp1955 corosync[4273]: [MAIN ] Corosync Cluster
Engine exiting with status 0 at main.c:2055.
Jan 7 17:20:00 ustlvcmsp1955 rgmanager[3342]: Quorum formed
====p1956
[root@ustlvcmsp1956 ~]# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum...
Timed-out waiting for cluster
[FAILED]
Stopping cluster:
Leaving fence domain... [ OK ]
Stopping gfs_controld... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown: [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
---logs
Jan 7 17:21:41 ustlvcmsp1956 kernel: SCTP: Hash tables configured (established
65536 bind 65536)
Jan 7 17:21:41 ustlvcmsp1956 kernel: DLM (built Dec 12 2014 16:06:44) installed
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Corosync built-in
features: nss dbus rdma snmp
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Successfully read
config from /etc/cluster/cluster.conf
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Successfully parsed
cman config
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [TOTEM ] Initializing transport
(UDP/IP Multicast).
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [TOTEM ] The network interface
[10.30.197.110] is now up.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [CMAN ] CMAN 3.0.12.1 (built
Jul 3 2014 11:37:43) started
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync CMAN membership service 2.90
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
openais checkpoint service B.01.01
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync extended virtual synchrony service
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync configuration service
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync cluster closed process group service v1.01
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync cluster config database access v1.01
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync profile loading service
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine loaded:
corosync cluster quorum service v0.1
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Compatibility mode set
to whitetank. Using V1 and V2 of the synchronization engine.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Members[1]: 3
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Members[1]: 3
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.110) ; members(old:0 left:0)
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Members[2]: 2 3
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [QUORUM] Members[2]: 2 3
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.109) ; members(old:1 left:0)
Jan 7 17:21:42 ustlvcmsp1956 corosync[4718]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:21:42 ustlvcmsp1956 rgmanager[3685]: Waiting for quorum to form
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Unloading all Corosync
service engines.
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync extended virtual synchrony service
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync configuration service
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync cluster closed process group service v1.01
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync cluster config database access v1.01
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync profile loading service
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: openais checkpoint service B.01.01
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync CMAN membership service 2.90
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [SERV ] Service engine
unloaded: corosync cluster quorum service v0.1
Jan 7 17:22:31 ustlvcmsp1956 corosync[4718]: [MAIN ] Corosync Cluster
Engine exiting with status 0 at main.c:2055.
Jan 7 17:22:32 ustlvcmsp1956 rgmanager[3685]: Quorum formed
=========p1957
[root@ustlvcmsp1957 ~]# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... Timed-out waiting for cluster
[FAILED]
Stopping cluster:
Leaving fence domain... [ OK ]
Stopping gfs_controld... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown: [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
---logs
Jan 7 17:24:20 ustlvcmsp1957 kernel: SCTP: Hash tables configured (established
65536 bind 65536)
Jan 7 17:24:20 ustlvcmsp1957 kernel: DLM (built Dec 12 2014 16:06:44) installed
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Corosync built-in
features: nss dbus rdma snmp
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Successfully read
config from /etc/cluster/cluster.conf
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Successfully parsed
cman config
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [TOTEM ] Initializing
transport (UDP/IP Multicast).
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [TOTEM ] The network interface
[10.30.197.111] is now up.
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [CMAN ] CMAN 3.0.12.1 (built
Jul 3 2014 11:37:43) started
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync CMAN membership service 2.90
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: openais checkpoint service B.01.01
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync extended virtual synchrony service
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync configuration service
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync cluster closed process group service v1.01
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync cluster config database access v1.01
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync profile loading service
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Compatibility mode
set to whitetank. Using V1 and V2 of the synchronization engine.
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [QUORUM] Members[1]: 4
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [QUORUM] Members[1]: 4
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.111) ; members(old:0 left:0)
Jan 7 17:24:21 ustlvcmsp1957 corosync[15095]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:24:53 ustlvcmsp1957 kernel: __ratelimit: 1 callbacks suppressed
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Unloading all
Corosync service engines.
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync extended virtual synchrony service
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync configuration service
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync cluster closed process group service v1.01
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync cluster config database access v1.01
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync profile loading service
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: openais checkpoint service B.01.01
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync CMAN membership service 2.90
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [SERV ] Service engine
unloaded: corosync cluster quorum service v0.1
Jan 7 17:25:10 ustlvcmsp1957 corosync[15095]: [MAIN ] Corosync Cluster
Engine exiting with status 0 at main.c:2055.
=========p1958
[root@ustlvcmsp1958 ~]# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... Timed-out waiting for cluster
[FAILED]
Stopping cluster:
Leaving fence domain... [ OK ]
Stopping gfs_controld... [ OK ]
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown: [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
--logs
Jan 7 17:26:44 ustlvcmsp1958 kernel: SCTP: Hash tables configured (established
65536 bind 65536)
Jan 7 17:26:44 ustlvcmsp1958 kernel: DLM (built Dec 12 2014 16:06:44) installed
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Corosync Cluster
Engine ('1.4.7'): started and ready to provide service.
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Corosync built-in
features: nss dbus rdma snmp
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Successfully read
config from /etc/cluster/cluster.conf
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Successfully parsed
cman config
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [TOTEM ] Initializing
transport (UDP/IP Multicast).
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [TOTEM ] The network interface
[10.30.197.112] is now up.
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [CMAN ] CMAN 3.0.12.1 (built
Jul 3 2014 11:37:43) started
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync CMAN membership service 2.90
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: openais checkpoint service B.01.01
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync extended virtual synchrony service
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync configuration service
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync cluster closed process group service v1.01
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync cluster config database access v1.01
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync profile loading service
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [QUORUM] Using quorum provider
quorum_cman
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Compatibility mode
set to whitetank. Using V1 and V2 of the synchronization engine.
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [QUORUM] Members[1]: 5
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [QUORUM] Members[1]: 5
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.112) ; members(old:0 left:0)
Jan 7 17:26:45 ustlvcmsp1958 corosync[15211]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:26:46 ustlvcmsp1958 rgmanager[14069]: Waiting for quorum to form
Jan 7 17:26:59 ustlvcmsp1958 corosync[15211]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:26:59 ustlvcmsp1958 corosync[15211]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.112) ; members(old:1 left:0)
Jan 7 17:26:59 ustlvcmsp1958 corosync[15211]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:27:12 ustlvcmsp1958 corosync[15211]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:27:12 ustlvcmsp1958 corosync[15211]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.112) ; members(old:1 left:0)
Jan 7 17:27:12 ustlvcmsp1958 corosync[15211]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:27:25 ustlvcmsp1958 corosync[15211]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:27:25 ustlvcmsp1958 corosync[15211]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.112) ; members(old:1 left:0)
Jan 7 17:27:25 ustlvcmsp1958 corosync[15211]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [CPG ] chosen downlist:
sender r(0) ip(10.30.197.112) ; members(old:1 left:0)
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [MAIN ] Completed service
synchronization, ready to provide service.
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Unloading all
Corosync service engines.
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync extended virtual synchrony service
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync configuration service
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync cluster closed process group service v1.01
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync cluster config database access v1.01
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync profile loading service
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: openais checkpoint service B.01.01
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync CMAN membership service 2.90
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [SERV ] Service engine
unloaded: corosync cluster quorum service v0.1
Jan 7 17:27:39 ustlvcmsp1958 corosync[15211]: [MAIN ] Corosync Cluster
Engine exiting with status 0 at main.c:2055.
Jan 7 17:27:39 ustlvcmsp1958 rgmanager[14069]: Quorum formed
--
Linux-cluster mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-cluster
