On 23/11/2018 16:36, Jan Friesse wrote:
lejeczek,
On 15/10/2018 07:24, Jan Friesse wrote:
lejeczek,
hi guys,
I have a 3-node cluser(centos 7.5), 2 nodes seems fine but third(or
probably something else in between) is not right.
I see this:
$ pcs status --all
Cluster name: CC
Stack: corosync
Current DC: whale.private (version 1.1.18-11.el7_5.3-2b07d5c5a9) -
partition with quorum
Last updated: Fri Oct 12 15:40:39 2018
Last change: Fri Oct 12 15:14:57 2018 by root via crm_resource on
whale.private
3 nodes configured
8 resources configured (1 DISABLED)
Online: [ rental.private whale.private ]
OFFLINE: [ rider.private ]
and that third node logs:
[TOTEM ] FAILED TO RECEIVE
[TOTEM ] A new membership (10.5.6.100:2504344) was formed.
Members left: 2 4
[TOTEM ] Failed to receive the leave message. failed: 2 4
[QUORUM] Members[1]: 1
[MAIN ] Completed service synchronization, ready to provide
service.
[TOTEM ] A new membership (10.5.6.49:2504348) was formed. Members
joined: 2 4
[TOTEM ] FAILED TO RECEIVE
and it just keeps going like that.
Sometimes reboot(or stop of services + wait + start) of that third
node would help.
But, I get this situation almost every time a node gets (orderly)
shut down or reboot.
Network-wise, connectivity, seem okey. Where to start?
A little more information would be helpful (corosync version, used
protocol - udpu/udp, corosync.conf, ...), but few possible problems:
- If UDP (multicast) is used, try UDPU
- Check firewall
- Try reduce MTU used by corosync (option netmtu in corosync.conf)
Regards,
Honza
One thing I remember - could it be that because at the time of
cluster formation(and for some time after) one of the nodes had a
different ruby version from what other nodes had?
Probably not, because corosync itself does not have any dependency on
ruby.
It might have been the root cause. I do not want to jinx it but I
removed and added two nodes which seems like were in some kind of
conflict (now with save ruby on all) and seems problem is gone, so far.
But 'pcsd' does use ruby, no?
I cannot remember when this problem started to appear, was if from
the beginning or later, cannot say.
I'm on Centos 7.6. I do not think I use UDP (other then creation of
some resources and constrains it's a "vanilla" cluster). I use a
That's why I've asked for config files ;)
"non-default" MTU on the ifaces cluster uses, and also, those
interfaces are net-team devices. But still.. why it always be that
one node (all
So it's probably really MTU, please try change option netmtu in
corosync.conf.
are virtually identical)
Evil is usually hidden in detail so virtually identical may mean it's
not identical enough.
many thanks, L.
Np, but I'm not sure if hints were useful for you or not.
Regards,
Honza
many thanks, L
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
