lejeczek,
On 15/10/2018 07:24, Jan Friesse wrote:
lejeczek,
hi guys,
I have a 3-node cluser(centos 7.5), 2 nodes seems fine but third(or
probably something else in between) is not right.
I see this:
$ pcs status --all
Cluster name: CC
Stack: corosync
Current DC: whale.private (version 1.1.18-11.el7_5.3-2b07d5c5a9) -
partition with quorum
Last updated: Fri Oct 12 15:40:39 2018
Last change: Fri Oct 12 15:14:57 2018 by root via crm_resource on
whale.private
3 nodes configured
8 resources configured (1 DISABLED)
Online: [ rental.private whale.private ]
OFFLINE: [ rider.private ]
and that third node logs:
[TOTEM ] FAILED TO RECEIVE
[TOTEM ] A new membership (10.5.6.100:2504344) was formed. Members
left: 2 4
[TOTEM ] Failed to receive the leave message. failed: 2 4
[QUORUM] Members[1]: 1
[MAIN ] Completed service synchronization, ready to provide service.
[TOTEM ] A new membership (10.5.6.49:2504348) was formed. Members
joined: 2 4
[TOTEM ] FAILED TO RECEIVE
and it just keeps going like that.
Sometimes reboot(or stop of services + wait + start) of that third
node would help.
But, I get this situation almost every time a node gets (orderly)
shut down or reboot.
Network-wise, connectivity, seem okey. Where to start?
A little more information would be helpful (corosync version, used
protocol - udpu/udp, corosync.conf, ...), but few possible problems:
- If UDP (multicast) is used, try UDPU
- Check firewall
- Try reduce MTU used by corosync (option netmtu in corosync.conf)
Regards,
Honza
One thing I remember - could it be that because at the time of cluster
formation(and for some time after) one of the nodes had a different ruby
version from what other nodes had?
Probably not, because corosync itself does not have any dependency on ruby.
I cannot remember when this problem started to appear, was if from the
beginning or later, cannot say.
I'm on Centos 7.6. I do not think I use UDP (other then creation of some
resources and constrains it's a "vanilla" cluster). I use a
That's why I've asked for config files ;)
"non-default" MTU on the ifaces cluster uses, and also, those interfaces
are net-team devices. But still.. why it always be that one node (all
So it's probably really MTU, please try change option netmtu in
corosync.conf.
are virtually identical)
Evil is usually hidden in detail so virtually identical may mean it's
not identical enough.
many thanks, L.
Np, but I'm not sure if hints were useful for you or not.
Regards,
Honza
many thanks, L
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
