>>> Steffen Vinther Sørensen <svint...@gmail.com> schrieb am 04.01.2021 um 16:08 in Nachricht <CALhdMBjMMHRF3ENE+=uHty7Lb9vku0o1a6+izpm3zpiM=re...@mail.gmail.com>: > Hi all, > I am trying to stabilize a 3-node CentOS7 cluster for production > usage, VirtualDomains and GFS2 resources. However this following use > case ends up with node1 fenced, and some Virtualdomains in FAILED > state. > > ------------------ > pcs standby node2 > # everything is live migrated to the other 2 nodes > > pcs stop node2 > pcs start node2 > pcs unstandby node2 > # node2 is becoming part of the cluster again, since resource > stickiness is >0 no resources are migrated at this point. > > # time of logs is 13:58:07 > pcs standby node 3 > > # node1 gets fenced after a short while > > # time of log 14:16:02 and repeats every 15 mins > node3 log ? > ------------------ > > > I looked through the logs but I got no clue what is going wrong, > hoping someone may be able to provide a hint.
Next time, also indicate which node is DC; it helps to pick the right log ;-) Quite a lot of systemd and SSH connections IMHO... At 13:58:27 node1 (kvm03-node01.avigol-gcs.dk) seems gone. Thus (I guess): Jan 04 13:59:03 kvm03-node02 stonith-ng[28902]: notice: Requesting that kvm03-node03.avigol-gcs.dk perform 'reboot' action targeting kvm03-node01.avigol-gcs.dk Jan 04 13:59:22 kvm03-node02 crmd[28906]: notice: Peer kvm03-node01.avigol-gcs.dk was terminated (reboot) by kvm03-node03.avigol-gcs.dk on behalf of stonith-api.33494: OK Maybe your network was flooded during migration of VMs?: Jan 04 13:58:33 kvm03-node03 corosync[37794]: [TOTEM ] Retransmit List: 1 2 You can limit the number of simultaneous migrations, BTW. Jan 04 13:58:33 kvm03-node03 cib[37814]: notice: Node kvm03-node01.avigol-gcs.dk state is now member Jan 04 13:58:33 kvm03-node03 cib[37814]: notice: Node kvm03-node01.avigol-gcs.dk state is now lost Jan 04 13:58:33 kvm03-node03 crmd[37819]: warning: No reason to expect node 1 to be down The above explains fencing. Regards, Ulrich > > Please find attached > > output of 'pcs config' > logs from all 3 nodes > the bzcats of pe-error-24.bz2 and pe-error-25.bz2 from node3 _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
