Hi! My personal favorite to examine the cluster state is "crm_mon -1Arfj". First I'd make sure that the cluster is in a "clean state" before starting your test. "crm_resource -C" may be your friend.
Also capture the state which VM is running on which hypervisor before you pull the cable. Also be aware that your hypervisor cluster may have it's own failover strategy when you pull the cable. Maybe the better test would be to add a "blackhole route" for the other nodes on one node. That way, the hypervisors would still be able to communicate. Knowing what you are actually doing is a good starting point for testing. Apart from that if your cluster has a failover mechanism, why not skip the "cluster inside" and run a plain VM with services in that hypervisor cluster? Kind regards, Ulrich Windl > -----Original Message----- > From: Users <[email protected]> On Behalf Of Chris Lumens > via Users > Sent: Monday, November 2, 2026 10:44 PM > To: [email protected] > Cc: Chris Lumens <[email protected]> > Subject: [EXT] [EXT] Re: [ClusterLabs] Re: pacemaker&corosync services > shutting off after the network cable is disconnected > > The easiest way to get to the bottom of this problem would be if I could > reproduce it. I could then `git bisect` between the known good and > known bad releases of pacemaker and track down the offending commit. > > I could also tell you how to do that, but it's more work than I'd want > to push off onto you. > > Would you be willing to send me your CIB file, after editing out any > sensitive information like passwords, internal hostnames, etc.? You can > just email it to me directly. There's no need to send it to the list > where it'll be archived. > > > The test is simply to watch the behavior of the cluster when the cable > > is removed from the hypervisor which runs the VM with the VIP resource > > started on it. When the cable is pulled, the remaining nodes behave as > > expected, meaning, the resources successfully start on other nodes. The > > problem I saw is the unexpected exit of the pacemaker service, which you > > can see in the log file I attached. > > Unfortunately, I don't have a setup here where I have enough physical > hardware. Pretty much everything I do on clusters involves VMs, which > means however I'm bringing down the network is totally different from > what you're doing. Maybe I can find someone who's got such an > environment, however. > > > So, my first question. Based on the log file, can anyone tell me why > > the pacemaker service exiting? > > > > Should pacemaker be exiting every time it loses quorum? I ask because > > I've performed several more tests and the behavior of pacemaker service > > is inconsistent. Sometimes, it doesn't exit. > > > > I am very confused by this inconsistency. Any insight would be > > appreciated. > > > > An interesting environmental issue. When the network cable is removed > > from the hypervisor, the load average, according to "top", start going > > up very quickly. I mention it because it may be affecting what I see. > I've gotta say, whatever is going on in your log file is just utterly > bizarre. pacemakerd repeatedly detects that pengine (the old name for > pacemakerd-schedulerd) is unresponsive to IPC, and then tries to kill > it. Apparently that doesn't work because it tries again and again, > something like 130 times. > > I also don't see what pacemaker-schedulerd could possibly be up to. > After it stops all resources (this is due to no-quorum-policy), it's > just never heard from again. > > It's possible the load average has something to do with it. One thing > you could do is set > PCMK_trace_files="controld_throttle.c,controld_messages.c" in > /etc/sysconfig/pacemaker and restart. This will likely result in a lot > of logs. Maybe that will turn up something interesting. > > Another thought... is there anything in the system log about OOM killing > processes, or anything like that? > > - Chris > > _______________________________________________ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
