The easiest way to get to the bottom of this problem would be if I could
reproduce it. I could then `git bisect` between the known good and
known bad releases of pacemaker and track down the offending commit.
I could also tell you how to do that, but it's more work than I'd want
to push off onto you.
Would you be willing to send me your CIB file, after editing out any
sensitive information like passwords, internal hostnames, etc.? You can
just email it to me directly. There's no need to send it to the list
where it'll be archived.
The test is simply to watch the behavior of the cluster when the cable
is removed from the hypervisor which runs the VM with the VIP resource
started on it. When the cable is pulled, the remaining nodes behave as
expected, meaning, the resources successfully start on other nodes. The
problem I saw is the unexpected exit of the pacemaker service, which you
can see in the log file I attached.
Unfortunately, I don't have a setup here where I have enough physical
hardware. Pretty much everything I do on clusters involves VMs, which
means however I'm bringing down the network is totally different from
what you're doing. Maybe I can find someone who's got such an
environment, however.
So, my first question. Based on the log file, can anyone tell me why
the pacemaker service exiting?
Should pacemaker be exiting every time it loses quorum? I ask because
I've performed several more tests and the behavior of pacemaker service
is inconsistent. Sometimes, it doesn't exit.
I am very confused by this inconsistency. Any insight would be
appreciated.
An interesting environmental issue. When the network cable is removed
from the hypervisor, the load average, according to "top", start going
up very quickly. I mention it because it may be affecting what I see.
I've gotta say, whatever is going on in your log file is just utterly
bizarre. pacemakerd repeatedly detects that pengine (the old name for
pacemakerd-schedulerd) is unresponsive to IPC, and then tries to kill
it. Apparently that doesn't work because it tries again and again,
something like 130 times.
I also don't see what pacemaker-schedulerd could possibly be up to.
After it stops all resources (this is due to no-quorum-policy), it's
just never heard from again.
It's possible the load average has something to do with it. One thing
you could do is set
PCMK_trace_files="controld_throttle.c,controld_messages.c" in
/etc/sysconfig/pacemaker and restart. This will likely result in a lot
of logs. Maybe that will turn up something interesting.
Another thought... is there anything in the system log about OOM killing
processes, or anything like that?
- Chris
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
