Hi 邓尧,
This question requires a little longer answer.
The delay that Kristoffer talks about is where you set one node to
wait a certain number of seconds before fencing the other node. In the
opposite direction, the other node does not wait to fence, so in the
case where both nodes try to fence at the same time, the one that pauses
will lose because the one that did not pause will shut down the slow
node before it's timer expires. Generally, you pick a node you prefer to
survive in this case and add 'delay="15"' to it's fence/stonith
configuration.
This isn't enough though. Let me explain why in a moment.
First, consider for example, say you want node 1 to win in these
cases. When you setup node 1's fence method, which is the instructions
the rest of the cluster uses to fence node 1, you add 'delay="15"'. For
node 2, you set no delay at all. Now something breaks the connection
between the nodes corosync communications, but not fencing (say a stray
iptables rule or similar). Both nodes see the other as lost and both
nodes immediately start to fence the peer. Node 1 gathers up the fencing
details on node 2 and immediately calls the fence. Meanwhile, node 2
gathers up the details to fence node 1, sees a 15 second delays and
pauses. Node 1 will win.
Now the problem.
Consider that your switch (or stack of switches) between the nodes
fails. Consider a fence-loop or an errant command rebooting the stack.
All channels are lost, so neither corosync traffic or fencing works.
Again, node 1 immediately calls a fence against node 2. This time
though, the fence fails, so node 1 goes into a loop trying to fence node
1. At the same time, node 2 starts a fence, pauses 15 seconds, then
calls the fence against node 1. This also fails, so it goes into a loop
as well.
Once the switches recovery, both nodes are sitting there trying to
fence the other immediately, without delay. So you're back to the
problem where both nodes could succeed in fencing the other, leaving
both nodes powered off.
This scenario is very possible when IPMI fencing is used. Normally,
if 'acpid' is running on your nodes, they will react to the IPMI BMC
"pressing the power button" by beginning a graceful shut down. The BMC
presses and holds the power button, so after four seconds, the node will
forcibly power off. However, this is more than enough time for the
slower node to begin shut down of it's peer.
So to prevent this, disable acpid.
In most systems, when acpid is _not_ running, the system will
instantly power off when it receives a power-button event. In this case,
the time between the faster node pressing the slower node's power button
and the slow node being disabled is nearly instant. *Far* less likely
that the slower node will get it's fence call out before dying.
So in short;
1. Pick a node you prefer to win, add 'delay="15"' to it's fence/stonith
configuration. Set no delay on the other node.
2. Disable acpid.
Hope this helps!
digimer
On 24/03/14 10:28 PM, 邓尧 wrote:
How much time delay difference is enough ? I'm currently setting the
delays at 5s and 15s, dual fencing does disappear, but I'm sure whether
the difference is large enough for all situations.
On Mon, Mar 24, 2014 at 8:00 PM, Kristoffer Grönlund <kgronl...@suse.com
<mailto:kgronl...@suse.com>> wrote:
On Mon, 24 Mar 2014 19:41:16 +0800
邓尧 <tors...@gmail.com <mailto:tors...@gmail.com>> wrote:
> I'm configuring a 2-node cluster on CentOS 6 with corosync +
> pacemaker + cman. Everything works well except fencing.
>
> I'm using IPMI as the fencing device, I know it's not the best
fencing
> device, but this is the only option I have.
> Manual fencing a node (pcs stonith fence <node>) works well: peer
> node goes offline, resources migrate as expected. However, if network
> traffic between the two nodes is blocked by iptables, both nodes will
> try to fence each other, and both would success, which result in both
> nodes offline.
>
> How to avoid such problem ?
You'll need to have one node take priority, by setting different
delays on the two nodes so that one of them will fence before the
other (see the "delay" parameter).
--
// Kristoffer Grönlund
// kgronl...@suse.com <mailto:kgronl...@suse.com>
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
