Sent: Wed Jan 12 2011 09:25:39 GMT-0700 (Mountain Standard Time)
From: Patrick H. <pacema...@feystorm.net>
To: pacemaker@oss.clusterlabs.org
Subject: Re: [Pacemaker] Speed up resource failover?
Sent: Wed Jan 12 2011 01:56:31 GMT-0700 (Mountain Standard Time)
From: Lars Ellenberg <lars.ellenb...@linbit.com>
To: pacemaker@oss.clusterlabs.org
Subject: Re: [Pacemaker] Speed up resource failover?
On Wed, Jan 12, 2011 at 09:30:41AM +0100, Robert van Leeuwen wrote:
-----Original message-----
To: pacemaker@oss.clusterlabs.org;
From: Patrick H. <pacema...@feystorm.net>
Sent: Wed 12-01-2011 00:06
Subject: [Pacemaker] Speed up resource failover?
Attachment: inline.txt
As it is right now, pacemaker seems to take a long time (in computer terms) to
fail over resources from one node to the other. Right now, I have 477 IPaddr2
resources evenly distributed among 2 nodes. When I put one node in standby, it
takes approximately 5 minutes to move the half of those from one node to the
other. And before you ask, theyre because of SSL http virtual hosting. I have
no order rules, colocations or anything on those resources, so it should be
able migrate the entire list simultaneously, but it seems to do them
sequentially. Is there any way to make it migrate the resources in parallel? Or
at the very least speed it up?
Patrick,
It's probably not so much the cluster suite but is has to do with the specific resource script.
For a proper takeover of a IP you have to do an arp "deregister/register".
This will take a few seconds.
This is apparently not true :-/
I have attached a portion of the lrmd log showing an example of this.
Notice that the very first line it starts the vip_55.63 resource, and
then immediately on the next line it exits successfully.
Another point of note is that somehow after the script already exited,
lrmd logs the stderr output from it. I'm not sure if its just delayed
logging or what. However, even if the script is still running, notice
that there is a huge time gap between 16:11:01 and 16:11:25 where its
just sitting there doing nothing.
I even did a series of `ps` commands to watch for the processes, and
it starts up a bunch of them, and then they all exit, and it sits
there for a long period before starting up more. So it is definitely
not the resource script slowing it down.
Also, in the log, notice that its only starting up a few scripts every
second. It should be able to fire off every single script at the exact
same time.
As long as a resource script is busy the cluster suite will not start the next
action.
Parallel execution is not possible in the cluster suite as far as I know.
(without being a programmer myself I would expect it is pretty tricky to implement
parallelization "code-wise" and making 100% sure the cluster does not break)
You could consider to edit the IPaddr2 resource script so it does not wait for
the arp commands.
At you're own risk of course ;-)
There is the cluster option "batch-limit" (in the cib), see
"configuration explained".
and there is lrmd "max-children" (can be set in some /etc/defaults/ or
/etc/sysconfig file, should be set by the init script).
you can set it manually with lrmadmin -p max-children $some_number
That should help you a bit.
But don't overdo. Raise them slowly ;-)
batch-limit it says defaults to 30 which seems like a sane value. I
tried playing with the max-children and upped it to 30 as well, but to
no effect. It does seem to be launching 30 instances of the IPaddr2
script at a time (as can be seen from the attached log), but the
problem is apparently that its sitting there for long periods of time
before starting up the next batch. I would think that when one of the
30 completes, it would launch another to take its place. But instead
it launches 30, then sits there for a while, then launches another 30.
Oh, and its not waiting for the resource to stop on the other node
before it starts it up either.
Here's the lrmd log for resource vip_55.63 from the 'ha02' node (the
node I put into standby)
Jan 12 16:10:24 ha02 lrmd: [5180]: info: rsc:vip_55.63:1444: stop
Jan 12 16:10:24 ha02 lrmd: [5180]: info: Managed vip_55.63:stop process
19063 exited with return code 0.
And here's the lrmd log for the same resource on 'ha01'
Jan 12 16:10:50 ha01 lrmd: [4707]: info: rsc:vip_55.63:1390: start
Jan 12 16:10:50 ha01 lrmd: [4707]: info: Managed vip_55.63:start process
8826 exited with return code 0.
Notice that it stopped it a full 36 seconds before it tried to start it
on the other node. The times on both boxes are in sync, so its not that
either.
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
