On Wed, Jan 13, 2010 at 8:19 AM, Miki Shapiro <miki.shap...@coles.com.au>wrote:
> Separate to my earlier post re CRM DC election in a 2-way cluster, I’m > chasing up the (separate) issue of making the cluster a CROSS-SITE one. > > > > As stated in yay other thread, I’m running a 2-way quorum-agnostic cluster > on a SLES11, openais, pacemaker, drbd (… clvm, ocfs2, ctdb, nfs, etc) on HP > Blades. > > > > A few old threads (with a rather elaborate answer from Lars) indicate that > as of March 2009 split-site wasn’t yet thoroughly supported as WAN > connectivity issues were not thoroughly addressed, and that as of then > quorumd was not yet sufficiently robust/tested/PROD-ready. > > > > What we decided we want to do is rely on an extremely simple (and hopefully > by inference predictable and reliable) arbitrator - a THIRD linux server > that lives at a SEPARATE THIRD site altogether with no special HA-related > daemons running on it. > > > > I’ll build a STONITH ocf script, configure it as a cloned STONITH resource > running on both nodes, and it will do roughly this when pinging the other > node (via either one or two redundant links) will fail: > > > > ssh arbitrator mkdir /tmp/$clustername && shoot-other-node || > hard-suicide-NOW > > > > Thus, when split, the nodes race to the arbitrator. > > First to run the mkdir command on the arbitrator (and get rc=0) wins, gets > the long straw and lives. Loser gets shot (either by its peer if WAN allows > peer to communicate with soon-to-be-dead node’s iLO or by said node sealing > its own fate). > > > > Primary failure modes not accounted for by a run-of-the-mill non-split-site > cluster are thus: > > > > 1. One node cut off – cutoff node will fail the race and suicide. > Good node will succeed and proceed to provide service. > > 2. Nodes cut off from each other but can both access the arbitrator > – slower node will suicide. Faster node will succeed and proceed to provide > service. > > 3. Both nodes are cut off, or the comms issue affects both > node1<->node2 comms AND all ->arbitrator comms (double failure). – both > nodes suicide (and potentially leave me with two inconsistent and > potentially corrupt filesystems). Can’t see an easy way around this one (can > anyone?) > Basically thats the part that the stuff we haven't written yet is supposed to address. You want to avoid the "|| hard-suicide-NOW" part of your logic, but you can't safely do that unless there is some way to stop the services on the non-connected node(s) - preferably _really_ quickly. What about setting no-quorum-policy to freeze and making the third node a full cluster member (that just doesn't run any resources)? That way, if you get a 1-1-1 split the nodes will leave all services running where they were and while it waits for quorum. And if it heals into a 1-2 split, then the majority will terminate the rogue node and acquire all the services. The biggest problem is the reliability of your links and stonith devices - give particular thought to how you'd fence _node_ A if comms to _site_ A are down.... > > > > > Looks to me like this can easily be implemented without any fancy quorum > servers (on top of the required little ocf script and the existence of the > arbitrator) > > > > Does anyone have thoughts on this? Am I ignoring any major issues, or > reinventing the wheel, or should this this potentially work as I think it > will? > > > > Thanks! J > > > > And a little addendum which just occurred to me re transient WAN network > issues: > > > > 1. Transient big (>2min) network issues will land me with a cluster > that needs a human to turn on one node on every time they happen. Bad. > > > > My proposed solution: classify a peer-failure as a WAN-problem by pinging > peer node’s core router when peer node appears dead, if router dead too > touch a WAN-problem-flagfile, and so long as the flag-file sits there the > survivor pings (done via ocf ping resource) other-side-router until it comes > online, then shooting a “check-power-status && > O-GOD-IT-STILL-LIVES-KILL-IT-NOW || power-it-on” command to the peer’s iLO > (and promptly delete the flag). > > > > Implementation cost: a wee bit of scripting and a wee bit of pacemaker > configuration. > > > > 2. Transient small network issues will require stretching > pacemaker’s default timeouts sufficiently to avoid (or end up in the item 1 > bucket above) > > > > Am very keen to know what the gurus think J J > > * * > > *Miki Shapiro* > > Linux Systems Engineer > Infrastructure Services & Operations > > > 745 Springvale Road > Mulgrave 3170 Australia > Email miki.shap...@coles.com.au > Phone: 61 3 854 10520 > > Fax: 61 3 854 10558 > > > > ______________________________________________________________________ > This email and any attachments may contain privileged and confidential > information and are intended for the named addressee only. If you have > received this e-mail in error, please notify the sender and delete > this e-mail immediately. Any confidentiality, privilege or copyright > is not waived or lost because this e-mail has been sent to you in > error. It is your responsibility to check this e-mail and any > attachments for viruses. No warranty is made that this material is > free from computer virus or any other defect or error. Any > loss/damage incurred by using this material is not the sender's > responsibility. The sender's entire liability will be limited to > resupplying the material. > ______________________________________________________________________ > > _______________________________________________ > Pacemaker mailing list > Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > >
<<image001.png>>
<<image002.png>>
_______________________________________________ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker