There are different cases for split brain and how to test the monitor code depends on what signals you are using - but most usually, a split brain case can be created by artificially split two quorum out of a single quorum through manual configuration change (e.g. a 7 quorum servers can be split into a 4 quorum with a 3 quorum - just exclude the servers from each config).
Testing aside, internally we've been using the cardinality of QuorumPeer's ServerState and this turns out to be a pretty reliable signal for dual leader cases. For data consistency check in split brain case there is real time data consistency check built in ZooKeeper in recent release (at least for 3.6), and there are some built in metrics to signal the inconsistencies. On Thu, Jun 18, 2020 at 5:01 AM Enrico Olivelli <eolive...@gmail.com> wrote: > I told about using iptables only in order to create a network partition. > > If you want a simple split brain just switch every server in the cluster to > standalone mode (comment server.xx lines) > I saw it happening in production, it was so nasty :-( > > Enrico > > Il giorno gio 18 giu 2020 alle ore 13:50 Jordan Zimmerman < > jor...@jordanzimmerman.com> ha scritto: > > > Way back in 2013 I wrote about a scenario whereby ZooKeeper will have two > > leader nodes. I don’t know if it still applies but maybe it will be > helpful > > to you: > > > http://zookeeper-user.578899.n2.nabble.com/Rolling-config-change-considered-harmful-td7578761.html > > > > ==================== > > Jordan Zimmerman > > > > > On Jun 18, 2020, at 4:07 AM, Tim Ward <tim.w...@featurespace.co.uk> > > wrote: > > > > > > From: Szalay-Bekő Máté <szalay.beko.m...@gmail.com> > > >> Honestly, I can not really think of any way to get split-brain. If > > someone can, then let us know so that we can fix it ;) > > > > > > This issue arose because, apparently, we did get a split brain once > upon > > a time. The hints I've got however are that this happened in a pre-3.5 > > version of Zookeeper, involving a (botched?) rescaling from 3 to 7 nodes. > > From which hints I'm guessing that something got fixed in 3.5 so that > this > > approach to deliberately provoking a split brain would not work with the > > 3.5.7 we're now using. > > > > > > Tim Ward > > > > > > This message, and any files/attachments transmitted together with it, > is > > intended for the use only of the person (or persons) to whom it is > > addressed. It may contain information which is confidential and/or > > protected by legal privilege. Accordingly, any dissemination, > distribution, > > copying or use of this message, or any part of it or anything sent > together > > with it, other than by intended recipients, may constitute a breach of > > civil or criminal law and is hereby prohibited. Unless otherwise stated, > > any views expressed in this message are those of the person sending it > and > > not the sender's employer. No responsibility, legal or otherwise, of > > whatever nature, is accepted as to the accuracy of the contents of this > > message or for the completeness of the message as received. Anyone who is > > not the intended recipient of this message is advised to make no use of > it > > and is requested to contact Featurespace Limited as soon as possible. Any > > recipient of this message who has knowledge or suspects that it may have > > been the subject of unauthorised interception or alteration is also > > requested to contact Featurespace Limited. > > >
