Hi all,
I am currently working on a consensus protocol for Apache Kafka and I
am trying to understand Apache ZooKeeper's behaviour when the majority
of the quorum loses their transaction and snapshot directory
("version-2"). I ran the following experiment with
apache-zookeeper-3.8.0:
A) On disk I created the following 3 configurations:
$ cat zk1/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/jsancio/work/apache-zookeeper-3.8.0-bin/zk1/data
clientPort=2181
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
$ cat zk1/data/myid
1
$ cat zk2/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/jsancio/work/apache-zookeeper-3.8.0-bin/zk2/data
clientPort=2182
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
$ cat zk2/data/myid
2
$ cat zk3/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/jsancio/work/apache-zookeeper-3.8.0-bin/zk3/data
clientPort=2183
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
$ cat zk3/data/myid
3
B) I started the majority of the nodes (1, 2). The ensemble was
established and I was able to create a znode using the CLI.
C) I shutdown all of the nodes (1, 2 since I never started node 3). To
simulate a disk failure I deleted the content of the transaction and
snapshot directory (version-2) for node 2.
D) I started the majority of the nodes (2, 3). The ensemble was
established and I was able to establish a connection with the CLI.
E) I finally started node 1 which had the committed transactions and
snapshots. The znode created in step B) was not present.
Based on your understanding of Apache ZooKeeper is the expected
behavior? Or did I configure and bootstrap the cluster incorrectly?
For example, another possible outcome is that the ensemble would not
be established in step D.
Any feedback helping me understand this is greatly appreciated!
--
-José
