Hi Thomas, Here is what would happen in the scenario you mentioned. > Great - thanks Mahadev. > > Not to drag this on more than necessary, please bear with me for one > more example of 'amnesia' that comes to mind. I have a set of ZooKeeper > servers A, B, C. > - C is currently not running, A is the leader, B is the follower. > - A proposes zxid1 to A and B, both acknowledge. > - A asks A to commit (which it persists), but before the same commit > request reaches B, all servers go down (say a power failure). In this case, the zookeeper protocol says that zxid1 would be available only if the client gets a success. So zxid1 may or may not get committed if A and B come up later. ( this is a different scenario then what you mention later).
> - Later, B and C come up (A is slow to reboot), but B has lost all state > due to disk failure. This is how zookeeper would work in this scenario --- Now since we have B and C come up and B has the most recent state but loses it, then zookeeper is clueless about this. So C would say I have the some zxid say zxid-n and B would say that I have zxid = 0 (since its stateless) and C would become a leader (since it has the highest zxid). This would lead to loss of data and loss of state in zookeeper. That's what I meant when I mentioned that zookeeper relies heavily on the state being persisted on disk. > - C becomes the new leader and perhaps continues with some more new > transactions. > Now if A comes back again, C would say that its the leader and ask A to truncate all the transactions that A had to come to sync with C. Again, you can see that how persistence loss can trigger state loss in zookeeper. If its just minority of servers failing then this can be taken care of by zookeeper but in this scenario is C failing and then being brought up with an inconsisten state with another failure of A and data loss of B -- which zookeeper cannot handle. I hope this helps. mahadev On 12/16/08 4:02 PM, "Thomas Vinod Johnson" <thomas.john...@sun.com> wrote: > Mahadev Konar wrote: >> Hi Thomas, >> >> >> >> >>> More generally, is it a safe assumption to make that the ZooKeeper >>> service will maintain all its guarantees if a minority of servers lose >>> persistent state (due to bad disks, etc) and restart at some point in >>> the future? >>> >> Yes that is true. >> >> > Likely I'm misunderstanding the protocol, but have I effectively lost > zxid1 at this point? What would happen when A comes back up? > > Thanks.
