there are two things to keep in mind when thinking about this issue:
1) if a zk client is disconnected from the cluster, the client is
essentially in limbo. because the client cannot talk to a server it
cannot know if its session is still alive. it also cannot close its session.
2) the client only finds out about session expiration events when the
client reconnects to the cluster. if zk tells a client that its session
is expired, the ephemerals that correspond to that session will already
be cleaned up.
one of the main design points about zk is that zk only gives correct
information. if zk cannot give correct information, it basically says "i
don't know". connection loss exceptions and disconnected states are
basically "i don't know".
generally applications we design go into a "safe" mode, meaning they may
serve reads but reject changes, when disconnected from zk and only kill
themselves when they find out their session has expired.
ben
ps - session information is replicated to all zk servers, so if a leader
dies, all replicas know the sessions that are currently active and their
timeouts.
On 08/16/2010 09:03 PM, Ted Dunning wrote:
Ben or somebody else will have to repeat some of the detailed logic for
this, but it has
to do with the fact that you can't be sure what has happened during the
network partition.
One possibility is the one you describe, but another is that the partition
happened because
a majority of the ZK cluster lost power and you can't see the remaining
nodes. Those nodes
will continue to serve any files in a read-only fashion. If the partition
involves you losing
contact with the entire cluster at the same time a partition of the cluster
into a quorum and
a minority happens, then your ephemeral files could continue to exist at
least until the breach
in the cluster itself is healed.
Suffice it to say that there are only a few strategies that leave you with a
coherent picture
of the universe. Importantly, you shouldn't assume that the ephemerals will
disappear at
the same time as the session expiration event is delivered.
On Mon, Aug 16, 2010 at 8:31 PM, Qing Yan<qing...@gmail.com> wrote:
Ouch, is this the current ZK behavior? This is unexpected, if the
client get partitioned from ZK cluster, he should
get notified and take some action(e.g. commit suicide) otherwise how
to tell a ephemeral node is really
up or down? Zombie can create synchronization nightmares..
On Mon, Aug 16, 2010 at 7:22 PM, Dave Wright<wrig...@gmail.com> wrote:
Another possible cause for this that I ran into recently with the c
client -
you don't get the session expired notification until you are reconnected
to
the quorum and it informs you the session is lost. If you get
disconnected
and can't reconnect you won't get the notification. Personally I think
the
client api should track the session expiration time locally and
information
you once it's expired.
On Aug 16, 2010 2:09 AM, "Qing Yan"<qing...@gmail.com> wrote:
Hi Ted,
Do you mean GC problem can prevent delivery of SESSION EXPIRE event?
Hum...so you have met this problem before?
I didn't see any OOM though, will look into it more.
On Mon, Aug 16, 2010 at 12:46 PM, Ted Dunning<ted.dunn...@gmail.com>
wrote:
I am assuming that y...
