Re: are ephemeral nodes removed when client receives session expiration
Sorry, I was wrong on the session not checked in the learner and mark the
session closing only run on the leader part. However, it looks like the
session on the learner (the second server) will not expire as it was a reopened
session and as far as I see the session time out is partly set by the client
(and adjusted by the server). So the session on the new server looks valid
(until the learner gets the closeSession request). Any ideas? Maybe I can try
to add a test?
> On Feb 8, 2017, at 11:24 PM, Ryan Zhang wrote:
>
> I am sorry that I still didn’t get it. Let me post my scenarios again
>
> 1. Client A connected to Leader with session S
> 2. Session S expired on Leader, which according to the code below it will
> set the session as closing
> 3. Leader then send out a close session request to its own first processor
> which will go through the usual pipeline as you mentioned
> 4. Client A send a request which will go through the checkSession code I
> pasted below and it should get a SessionExpiredException.
> 5. Client now knows its session expired but for some reason it tries to
> connect to another server B and issues a read.
>
> Now this is a race between the following two chain of operations
>
> 1. The closesession request needs to go through quorum and get the majority
> and server B get the commit (or inform) request and actually kill the session
> in the final request processor
>
> 2. The read operation that goes through learnerZK process chain in memory (as
> far as I see, there is no session check on read at all so it won’t know until
> it reads the tree)
>
> The first chain can get stuck on a various places like some quorum nodes are
> stuck on some other proposals so it looks to me that the second chain is very
> much likely to win as it does not need to go through any quorum operation.
> The key issue is the client needs to reconnect fast (or in parallel as the
> original post seems to indicate). I think I can definitely simulate this with
> a test but it will be tricky to make it pass/fail deterministically so I
> didn’t try.
>
>
> you seems to agree that my step 5 will happen if the client connect with a
> valid session ID so there won’t be any quorum operation before establishing
> the session. So the race after that can happen, why not? Also, there is no
> session check for read operation on the learners as far as I can see (please
> point me to the code if you find it). Furthermore, the mark the session
> closing code I posted only run on the lead as far as I can see (again, please
> point me to the code)
>
> Just to repeat, the race is between the learner gets the quorum closeSession
> and the client issue a read. No?
>
>
> On Feb 8, 2017, at 9:03 PM, Michael Han
> mailto:h...@cloudera.com>> wrote:
>
> Hi Ryan,
>
>>> From the code I saw, there is no quorum operation when a ZK reconnect with
>>> a session ID
>
> The "createSession(cnxn, passwd, sessionTimeout)" will submit a session
> create request that will go through processor pipeline, and that is where the
> quorum operation is involved. Also note that while we are creating a new
> session for client, we disabled server socket connection appertains to this
> client, so server would not process any requests from the client, until the
> session is finalized (we then enable the same socket connection).
>
> There is another case here - covered by the "else" clause, which is when the
> client request embeds a valid session ID. If at this point in time, ZooKeeper
> has already started the session clean up for this particular session ID, then
> the session will be marked as closing. As you noticed previously, ZK
> validates session for every operation (even the read ones), and a session in
> the state of "closing" will not pass validation. On client side it will
> receive SessionExpire event (as a result of failed session validation) for
> the read requests it previously issued on ephemeral nodes, so our invariant
> still holds.
>
> Hope these help.
>
>
> On Wed, Feb 8, 2017 at 11:55 AM, Ryan Zhang
> mailto:yangzhangr...@hotmail.com>> wrote:
> Hi, Michael, I am sorry that I spoke too soon. From the code I saw, there
> is no quorum operation when a ZK reconnect with a session ID (pasted the code
> below). Did I miss something again?
>
>
> long sessionId = connReq.getSessionId();
> if (sessionId == 0) {
>LOG.info("Client attempting to establish new session at "
>+ cnxn.getRemoteSocketAddress());
>createSession(cnxn, passwd, sessionTimeout);
> } else {
>long clientSessionId = connReq.getSessionId();
>LOG.info(&qu
Re: are ephemeral nodes removed when client receives session expiration
I am sorry that I still didn’t get it. Let me post my scenarios again
1. Client A connected to Leader with session S
2. Session S expired on Leader, which according to the code below it will set
the session as closing
3. Leader then send out a close session request to its own first processor
which will go through the usual pipeline as you mentioned
4. Client A send a request which will go through the checkSession code I pasted
below and it should get a SessionExpiredException.
5. Client now knows its session expired but for some reason it tries to connect
to another server B and issues a read.
Now this is a race between the following two chain of operations
1. The closesession request needs to go through quorum and get the majority and
server B get the commit (or inform) request and actually kill the session in
the final request processor
2. The read operation that goes through learnerZK process chain in memory (as
far as I see, there is no session check on read at all so it won’t know until
it reads the tree)
The first chain can get stuck on a various places like some quorum nodes are
stuck on some other proposals so it looks to me that the second chain is very
much likely to win as it does not need to go through any quorum operation. The
key issue is the client needs to reconnect fast (or in parallel as the original
post seems to indicate). I think I can definitely simulate this with a test but
it will be tricky to make it pass/fail deterministically so I didn’t try.
you seems to agree that my step 5 will happen if the client connect with a
valid session ID so there won’t be any quorum operation before establishing the
session. So the race after that can happen, why not? Also, there is no session
check for read operation on the learners as far as I can see (please point me
to the code if you find it). Furthermore, the mark the session closing code I
posted only run on the lead as far as I can see (again, please point me to the
code)
Just to repeat, the race is between the learner gets the quorum closeSession
and the client issue a read. No?
On Feb 8, 2017, at 9:03 PM, Michael Han
mailto:h...@cloudera.com>> wrote:
Hi Ryan,
>> From the code I saw, there is no quorum operation when a ZK reconnect with
>> a session ID
The "createSession(cnxn, passwd, sessionTimeout)" will submit a session create
request that will go through processor pipeline, and that is where the quorum
operation is involved. Also note that while we are creating a new session for
client, we disabled server socket connection appertains to this client, so
server would not process any requests from the client, until the session is
finalized (we then enable the same socket connection).
There is another case here - covered by the "else" clause, which is when the
client request embeds a valid session ID. If at this point in time, ZooKeeper
has already started the session clean up for this particular session ID, then
the session will be marked as closing. As you noticed previously, ZK validates
session for every operation (even the read ones), and a session in the state of
"closing" will not pass validation. On client side it will receive
SessionExpire event (as a result of failed session validation) for the read
requests it previously issued on ephemeral nodes, so our invariant still holds.
Hope these help.
On Wed, Feb 8, 2017 at 11:55 AM, Ryan Zhang
mailto:yangzhangr...@hotmail.com>> wrote:
Hi, Michael, I am sorry that I spoke too soon. From the code I saw, there is
no quorum operation when a ZK reconnect with a session ID (pasted the code
below). Did I miss something again?
long sessionId = connReq.getSessionId();
if (sessionId == 0) {
LOG.info("Client attempting to establish new session at "
+ cnxn.getRemoteSocketAddress());
createSession(cnxn, passwd, sessionTimeout);
} else {
long clientSessionId = connReq.getSessionId();
LOG.info("Client attempting to renew session 0x"
+ Long.toHexString(clientSessionId)
+ " at " + cnxn.getRemoteSocketAddress());
if (serverCnxnFactory != null) {
serverCnxnFactory.closeSession(sessionId);
}
if (secureServerCnxnFactory != null) {
secureServerCnxnFactory.closeSession(sessionId);
}
cnxn.setSessionId(sessionId);
reopenSession(cnxn, sessionId, passwd, sessionTimeout);
}
On Feb 7, 2017, at 3:46 PM, Ryan Zhang
mailto:yangzhangr...@hotmail.com>> wrote:
Thank you very much Michael, I am now clear.
On Feb 7, 2017, at 3:21 PM, Michael Han
mailto:h...@cloudera.com><mailto:h...@cloudera.com>> wrote:
Hi Ryan,
Thanks for the elaborations. I see why you are confused. Your case is valid,
except in step 5:
Client now knows its session expired but for some reason it tries to connect to
another server B and issues a read.
At this point of time, there
Re: are ephemeral nodes removed when client receives session expiration
Hi, Michael, I am sorry that I spoke too soon. From the code I saw, there is
no quorum operation when a ZK reconnect with a session ID (pasted the code
below). Did I miss something again?
long sessionId = connReq.getSessionId();
if (sessionId == 0) {
LOG.info("Client attempting to establish new session at "
+ cnxn.getRemoteSocketAddress());
createSession(cnxn, passwd, sessionTimeout);
} else {
long clientSessionId = connReq.getSessionId();
LOG.info("Client attempting to renew session 0x"
+ Long.toHexString(clientSessionId)
+ " at " + cnxn.getRemoteSocketAddress());
if (serverCnxnFactory != null) {
serverCnxnFactory.closeSession(sessionId);
}
if (secureServerCnxnFactory != null) {
secureServerCnxnFactory.closeSession(sessionId);
}
cnxn.setSessionId(sessionId);
reopenSession(cnxn, sessionId, passwd, sessionTimeout);
}
On Feb 7, 2017, at 3:46 PM, Ryan Zhang
mailto:yangzhangr...@hotmail.com>> wrote:
Thank you very much Michael, I am now clear.
On Feb 7, 2017, at 3:21 PM, Michael Han
mailto:h...@cloudera.com><mailto:h...@cloudera.com>> wrote:
Hi Ryan,
Thanks for the elaborations. I see why you are confused. Your case is valid,
except in step 5:
Client now knows its session expired but for some reason it tries to connect to
another server B and issues a read.
At this point of time, there are two possible cases:
* Client A's session has been cleaned up. As part of the session clean up, the
ephemeral nodes appertain with A will be gone. After A reconnect to the quorum,
it will not see the old ephemeral nodes.
* Client A's session has not finished cleaned up. In this case, if client A is
trying to issue a read, it has to first establish a new session with quorum.
Because the session clean up transaction of A is already in flight, and because
of the sequential consistency guarantee provided by ZooKeeper, client A will
not be able to reconnect to the quorum / establish the session, until the
previous session up transaction has finished. So we will be back to first case
here, which also guarantee the old ephemeral nodes are gone.
On Tue, Feb 7, 2017 at 2:02 PM, Ryan Zhang
mailto:yangzhangr...@hotmail.com><mailto:yangzhangr...@hotmail.com>>
wrote:
Hi, Michael,
Thank you for your clarification. What you said seems to be consistent with
what I saw in the code. However, I am still confused by your conclusion. My
conclusion from the code is that it is totally possible for a client to get
session expired and reconnect to another sever to still see the ephemeral node.
The exact reason is caused by the code I pasted below. I am not sure if we are
on the same page as you seem to suggest that it is not possible. Let me
elaborate a bit on how this can happen.
1. Client A connected to Leader with session S
2. Session S expired on Leader, which according to the code below it will set
the session as closing
3. Leader then send out a close session request to its own first processor
which will go through the usual pipeline as you mentioned
4. Client A send a request which will go through the checkSession code I pasted
below and it should get a SessionExpiredException.
5. Client now knows its session expired but for some reason it tries to connect
to another server B and issues a read.
Now this is a race between the following two chain of operations
1. The closesession request needs to go through quorum and get the majority and
server B get the commit (or inform) request and actually kill the session in
the final request processor
2. The read operation that goes through learnerZK process chain in memory (as
far as I see, there is no session check on read at all so it won’t know until
it reads the tree)
The first chain can get stuck on a various places like some quorum nodes are
stuck on some other proposals so it looks to me that the second chain is very
much likely to win as it does not need to go through any quorum operation. The
key issue is the client needs to reconnect fast (or in parallel as the original
post seems to indicate). I think I can definitely simulate this with a test but
it will be tricky to make it pass/fail deterministically so I didn’t try.
Am I missing something?
-Ryan
On Feb 7, 2017, at 1:24 PM, Michael Han
mailto:h...@cloudera.com><mailto:h...@cloudera.com>> wrote:
Hi Ryan,
I am not sure what you were confused about regarding session cleaning up code.
Here is my understanding, hope it helps.
* Session clean up is started from marking the state of a session as closed, as
you noticed. This is because each session clean up will take a while so we need
make sure that during session clean up, server will not continue processing the
requests from the client that appertain with this session.
* Once session is marked as closing, we will send a request so the closing of
t
Re: are ephemeral nodes removed when client receives session expiration
Thank you very much Michael, I am now clear. On Feb 7, 2017, at 3:21 PM, Michael Han mailto:h...@cloudera.com>> wrote: Hi Ryan, Thanks for the elaborations. I see why you are confused. Your case is valid, except in step 5: >> Client now knows its session expired but for some reason it tries to >> connect to another server B and issues a read. At this point of time, there are two possible cases: * Client A's session has been cleaned up. As part of the session clean up, the ephemeral nodes appertain with A will be gone. After A reconnect to the quorum, it will not see the old ephemeral nodes. * Client A's session has not finished cleaned up. In this case, if client A is trying to issue a read, it has to first establish a new session with quorum. Because the session clean up transaction of A is already in flight, and because of the sequential consistency guarantee provided by ZooKeeper, client A will not be able to reconnect to the quorum / establish the session, until the previous session up transaction has finished. So we will be back to first case here, which also guarantee the old ephemeral nodes are gone. On Tue, Feb 7, 2017 at 2:02 PM, Ryan Zhang mailto:yangzhangr...@hotmail.com>> wrote: Hi, Michael, Thank you for your clarification. What you said seems to be consistent with what I saw in the code. However, I am still confused by your conclusion. My conclusion from the code is that it is totally possible for a client to get session expired and reconnect to another sever to still see the ephemeral node. The exact reason is caused by the code I pasted below. I am not sure if we are on the same page as you seem to suggest that it is not possible. Let me elaborate a bit on how this can happen. 1. Client A connected to Leader with session S 2. Session S expired on Leader, which according to the code below it will set the session as closing 3. Leader then send out a close session request to its own first processor which will go through the usual pipeline as you mentioned 4. Client A send a request which will go through the checkSession code I pasted below and it should get a SessionExpiredException. 5. Client now knows its session expired but for some reason it tries to connect to another server B and issues a read. Now this is a race between the following two chain of operations 1. The closesession request needs to go through quorum and get the majority and server B get the commit (or inform) request and actually kill the session in the final request processor 2. The read operation that goes through learnerZK process chain in memory (as far as I see, there is no session check on read at all so it won’t know until it reads the tree) The first chain can get stuck on a various places like some quorum nodes are stuck on some other proposals so it looks to me that the second chain is very much likely to win as it does not need to go through any quorum operation. The key issue is the client needs to reconnect fast (or in parallel as the original post seems to indicate). I think I can definitely simulate this with a test but it will be tricky to make it pass/fail deterministically so I didn’t try. Am I missing something? -Ryan On Feb 7, 2017, at 1:24 PM, Michael Han mailto:h...@cloudera.com>> wrote: Hi Ryan, I am not sure what you were confused about regarding session cleaning up code. Here is my understanding, hope it helps. * Session clean up is started from marking the state of a session as closed, as you noticed. This is because each session clean up will take a while so we need make sure that during session clean up, server will not continue processing the requests from the client that appertain with this session. * Once session is marked as closing, we will send a request so the closing of the session not only applies to the leader but also on the quorum servers. This is just like all other requests that will go through the normal requests process pipeline. * A valid session is a prerequisite for any of the client operations (including read operations), so the liveness of the session is validated before processing a read operation. On Wed, Jan 25, 2017 at 2:25 PM, Ryan Zhang mailto:yangzhangr...@hotmail.com>> wrote: I am a bit confused by the code On Jan 25, 2017, at 1:33 PM, Michael Han mailto:h...@cloudera.com><mailto:h...@cloudera.com<mailto:h...@cloudera.com>>> wrote: Does ZK guarantee that ephemeral nodes from a client are removed on the sever by the time the client receives a session expiration event? "the server" is a vague definition, as ZooKeeper ensemble is composed of multiple servers :). Therefore, it seems to be possible for a client to connect to another server to see the node there. This seems the only case I can think of that lead to the inconsistent view from client side. I'll elaborate as follows, first the guarantees of ZooKeeper that&
Re: are ephemeral nodes removed when client receives session expiration
Hi, Michael,
Thank you for your clarification. What you said seems to be consistent with
what I saw in the code. However, I am still confused by your conclusion. My
conclusion from the code is that it is totally possible for a client to get
session expired and reconnect to another sever to still see the ephemeral node.
The exact reason is caused by the code I pasted below. I am not sure if we are
on the same page as you seem to suggest that it is not possible. Let me
elaborate a bit on how this can happen.
1. Client A connected to Leader with session S
2. Session S expired on Leader, which according to the code below it will set
the session as closing
3. Leader then send out a close session request to its own first processor
which will go through the usual pipeline as you mentioned
4. Client A send a request which will go through the checkSession code I pasted
below and it should get a SessionExpiredException.
5. Client now knows its session expired but for some reason it tries to connect
to another server B and issues a read.
Now this is a race between the following two chain of operations
1. The closesession request needs to go through quorum and get the majority and
server B get the commit (or inform) request and actually kill the session in
the final request processor
2. The read operation that goes through learnerZK process chain in memory (as
far as I see, there is no session check on read at all so it won’t know until
it reads the tree)
The first chain can get stuck on a various places like some quorum nodes are
stuck on some other proposals so it looks to me that the second chain is very
much likely to win as it does not need to go through any quorum operation. The
key issue is the client needs to reconnect fast (or in parallel as the original
post seems to indicate). I think I can definitely simulate this with a test but
it will be tricky to make it pass/fail deterministically so I didn’t try.
Am I missing something?
-Ryan
On Feb 7, 2017, at 1:24 PM, Michael Han
mailto:h...@cloudera.com>> wrote:
Hi Ryan,
I am not sure what you were confused about regarding session cleaning up code.
Here is my understanding, hope it helps.
* Session clean up is started from marking the state of a session as closed, as
you noticed. This is because each session clean up will take a while so we need
make sure that during session clean up, server will not continue processing the
requests from the client that appertain with this session.
* Once session is marked as closing, we will send a request so the closing of
the session not only applies to the leader but also on the quorum servers. This
is just like all other requests that will go through the normal requests
process pipeline.
* A valid session is a prerequisite for any of the client operations (including
read operations), so the liveness of the session is validated before processing
a read operation.
On Wed, Jan 25, 2017 at 2:25 PM, Ryan Zhang
mailto:yangzhangr...@hotmail.com>> wrote:
I am a bit confused by the code
On Jan 25, 2017, at 1:33 PM, Michael Han
mailto:h...@cloudera.com><mailto:h...@cloudera.com<mailto:h...@cloudera.com>>>
wrote:
Does ZK guarantee that ephemeral nodes from a client are removed on the
sever by the time the client receives a session expiration event?
"the server" is a vague definition, as ZooKeeper ensemble is composed of
multiple servers :).
Therefore, it seems to be possible for a client to connect to another
server to see the node there.
This seems the only case I can think of that lead to the inconsistent view
from client side. I'll elaborate as follows, first the guarantees of
ZooKeeper that's relevant to this case:
* ZooKeeper quorum should have already committed the transaction of closing
the session when a client receives the session expire event.
Here are the code that throws KeeperException.SessionExpiredException
public synchronized void checkSession(long sessionId, Object owner)
throws KeeperException.SessionExpiredException,
KeeperException.SessionMovedException,
KeeperException.UnknownSessionException {
……….
if (session.isClosing()) {
throw new KeeperException.SessionExpiredException();
}
}
Here is the code to set it to be closing directly
synchronized public void setSessionClosing(long sessionId) {
if (LOG.isTraceEnabled()) {
LOG.trace("Session closing: 0x" + Long.toHexString(sessionId));
}
SessionImpl s = sessionsById.get(sessionId);
if (s == null) {
return;
}
s.isClosing = true;
}
and here is the code that call the above
public void runImpl() throws InterruptedException {
while (running) {
……...
for (SessionImpl s : sessionExpiryQueue.poll()) {
setSessionClosing(s.sessionId);
expirer.expire(s);
}
}
}
the expire function looks like this
public void expire(
Re: are ephemeral nodes removed when client receives session expiration
I am a bit confused by the code
On Jan 25, 2017, at 1:33 PM, Michael Han
mailto:h...@cloudera.com>> wrote:
Does ZK guarantee that ephemeral nodes from a client are removed on the
sever by the time the client receives a session expiration event?
"the server" is a vague definition, as ZooKeeper ensemble is composed of
multiple servers :).
Therefore, it seems to be possible for a client to connect to another
server to see the node there.
This seems the only case I can think of that lead to the inconsistent view
from client side. I'll elaborate as follows, first the guarantees of
ZooKeeper that's relevant to this case:
* ZooKeeper quorum should have already committed the transaction of closing
the session when a client receives the session expire event.
Here are the code that throws KeeperException.SessionExpiredException
public synchronized void checkSession(long sessionId, Object owner)
throws KeeperException.SessionExpiredException,
KeeperException.SessionMovedException,
KeeperException.UnknownSessionException {
……….
if (session.isClosing()) {
throw new KeeperException.SessionExpiredException();
}
}
Here is the code to set it to be closing directly
synchronized public void setSessionClosing(long sessionId) {
if (LOG.isTraceEnabled()) {
LOG.trace("Session closing: 0x" + Long.toHexString(sessionId));
}
SessionImpl s = sessionsById.get(sessionId);
if (s == null) {
return;
}
s.isClosing = true;
}
and here is the code that call the above
public void runImpl() throws InterruptedException {
while (running) {
……...
for (SessionImpl s : sessionExpiryQueue.poll()) {
setSessionClosing(s.sessionId);
expirer.expire(s);
}
}
}
the expire function looks like this
public void expire(Session session) {
long sessionId = session.getSessionId();
close(sessionId);
}
and close function here
private void close(long sessionId) {
Request si = new Request(null, sessionId, 0, OpCode.closeSession, null,
null);
setLocalSessionFlag(si);
submitRequest(si);
}
so it looks to me that the session is marked as closing first and then the
closeSession is send. This will happen on
the lead only though as the checkSession is only called on the lead but it is
called even on read operation.
May I missing something?
* Clean up of ephemeral nodes associated with the session is part of the
closing session transaction, so for the quorum of servers who have already
committed the transaction, the ephemeral nodes should have gone already, on
those servers.
* ZooKeeper quorum would not have processed the new session establishment
request for the same client, until after the closing session request has
been processed because transactions are ordered across quorum.
Given these guarantees, if a client reestablishes a new session via
connecting to a server which was the quorum of servers that committed the
closing session transaction, then the client should not see the old
ephemeral node upon new session established.
ZooKeeper does not guarantee a write transaction occur synchronously across
all of the servers, since a write request only requires a quorum of servers
to acknowledge. As a result, it is valid that some servers might lag behind
the state of the quorum. I suspect this case is possible:
* Client receives session expire event, and client close its connection to
server A.
* Client reconnects to server B, which lags behind quorum, that does not
contain the changes to the data tree regarding ephemeral nodes.
* Client sees the ephemeral node so it does nothing. Later the node is
cleaned up when server B sync with quorum.
Client can ensure it always see the state of truth of the quorum by issuing
a sync() request before issuing a read request. A sync request will force
the server it's connecting to sync with the quorum. If Kafka does this,
will the bug go away? Of course, retry creating ephemeral nodes can also
solve the problem (there are possible other solutions as well, by having
client to do some book keeping work to differentiate versions between
ephemeral nodes).
On Wed, Jan 25, 2017 at 11:32 AM, Ryan Zhang
mailto:yangzhangr...@hotmail.com>>
wrote:
Good question, AFAIK, it’s not the case.
The server will throw an SessionExpiredException during checkSession call
as soon as the session is marked as isClosing. However, session expiration
actually requires a transaction (of type OpCode.closeSession) which will be
send to the leader to go through the quorum. The session and ephemeral
node will only be removed after the transaction is committed and processed
in the final processor on other nodes. Therefore, it seems to be possible
for a client to connect to another server to see the node there. I am not
entirely sure if it can use the same session id though, it seems possible
as the session close is only ba
Re: are ephemeral nodes removed when client receives session expiration
Good question, AFAIK, it’s not the case. The server will throw an SessionExpiredException during checkSession call as soon as the session is marked as isClosing. However, session expiration actually requires a transaction (of type OpCode.closeSession) which will be send to the leader to go through the quorum. The session and ephemeral node will only be removed after the transaction is committed and processed in the final processor on other nodes. Therefore, it seems to be possible for a client to connect to another server to see the node there. I am not entirely sure if it can use the same session id though, it seems possible as the session close is only based on the session expire time and there can be delays in session pings. On Jan 25, 2017, at 8:53 AM, Jun Rao mailto:jun...@gmail.com>> wrote: Hi, Does ZK guarantee that ephemeral nodes from a client are removed on the sever by the time the client receives a session expiration event? I am getting conflicting info on this ( https://issues.apache.org/jira/browse/KAFKA-4277). Could someone clarify? Thanks, Jun
Re: Zookeeper leader election takes a long time.
From that log, it looks like Node 3 and 2 formed a quorum (Node 1 can’t talk to Node2 for some reason). Even if Node 3 didn’t hear from Node 2 after 12:46, it is still a special case as we had a kinda network partition for about 10 mins. I wonder if you can repro this in an environment that Node 2 is definitely down? I will take a look at the log after 12:46 when I get a chance. > On Oct 14, 2016, at 1:40 PM, Anand Parthasarathy > wrote: > > Hi Michael and Ryan, > > Thanks for looking into the logs. > > Actually, in this case, node 2 was brought down ungracefully and was down > permanently. In this customer deployment, our product runs as a docker > container in a bare metal environment. It is possible that the docker proxy > that provides external connectivity to the zookeeper ports is down but the > container itself is up and running for several more minutes. If you grep > for "2 (n.sid)" you will see that there is no packets after 12:46. > Convergence after that did not happen until 15:57 or so. > > Thanks, > Anand. > > On Thu, Oct 13, 2016 at 5:10 PM, Michael Han wrote: > >>>> it started a new round but then I seem to see the election messages >> from Node 2 again. Any idea why? >> >> My guess is node 2 is back online and ZK service was started. In this use >> case node 2 does not get offline permanently, IIUC. >> >> >> On Thu, Oct 13, 2016 at 3:41 PM, Ryan Zhang >> wrote: >> >>> Hi, Anand, I took a look and I wonder how do you explain the following >>> >>> The N1.log starts at around >>> 2016-10-03 12:37:38,073 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1: >>> 5002:QuorumPeer@714] - LOOKING >>> >>> and it failed to talk to Node 2 (id: 2) >>> 2016-10-03 12:37:38,136 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCn >>> xManager@382] - Cannot open channel to 2 at election address >>> node2.controller.local/127.0.0.8:5004 >>> java.net.ConnectException: Connection refused >>>at java.net.PlainSocketImpl.socketConnect(Native Method) >>>at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock >>> etImpl.java:339) >>>at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl >>> ainSocketImpl.java:200) >>>at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket >>> Impl.java:182) >>>at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>>at java.net.Socket.connect(Socket.java:579) >>> >>> However, after failed to get any election messages from Node 3, it >>> started a new round but then I seem to see the election messages from Node >>> 2 again. Any idea why? >>> >>> 2016-10-03 12:38:32,815 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1: >>> 5002:QuorumPeer@714] - LOOKING >>> 2016-10-03 12:38:32,816 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1: >>> 5002:FileSnap@83] - Reading snapshot /var/lib/zookeeper/version-2/s >>> napshot.20002aa0b >>> 2016-10-03 12:38:32,875 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1: >>> 5002:FastLeaderElection@815] - New election. My id = 1, proposed >>> zxid=0x20002bfb5 >>> 2016-10-03 12:38:32,876 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLe >>> aderElection@597] - Notification: 1 (message format version), 1 >>> (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 >>> (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) >>> 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLe >>> aderElection@597] - Notification: 1 (message format version), 2 >>> (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 2 >>> (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) >>> 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLe >>> aderElection@597] - Notification: 1 (message format version), 2 >>> (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 >>> (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) >>> 2016-10-03 12:38:32,917 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLe >>> aderElection@597] - Notification: 1 (message format version), 3 >>> (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 >>> (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) >>> 2016-10-03 12:38:33,117 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1: >>> 5002:FastLeaderElection@849] - Notification time out: 400 >>> 2016-10-03 12:38:33,118 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLe >>> aderElection@597] - Notification: 1 (mess
Re: Zookeeper leader election takes a long time.
Btw, also, from n3.log it looks like id: 3 actually started to follow id:2 016-10-03 12:39:18,682 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x7 (n.round), LEADING (n.state), 2 (n.sid), 0x3 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:39:18,722 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x7 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x3 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:39:18,722 [myid:3] - INFO [QuorumPeer[myid=3]/127.0.0.1:5002:QuorumPeer@784] - FOLLOWING 2016-10-03 12:39:18,722 [myid:3] - INFO [QuorumPeer[myid=3]/127.0.0.1:5002:ZooKeeperServer@162] - Created server with tickTime 5000 minSessionTimeout 1 maxSessionTimeout 10 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2 2016-10-03 12:39:18,722 [myid:3] - INFO [QuorumPeer[myid=3]/127.0.0.1:5002:Follower@63] - FOLLOWING - LEADER ELECTION TOOK - 182 2016-10-03 12:39:18,756 [myid:3] - INFO [QuorumPeer[myid=3]/127.0.0.1:5002:Learner@323] - Getting a diff from the leader 0x30130 2016-10-03 12:39:18,760 [myid:3] - WARN [QuorumPeer[myid=3]/127.0.0.1:5002:Learner@374] - Got zxid 0x30001 expected 0x1 2016-10-03 12:39:18,763 [myid:3] - INFO [QuorumPeer[myid=3]/127.0.0.1:5002:FileTxnSnapLog@240] - Snapshotting: 0x30130 to /var/lib/zookeeper/version-2/snapshot.30130 2016-10-03 12:39:18,801 [myid:3] - WARN [QuorumPeer[myid=3]/127.0.0.1:5002:Follower@118] - Got zxid 0x30133 expected 0x1 On Oct 13, 2016, at 3:41 PM, Ryan Zhang mailto:yangzhangr...@hotmail.com>> wrote: Hi, Anand, I took a look and I wonder how do you explain the following The N1.log starts at around 2016-10-03 12:37:38,073 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:QuorumPeer@714] - LOOKING and it failed to talk to Node 2 (id: 2) 2016-10-03 12:37:38,136 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 2 at election address node2.controller.local/127.0.0.8:5004 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) However, after failed to get any election messages from Node 3, it started a new round but then I seem to see the election messages from Node 2 again. Any idea why? 2016-10-03 12:38:32,815 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:QuorumPeer@714] - LOOKING 2016-10-03 12:38:32,816 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FileSnap@83] - Reading snapshot /var/lib/zookeeper/version-2/snapshot.20002aa0b 2016-10-03 12:38:32,875 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@815] - New election. My id = 1, proposed zxid=0x20002bfb5 2016-10-03 12:38:32,876 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,917 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,117 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@849] - Notification time out: 400 2016-10-03 12:38:33,118 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,159 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,559 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElecti
Re: Zookeeper leader election takes a long time.
Hi, Anand, I took a look and I wonder how do you explain the following The N1.log starts at around 2016-10-03 12:37:38,073 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:QuorumPeer@714] - LOOKING and it failed to talk to Node 2 (id: 2) 2016-10-03 12:37:38,136 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 2 at election address node2.controller.local/127.0.0.8:5004 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) However, after failed to get any election messages from Node 3, it started a new round but then I seem to see the election messages from Node 2 again. Any idea why? 2016-10-03 12:38:32,815 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:QuorumPeer@714] - LOOKING 2016-10-03 12:38:32,816 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FileSnap@83] - Reading snapshot /var/lib/zookeeper/version-2/snapshot.20002aa0b 2016-10-03 12:38:32,875 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@815] - New election. My id = 1, proposed zxid=0x20002bfb5 2016-10-03 12:38:32,876 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 2 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,878 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:32,917 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,117 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@849] - Notification time out: 400 2016-10-03 12:38:33,118 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,159 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,559 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@849] - Notification time out: 800 2016-10-03 12:38:33,560 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LOOKING (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,561 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x20002bfb5 (n.zxid), 0x6 (n.round), LEADING (n.state), 2 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:33,601 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 3 (n.leader), 0x20002bfb5 (n.zxid), 0x4 (n.round), LEADING (n.state), 3 (n.sid), 0x2 (n.peerEpoch) LOOKING (my state) 2016-10-03 12:38:34,402 [myid:1] - INFO [QuorumPeer[myid=1]/127.0.0.1:5002:FastLeaderElection@849] - Notification time out: 1600 On Oct 8, 2016, at 10:05 AM, Anand Parthasarathy mailto:anpar...@avinetworks.com>> wrote: Hi Flavio, I have attached the logs from node 1 and node 3. Node 2 was powered off around 10-03 12:36. Leader election kept going until 10-03 15:57:16 when it finally converged. Thanks, Anand. On Sat, Oct 8, 2016 at 7:55 AM, Flavio Junqueira mailto:f...@apache.org>> wrote: Hi Anand, I don't understand whether 1 and 3 were able or even trying to connect to each other. They should be able to elect a leader between them and make progress. You might want to upload logs and let us know. -Flavio > On 08 Oct 2016, at 02:11, Anand Parthasarathy > mailto:anpar...@avinetworks.com>> wrote: > > Hi, > > We are currently using zookeeper 3.4.6 version and use a 3 node solution in > our system. We see that occas
Re: november meetup at facebook (take 2)
+1. My coworkers in twitter would be interested. > On Sep 30, 2016, at 2:35 PM, Raúl Gutiérrez Segalés > wrote: > > +1 (probably bringing along some people from Pinterest as well). > > -rgs > > On Sep 30, 2016 2:26 PM, "Marshall McMullen" > wrote: > > +1. I would love to attend along with a few of my coworkers and this date > works for us. > > On Fri, Sep 30, 2016 at 3:10 PM, Benjamin Reed wrote: > >> facebook would like to host a zookeeper meetup in our offices in menlo >> park, ca on november 17th (a thursday). before sending out an official >> invitation with details about logistics, i thought i would first do a > quick >> date check and make sure that there isn't a big scheduling conflict that > we >> didn't notice (like a big election or something like that...). it's a bit >> tricky to book facilities here, so we don't have a lot of options on > dates. >> >> would this date work for most people? >> >> thanx >> ben >> >> ps - should i cross post to dev@? i assume that most subscribers of user@ >> also subscribe to dev@ >>
