Zookeeper - Network partitioning behaviour

[Imesha Sudasingha]

Hi all,

When I was going through zookeeper, I noticed the following scenarios,

   - Suppose we have a 5 member quorum operating as normal. In a network
   partitioning where the leader and a member node goes into one partition and
   other 3 nodes get into the other network partition. I known then the side
   with the majority will elect a new leader and carry on to serve requests.
   But will the partition with a minority will also continue to serve read
   requests until a write request comes to the leader in the minority
   partition? How does zookeeper handle this occasion?
   - In the above scenario, what will happen to the "watches" that have
   already been registered to a node in the minority partition?

Can you clarify the above scenarios?
Thanks in advance!

-- 
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and  Engineering,
University of Moratuwa.
+94717086160
View in Linkedin 

RE: EXTERNAL: Zookeeper - Network partitioning behaviour

[Daly, Jeff]

410-379-2411

-Original Message-
From: Imesha Sudasingha [mailto:imesha...@cse.mrt.ac.lk] 
Sent: Wednesday, October 05, 2016 9:10 AM
To: user@zookeeper.apache.org
Subject: EXTERNAL: Zookeeper - Network partitioning behaviour

Hi all,

When I was going through zookeeper, I noticed the following scenarios,

   - Suppose we have a 5 member quorum operating as normal. In a network
   partitioning where the leader and a member node goes into one partition and
   other 3 nodes get into the other network partition. I known then the side
   with the majority will elect a new leader and carry on to serve requests.
   But will the partition with a minority will also continue to serve read
   requests until a write request comes to the leader in the minority
   partition? How does zookeeper handle this occasion?
   - In the above scenario, what will happen to the "watches" that have
   already been registered to a node in the minority partition?

Can you clarify the above scenarios?
Thanks in advance!

--
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and  Engineering, University of 
Moratuwa.
+94717086160
View in Linkedin 

RE: EXTERNAL: Zookeeper - Network partitioning behaviour

[Daly, Jeff]

-Original Message-
From: Imesha Sudasingha [mailto:imesha...@cse.mrt.ac.lk] 
Sent: Wednesday, October 05, 2016 9:10 AM
To: user@zookeeper.apache.org
Subject: EXTERNAL: Zookeeper - Network partitioning behaviour

Hi all,

When I was going through zookeeper, I noticed the following scenarios,

   - Suppose we have a 5 member quorum operating as normal. In a network
   partitioning where the leader and a member node goes into one partition and
   other 3 nodes get into the other network partition. I known then the side
   with the majority will elect a new leader and carry on to serve requests.
   But will the partition with a minority will also continue to serve read
   requests until a write request comes to the leader in the minority
   partition? How does zookeeper handle this occasion?
   - In the above scenario, what will happen to the "watches" that have
   already been registered to a node in the minority partition?

Can you clarify the above scenarios?
Thanks in advance!

--
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and  Engineering, University of 
Moratuwa.
+94717086160
View in Linkedin 

Recall: EXTERNAL: Zookeeper - Network partitioning behaviour

[Daly, Jeff]

Daly, Jeff (US) would like to recall the message, "EXTERNAL: Zookeeper - 
Network partitioning behaviour".

RE: ZooKeeper clients does not handle new error codes properly

[Mohammad arshad]

Thanks Michael Han for the information.
It is the same place in my case as well.

-Original Message-
From: Michael Han [mailto:h...@cloudera.com] 
Sent: 04 October 2016 23:08
To: UserZooKeeper
Cc: d...@zookeeper.apache.org
Subject: Re: ZooKeeper clients does not handle new error codes properly

Hi Arshad,

I am curious where the IllegalArgumentException was thrown in code in your use 
case - I am asking because I spotted that in KeeperException.create, we have a 
default fall through [1] that throws IllegalArgumentException, and if this is 
not the place where you observed the IllegalArgumentException in your upgrade 
use case, then we may also need think to change the default fall through here 
from throwing IllegalArgumentException to return something like 
SystemErrorException, because it seems possible that unknown / not mapped error 
codes can be passed to KeeperException.create that will lead to 
IllegalArgumentException that client can't handle well?

[1] https://github.com/apache/zookeeper/blob/master/src/
java/main/org/apache/zookeeper/KeeperException.java#L144

On Tue, Oct 4, 2016 at 8:21 AM, Mohammad arshad 
wrote:

> Thanks Benjamin Reed,
> Client protocol version is 0, it is not changed. Also there is no 
> logic executed at the server side based on the client protocol version.
> Increasing the client protocol version and converting the errors at 
> server may be the ideal and more precise solution but to handle it in 
> a generic way can we convert the unknown error to 
> KeeperException.SystemErrorException
> at client side? Do you foresee any problem in this solution?
>
> Thanks
> -Arshad
>
>
> -Original Message-
> From: Benjamin Reed [mailto:br...@apache.org]
> Sent: 04 October 2016 10:27
> To: user@zookeeper.apache.org
> Cc: DevZooKeeper
> Subject: Re: ZooKeeper clients does not handle new error codes 
> properly
>
> did we bump the protocol version when we added the new errors? the 
> server could do the conversion when it responds to older clients.
>
> On Mon, Oct 3, 2016 at 3:05 AM, Flavio Junqueira  wrote:
>
> > Hi Arshad,
> >
> > It makes sense to me. What if we convert unknown server errors to 
> > KeeperException.SystemErrorException? This is a generic error and it 
> > extends KeeperException.
> >
> > I don't see it as a big issue to make this change, but others may 
> > feel differently. If we do it, then we will need a release note 
> > pointing out the change of behavior.
> >
> > -Flavio
> >
> > > On 03 Oct 2016, at 08:54, Mohammad arshad 
> > > 
> > wrote:
> > >
> > > Hi All,
> > > In Zookeeper rolling upgrade scenario where server is new but 
> > > client is
> > old, when sever sends error code which is not understood by a 
> > client, client throws IllegalArgumentException. Generally 
> > IllegalArgumentException is not handled by any of the ZK applications.
> > It is too generic. How to handle this scenario in ZK applications?
> > > My understanding is instead of throwing IllegalArgumentException 
> > > we
> > should throw a subclass of KeeperException, for example 
> > InvalidErrorCodeException, so that zk apps can take more specific action.
> > > Any thoughts?
> > >
> > > Thanks
> > > -Arshad
> > >
> >
> >
>



--
Cheers
Michael.

tracing client request/responses

[Benjamin Reed]

anyone know how to turn on tracing to get a file that i can run through the
TraceFormatter? i see the 4 letter works to set tracing masks, but i can't
seem to find the switch to actually turn on tracing.

thanx
ben

Re: tracing client request/responses

[Raúl Gutiérrez Segalés]

On 5 October 2016 at 17:39, Benjamin Reed  wrote:

> anyone know how to turn on tracing to get a file that i can run through the
> TraceFormatter? i see the 4 letter works to set tracing masks, but i can't
> seem to find the switch to actually turn on tracing.
>

maybe it helps: i haven't had much luck with trace (too noisy); so i end up
sniffing at packets as they
go out of the client and into the server (and back again) by running
zk-dump on the two ends:

https://github.com/twitter/zktraffic


-rgs

Re: tracing client request/responses

[Benjamin Reed]

yeah i tried zk-dump, but it ended up missing way too many packets.

On Wed, Oct 5, 2016 at 5:50 PM, Raúl Gutiérrez Segalés 
wrote:

> On 5 October 2016 at 17:39, Benjamin Reed  wrote:
>
> > anyone know how to turn on tracing to get a file that i can run through
> the
> > TraceFormatter? i see the 4 letter works to set tracing masks, but i
> can't
> > seem to find the switch to actually turn on tracing.
> >
>
> maybe it helps: i haven't had much luck with trace (too noisy); so i end up
> sniffing at packets as they
> go out of the client and into the server (and back again) by running
> zk-dump on the two ends:
>
> https://github.com/twitter/zktraffic
>
>
> -rgs
>

Re: Zookeeper - Network partitioning behaviour

[Rakesh Radhakrishnan]

Hi Imesha Sudasingha,

For example, we have A,B,C,D,E five servers, that formed quorum and assume
A is the Leader.  Again assume network partition happened between
A,B(minority region)  and C,D,E. As we know, 3 is the majority factor to
form quorum. Since Leader A is in the minority region, the entire quorum
will get shutdown and all the servers will automatically moves to leader
election phase. Now, A & B will do sending notifications each other and
will never succeed to form quorum due to <3 factor. On the other side, C,D
and E will participate each other and elect one of them as Leader.

>But will the partition with a minority will also continue to serve read
>requests until a write request comes to the leader in the minority
>partition?
After the partition, all the servers in the minority region will get
shutdown and moves to leader election phase. All the client sessions
connected to these servers will be disconnected and
will receive "KeeperState.Disconnected" event to their watchers, if any
registered.

But ZooKeeper supports read-only server mode. In this mode, client can
connect to the read-only server even when the server might be partitioned
from the quorum.

Reference:- Please read the section "Read Only Mode Server" in the apache
document link, https://zookeeper.apache.org/doc/r3.4.9/zookeeperAdmin.html


>In the above scenario, what will happen to the "watches" that have
>already been registered to a node in the minority partition?
Since the quorum is re-forming, all the client session watchers will
receive "KeeperState.Disconnected" event. These client session will start
sending connection request to all quorum servers(A,B,C,D,E) to re-establish
the connection. In my above example, C,D,E re-forms quorum successfully and
client sessions will reconnect to one of these servers in the quorum(am
assuming zkclient has the C,D,E server host address configured). Clients
automatically reset watches during successful session reconnect. There is a
feature to disable this watch resetting, please read the configuration
"zookeeper.disableAutoWatchReset" section in the following apache doc link
to understand more on this,
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html

Regards,
Rakesh

On Wed, Oct 5, 2016 at 6:39 PM, Imesha Sudasingha 
wrote:

> Hi all,
>
> When I was going through zookeeper, I noticed the following scenarios,
>
>- Suppose we have a 5 member quorum operating as normal. In a network
>partitioning where the leader and a member node goes into one partition
> and
>other 3 nodes get into the other network partition. I known then the
> side
>with the majority will elect a new leader and carry on to serve
> requests.
>But will the partition with a minority will also continue to serve read
>requests until a write request comes to the leader in the minority
>partition? How does zookeeper handle this occasion?
>- In the above scenario, what will happen to the "watches" that have
>already been registered to a node in the minority partition?
>
> Can you clarify the above scenarios?
> Thanks in advance!
>
> --
> *Imesha Sudasingha*
> Undergraduate of Department of Computer Science and  Engineering,
> University of Moratuwa.
> +94717086160
> View in Linkedin 
>