Re: How find if the zxid is reaching the limit (zxid lower 32 bits have rolled over, forcing re-election)

[rammohan ganapavarapu]

Hi,

One more question, does the proposal count metric get reset for every
leader election? What i observed is that the old leader still has the
highest proposal count.

[image: image.png]
Thanks,
Ram


On Wed, Aug 24, 2022 at 4:51 AM Szalay-Bekő Máté 
wrote:

> Hello Ram,
>
> sorry, I don't really understand the question. The zxid is a 64 bit long
> number. The upper 32 bits are coding an election epoch number (a logical
> time / counter for leader elections), while the bottom 32 bits are counting
> / providing an auto incremented id for all the changes made (committed) in
> ZooKeeper. As far as I understood, the followers are sending proposals to
> the leader, and each accepted (committed) proposal will result in an
> increase in the zxid. The "current" / "latest" zxid is the same in the
> whole cluster (of course followers can lag behind a little, but not much in
> theory. if they are in-sync and part of the quorum).
>
> My understanding is that what you want to catch, is the event when the
> lower 32 bits of the zxid is approaching 0x . As when the last 32
> bits of the zxid is reaching 0x, then a new leader election will be
> triggered automatically and ZooKeeper won't be able to serve for a short
> period of time. And I guess you want to control this event and maybe
> restart the leader manually in a time what is suiting you better?
>
> But maybe I misunderstood your question.
>
> Máté
>
> On Tue, Aug 23, 2022 at 11:00 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Máté,
> >
> > Thanks for quick reply, yes i did see that srvr command can give the
> > current zxid, I also see a metric in mntr "proposal_count" which gives
> > total proposals and when we hit the zxid limit that is matching with the
> > proposal_count  2^32=*4,294,967,296)*metric. So i am trying to understand
> > how this zxid will get incitement ? I don't see zxid in logs for normal
> > events other than leader elections time.
> >
> > Ram
> >
> >
> >
> > On Tue, Aug 23, 2022 at 10:10 AM Szalay-Bekő Máté <
> > szalay.beko.m...@gmail.com> wrote:
> >
> > > Hello!
> > >
> > > I think the "srvr" 4-letter-word diagnostic command should print you
> the
> > > current zxid. Also the similar command works on the Admin Rest API (if
> it
> > > is enabled).
> > >
> > > See:
> > >
> >
> https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands
> > >
> > > An example:
> > >
> > >
> > > echo srvr | nc localhost 2181
> > >
> > > Zookeeper version: 3.5.5-136-69648f116c849ccd757e97c26d3450022d4b1dae,
> > > built on 08/08/2022 11:04 GMT
> > > Latency min/avg/max: 0/0/1808
> > > Received: 9599434
> > > Sent: 9673689
> > > Connections: 41
> > > Outstanding: 0
> > > Zxid: 0x2000afcbf <- this line
> > > Mode: leader
> > > Node count: 1384
> > > Proposal sizes last/min/max: 32/32/4226
> > >
> > >
> > >
> > >
> > > Also the zxid is added to the name of the snapshots / transaction log
> > > files, which are flushed to the file system. Like:  log.  or
> > > snapshot.
> > >
> > > e.g.:
> > >
> > > ls -la -R /var/lib/zookeeper/version-2/
> > >
> > > /var/lib/zookeeper/version-2/:
> > > total 57328
> > > drwxr-xr-x 2 zookeeper zookeeper 4096 Aug 23 10:42 .
> > > drwxr-x--- 3 zookeeper zookeeper 4096 Aug  9 10:41 ..
> > > -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 acceptedEpoch
> > > -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 currentEpoch
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 17 10:09 log.20004c9fc
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 19 00:37 log.20005a541
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 20 18:43 log.20006fc19
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 21 21:40 log.200087550
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 06:30 log.200096ed6
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 17:05 log.2000a9c57
> > > -rw-r--r-- 1 zookeeper zookeeper  1372956 Aug 17 10:09
> snapshot.20005a540
> > > -rw-r--r-- 1 zookeeper zookeeper  1370403 Aug 19 00:37
> snapshot.20006fc18
> > > -rw-r--r-- 1 zookeeper zookeeper  1369122 Aug 20 18:43
> snapshot.20008754f
> > > -rw-r--r-- 1 zookeeper zookeeper  1369034 Aug 21 21:40
> snapshot.200096ed4
> > > -rw-r--r-- 1 zookeeper zookeeper  1379613 Aug 23 06:30
> snapshot.2000a9c56
> > >
> > >
> > >
> > > Best regards,
> > > Máté
> > >
> > > On Tue, Aug 23, 2022 at 6:55 PM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We recently had a leader election due to "*zxid lower 32 bits have
> > rolled
> > > > over, forcing re-election*". This is the first time we are seeing
> this
> > > and
> > > > trying to understand how to find if the ensemble is reaching that
> > limit.
> > > > Are there any metrics available in zk to track this? How can I
> estimate
> > > > when my zk cluster will reach this limit?
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>

Re: How find if the zxid is reaching the limit (zxid lower 32 bits have rolled over, forcing re-election)

[rammohan ganapavarapu]

Yes, your understanding is correct. I would like to predict it and control
the leader election by manual restart.

Thanks

On Wed, Aug 24, 2022, 4:51 AM Szalay-Bekő Máté 
wrote:

> Hello Ram,
>
> sorry, I don't really understand the question. The zxid is a 64 bit long
> number. The upper 32 bits are coding an election epoch number (a logical
> time / counter for leader elections), while the bottom 32 bits are counting
> / providing an auto incremented id for all the changes made (committed) in
> ZooKeeper. As far as I understood, the followers are sending proposals to
> the leader, and each accepted (committed) proposal will result in an
> increase in the zxid. The "current" / "latest" zxid is the same in the
> whole cluster (of course followers can lag behind a little, but not much in
> theory. if they are in-sync and part of the quorum).
>
> My understanding is that what you want to catch, is the event when the
> lower 32 bits of the zxid is approaching 0x . As when the last 32
> bits of the zxid is reaching 0x, then a new leader election will be
> triggered automatically and ZooKeeper won't be able to serve for a short
> period of time. And I guess you want to control this event and maybe
> restart the leader manually in a time what is suiting you better?
>
> But maybe I misunderstood your question.
>
> Máté
>
> On Tue, Aug 23, 2022 at 11:00 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Máté,
> >
> > Thanks for quick reply, yes i did see that srvr command can give the
> > current zxid, I also see a metric in mntr "proposal_count" which gives
> > total proposals and when we hit the zxid limit that is matching with the
> > proposal_count  2^32=*4,294,967,296)*metric. So i am trying to understand
> > how this zxid will get incitement ? I don't see zxid in logs for normal
> > events other than leader elections time.
> >
> > Ram
> >
> >
> >
> > On Tue, Aug 23, 2022 at 10:10 AM Szalay-Bekő Máté <
> > szalay.beko.m...@gmail.com> wrote:
> >
> > > Hello!
> > >
> > > I think the "srvr" 4-letter-word diagnostic command should print you
> the
> > > current zxid. Also the similar command works on the Admin Rest API (if
> it
> > > is enabled).
> > >
> > > See:
> > >
> >
> https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands
> > >
> > > An example:
> > >
> > >
> > > echo srvr | nc localhost 2181
> > >
> > > Zookeeper version: 3.5.5-136-69648f116c849ccd757e97c26d3450022d4b1dae,
> > > built on 08/08/2022 11:04 GMT
> > > Latency min/avg/max: 0/0/1808
> > > Received: 9599434
> > > Sent: 9673689
> > > Connections: 41
> > > Outstanding: 0
> > > Zxid: 0x2000afcbf <- this line
> > > Mode: leader
> > > Node count: 1384
> > > Proposal sizes last/min/max: 32/32/4226
> > >
> > >
> > >
> > >
> > > Also the zxid is added to the name of the snapshots / transaction log
> > > files, which are flushed to the file system. Like:  log.  or
> > > snapshot.
> > >
> > > e.g.:
> > >
> > > ls -la -R /var/lib/zookeeper/version-2/
> > >
> > > /var/lib/zookeeper/version-2/:
> > > total 57328
> > > drwxr-xr-x 2 zookeeper zookeeper 4096 Aug 23 10:42 .
> > > drwxr-x--- 3 zookeeper zookeeper 4096 Aug  9 10:41 ..
> > > -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 acceptedEpoch
> > > -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 currentEpoch
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 17 10:09 log.20004c9fc
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 19 00:37 log.20005a541
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 20 18:43 log.20006fc19
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 21 21:40 log.200087550
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 06:30 log.200096ed6
> > > -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 17:05 log.2000a9c57
> > > -rw-r--r-- 1 zookeeper zookeeper  1372956 Aug 17 10:09
> snapshot.20005a540
> > > -rw-r--r-- 1 zookeeper zookeeper  1370403 Aug 19 00:37
> snapshot.20006fc18
> > > -rw-r--r-- 1 zookeeper zookeeper  1369122 Aug 20 18:43
> snapshot.20008754f
> > > -rw-r--r-- 1 zookeeper zookeeper  1369034 Aug 21 21:40
> snapshot.200096ed4
> > > -rw-r--r-- 1 zookeeper zookeeper  1379613 Aug 23 06:30
> snapshot.2000a9c56
> > >
> > >
> > >
> > > Best regards,
> > > Máté
> > >
> > > On Tue, Aug 23, 2022 at 6:55 PM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We recently had a leader election due to "*zxid lower 32 bits have
> > rolled
> > > > over, forcing re-election*". This is the first time we are seeing
> this
> > > and
> > > > trying to understand how to find if the ensemble is reaching that
> > limit.
> > > > Are there any metrics available in zk to track this? How can I
> estimate
> > > > when my zk cluster will reach this limit?
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>

Re: How find if the zxid is reaching the limit (zxid lower 32 bits have rolled over, forcing re-election)

[rammohan ganapavarapu]

Máté,

Thanks for quick reply, yes i did see that srvr command can give the
current zxid, I also see a metric in mntr "proposal_count" which gives
total proposals and when we hit the zxid limit that is matching with the
proposal_count  2^32=*4,294,967,296)*metric. So i am trying to understand
how this zxid will get incitement ? I don't see zxid in logs for normal
events other than leader elections time.

Ram



On Tue, Aug 23, 2022 at 10:10 AM Szalay-Bekő Máté <
szalay.beko.m...@gmail.com> wrote:

> Hello!
>
> I think the "srvr" 4-letter-word diagnostic command should print you the
> current zxid. Also the similar command works on the Admin Rest API (if it
> is enabled).
>
> See:
> https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands
>
> An example:
>
>
> echo srvr | nc localhost 2181
>
> Zookeeper version: 3.5.5-136-69648f116c849ccd757e97c26d3450022d4b1dae,
> built on 08/08/2022 11:04 GMT
> Latency min/avg/max: 0/0/1808
> Received: 9599434
> Sent: 9673689
> Connections: 41
> Outstanding: 0
> Zxid: 0x2000afcbf <- this line
> Mode: leader
> Node count: 1384
> Proposal sizes last/min/max: 32/32/4226
>
>
>
>
> Also the zxid is added to the name of the snapshots / transaction log
> files, which are flushed to the file system. Like:  log.  or
> snapshot.
>
> e.g.:
>
> ls -la -R /var/lib/zookeeper/version-2/
>
> /var/lib/zookeeper/version-2/:
> total 57328
> drwxr-xr-x 2 zookeeper zookeeper 4096 Aug 23 10:42 .
> drwxr-x--- 3 zookeeper zookeeper 4096 Aug  9 10:41 ..
> -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 acceptedEpoch
> -rw-r--r-- 1 zookeeper zookeeper1 Aug 10 17:55 currentEpoch
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 17 10:09 log.20004c9fc
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 19 00:37 log.20005a541
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 20 18:43 log.20006fc19
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 21 21:40 log.200087550
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 06:30 log.200096ed6
> -rw-r--r-- 1 zookeeper zookeeper 67108880 Aug 23 17:05 log.2000a9c57
> -rw-r--r-- 1 zookeeper zookeeper  1372956 Aug 17 10:09 snapshot.20005a540
> -rw-r--r-- 1 zookeeper zookeeper  1370403 Aug 19 00:37 snapshot.20006fc18
> -rw-r--r-- 1 zookeeper zookeeper  1369122 Aug 20 18:43 snapshot.20008754f
> -rw-r--r-- 1 zookeeper zookeeper  1369034 Aug 21 21:40 snapshot.200096ed4
> -rw-r--r-- 1 zookeeper zookeeper  1379613 Aug 23 06:30 snapshot.2000a9c56
>
>
>
> Best regards,
> Máté
>
> On Tue, Aug 23, 2022 at 6:55 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > We recently had a leader election due to "*zxid lower 32 bits have rolled
> > over, forcing re-election*". This is the first time we are seeing this
> and
> > trying to understand how to find if the ensemble is reaching that limit.
> > Are there any metrics available in zk to track this? How can I estimate
> > when my zk cluster will reach this limit?
> >
> > Thanks,
> > Ram
> >
>

Re: How find if the zxid is reaching the limit (zxid lower 32 bits have rolled over, forcing re-election)

[rammohan ganapavarapu]

Hi,

We recently had a leader election due to "*zxid lower 32 bits have rolled
over, forcing re-election*". This is the first time we are seeing this and
trying to understand how to find if the ensemble is reaching that limit.
Are there any metrics available in zk to track this? How can I estimate
when my zk cluster will reach this limit?

Thanks,
Ram

Any known issues in java client with zookeeper version 3.4.14

[rammohan ganapavarapu]

Hi,

We recently observed that java clients (zkCli) in some of the environments
(RHEL 7 and 8) are sending connection RST even before session/connection
timeout, so I am wondering if there are any known issues.

Thanks,
Ram

Metrics to monitor zookeeper performance/Health

[rammohan ganapavarapu]

Hi,

What are some of the critical metrics that I should be using to monitor the
zookeeper performance and health?

Thanks,
Ram

Re: zktreeutils compilation/build error

[rammohan ganapavarapu]

+d...@zookeeper.apache.org

On Wed, Jun 2, 2021 at 8:53 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Any help would be appreciated!
>
> Thanks,
> Ram
>
> On Thu, May 20, 2021 at 7:58 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to build a zktreeutils from the 3.6.2 source and i am getting
>> below error:
>>
>> Build environment:
>> OS: DebianJDK: JDK11
>> Build procedure:
>> 1. Full build : mvn clean install -Pfull-build -DskipTests <- this was
>> successful.
>>
>> 2. Build C-client:  (successful)
>> cd /apache-zookeeper-3.6.2/zookeeper-client/zookeeper-client-c
>> ./configure
>> make
>>
>> 3. Build zktreeutils
>>
>> autoreconf -if
>> ./configure (successful) ( i had to update the configure file with
>> c-client library path
>> grep client configure
>> # Zookeeper C client
>>
>> ZOOKEEPER_PATH=${BUILD_PATH}/../../zookeeper-client/zookeeper-client-c/target/c)
>> make (failed)
>>
>> No need to remake target 'ZkAdaptor.cc'.
>>Finished prerequisites of target file 'ZkAdaptor.o'.
>>   Must remake target 'ZkAdaptor.o'.
>> make[2]: Entering directory
>> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
>> g++ -DHAVE_CONFIG_H -I. -I..
>>  
>> -I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/include
>> -I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/generated
>> -I../include -I/usr/local/include -I/usr/include -I/usr/include/libxml2 -g
>> -O2 -MT ZkAdaptor.o -MD -MP -MF .deps/ZkAdaptor.Tpo -c -o ZkAdaptor.o
>> ZkAdaptor.cc
>> Putting child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802 on the chain.
>> Live child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802
>> In file included from ZkAdaptor.cc:19:
>> ZkAdaptor.h:26:10: fatal error: zookeeper.h: No such file or directory
>>26 | #include "zookeeper.h"
>>   |  ^
>> compilation terminated.
>> Reaping losing child 0x55cb86a83200 PID 4000802
>> make[2]: *** [Makefile:383: ZkAdaptor.o] Error 1
>> Removing child 0x55cb86a83200 PID 4000802 from chain.
>> make[2]: Leaving directory
>> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
>> Reaping losing child 0x55dfe8963c90 PID 4000795
>> make[1]: *** [Makefile:362: all-recursive] Error 1
>> Removing child 0x55dfe8963c90 PID 4000795 from chain.
>> make[1]: Leaving directory
>> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil'
>> Reaping losing child 0x563fd4c34b00 PID 4000794
>> make: *** [Makefile:303: all] Error 2
>> Removing child 0x563fd4c34b00 PID 4000794 from chain.
>>
>>
>> Any help?
>>
>> Thanks,
>> Ram
>>
>>
>>

Re: zktreeutils compilation/build error

[rammohan ganapavarapu]

Any help would be appreciated!

Thanks,
Ram

On Thu, May 20, 2021 at 7:58 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> I am trying to build a zktreeutils from the 3.6.2 source and i am getting
> below error:
>
> Build environment:
> OS: DebianJDK: JDK11
> Build procedure:
> 1. Full build : mvn clean install -Pfull-build -DskipTests <- this was
> successful.
>
> 2. Build C-client:  (successful)
> cd /apache-zookeeper-3.6.2/zookeeper-client/zookeeper-client-c
> ./configure
> make
>
> 3. Build zktreeutils
>
> autoreconf -if
> ./configure (successful) ( i had to update the configure file with
> c-client library path
> grep client configure
> # Zookeeper C client
>
> ZOOKEEPER_PATH=${BUILD_PATH}/../../zookeeper-client/zookeeper-client-c/target/c)
> make (failed)
>
> No need to remake target 'ZkAdaptor.cc'.
>Finished prerequisites of target file 'ZkAdaptor.o'.
>   Must remake target 'ZkAdaptor.o'.
> make[2]: Entering directory
> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
> g++ -DHAVE_CONFIG_H -I. -I..
>  
> -I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/include
> -I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/generated
> -I../include -I/usr/local/include -I/usr/include -I/usr/include/libxml2 -g
> -O2 -MT ZkAdaptor.o -MD -MP -MF .deps/ZkAdaptor.Tpo -c -o ZkAdaptor.o
> ZkAdaptor.cc
> Putting child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802 on the chain.
> Live child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802
> In file included from ZkAdaptor.cc:19:
> ZkAdaptor.h:26:10: fatal error: zookeeper.h: No such file or directory
>26 | #include "zookeeper.h"
>   |  ^
> compilation terminated.
> Reaping losing child 0x55cb86a83200 PID 4000802
> make[2]: *** [Makefile:383: ZkAdaptor.o] Error 1
> Removing child 0x55cb86a83200 PID 4000802 from chain.
> make[2]: Leaving directory
> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
> Reaping losing child 0x55dfe8963c90 PID 4000795
> make[1]: *** [Makefile:362: all-recursive] Error 1
> Removing child 0x55dfe8963c90 PID 4000795 from chain.
> make[1]: Leaving directory
> '/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil'
> Reaping losing child 0x563fd4c34b00 PID 4000794
> make: *** [Makefile:303: all] Error 2
> Removing child 0x563fd4c34b00 PID 4000794 from chain.
>
>
> Any help?
>
> Thanks,
> Ram
>
>
>

zktreeutils compilation/build error

[rammohan ganapavarapu]

Hi,

I am trying to build a zktreeutils from the 3.6.2 source and i am getting
below error:

Build environment:
OS: DebianJDK: JDK11
Build procedure:
1. Full build : mvn clean install -Pfull-build -DskipTests <- this was
successful.

2. Build C-client:  (successful)
cd /apache-zookeeper-3.6.2/zookeeper-client/zookeeper-client-c
./configure
make

3. Build zktreeutils

autoreconf -if
./configure (successful) ( i had to update the configure file with c-client
library path
grep client configure
# Zookeeper C client
ZOOKEEPER_PATH=${BUILD_PATH}/../../zookeeper-client/zookeeper-client-c/target/c)
make (failed)

No need to remake target 'ZkAdaptor.cc'.
   Finished prerequisites of target file 'ZkAdaptor.o'.
  Must remake target 'ZkAdaptor.o'.
make[2]: Entering directory
'/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
g++ -DHAVE_CONFIG_H -I. -I..
 
-I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/include
-I/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/../../zookeeper-client/zookeeper-client-c/target/c/generated
-I../include -I/usr/local/include -I/usr/include -I/usr/include/libxml2 -g
-O2 -MT ZkAdaptor.o -MD -MP -MF .deps/ZkAdaptor.Tpo -c -o ZkAdaptor.o
ZkAdaptor.cc
Putting child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802 on the chain.
Live child 0x55cb86a83200 (ZkAdaptor.o) PID 4000802
In file included from ZkAdaptor.cc:19:
ZkAdaptor.h:26:10: fatal error: zookeeper.h: No such file or directory
   26 | #include "zookeeper.h"
  |  ^
compilation terminated.
Reaping losing child 0x55cb86a83200 PID 4000802
make[2]: *** [Makefile:383: ZkAdaptor.o] Error 1
Removing child 0x55cb86a83200 PID 4000802 from chain.
make[2]: Leaving directory
'/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil/src'
Reaping losing child 0x55dfe8963c90 PID 4000795
make[1]: *** [Makefile:362: all-recursive] Error 1
Removing child 0x55dfe8963c90 PID 4000795 from chain.
make[1]: Leaving directory
'/apache-zookeeper-3.6.2/zookeeper-contrib/zookeeper-contrib-zktreeutil'
Reaping losing child 0x563fd4c34b00 PID 4000794
make: *** [Makefile:303: all] Error 2
Removing child 0x563fd4c34b00 PID 4000794 from chain.


Any help?

Thanks,
Ram

End of Life for Zookeeper 3.6 and 3.7

[rammohan ganapavarapu]

Hi,

Where can I find EOL timelines for 3.6 and 3.7 or any other major version
of zk?

Thanks,
Ram

Zookeeper version 3.6 performance comparison

[rammohan ganapavarapu]

Hi,

Is there any performance/benchmarking comparison available for the
zookeeper version 3.6 with previous versions? We are planning to upgrade to
3.6 from 3.4.* and wanted to make sure we won't see perf drop.

Thanks,
Ram

Dynamic Reconfiguration usage

[rammohan ganapavarapu]

Hi,

Is the dynamic reconfiguration
 feature
only used for the server config or can be used for any other zookeeper
configuration parameters?

For example, can i change the log level from debug to info using dynamic
reconfig with out restarting processes.

Thanks,
Ram

Re: Curator and zookeeper version compatibility

[rammohan ganapavarapu]

Thank you all for quick response.



On Mon, Mar 8, 2021, 5:46 PM ashish soni  wrote:

> Hi,
>
> We upgraded zookeeper from 3.4.12 to 3.5.5 some months ago and at that time
> we were using curator 2.13.0. The upgrade was compatible with curator
> 2.13.0. I didn't try with any of the 3.6.x versions yet.
>
> Regards,
> Aishwarya Soni
>
> On Mon, Mar 8, 2021 at 5:08 PM Cameron McKenzie 
> wrote:
>
> > hey Ram,
> > Ultimately it comes down to whether the Zookeeper server you choose to
> run
> > is backwards compatible with a Zookeeper 3.4.x client that is package
> with
> > Curator 2.11.1. From memory, I think that it may work, but probably
> hasn't
> > been formally tested. I'll defer to someone with greater knowledge on
> this
> > as to whether it will work in practice.
> >
> > Obviously if you go with this approach, the Curator client will not have
> > access to any of the new features in ZK 3.5/3.6.
> > cheers
> >
> >
> >
> > On Tue, Mar 9, 2021 at 11:59 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am trying to find  the compatibility matrix for curator and
> zookeeper.
> > I
> > > am using curator 2.11.1 with zookeeper version 3.4.14. Now i am trying
> to
> > > upgrade zk version 3.6 or 3.5 and not sure if the same client works or
> > not.
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Curator and zookeeper version compatibility

[rammohan ganapavarapu]

Hi,

I am trying to find  the compatibility matrix for curator and zookeeper. I
am using curator 2.11.1 with zookeeper version 3.4.14. Now i am trying to
upgrade zk version 3.6 or 3.5 and not sure if the same client works or not.

Thanks,
Ram

Re: Zookeeper access logs?

[rammohan ganapavarapu]

Partrick,

Thanks for the references, so what loglevel i should be setting to log read
access? I can't rely on connections as the client may be using a pool and
it doesn't translate well with number of reads and writes by a client right?

Ram

On Tue, Jun 23, 2020 at 3:55 PM Patrick Hunt  wrote:

> There is audit capability in newer versions:
> https://zookeeper.apache.org/doc/r3.6.1/zookeeperAuditLogs.html
>
> Earlier versions include support for dumping the transaction log (which
> goes back a long way):
>
> https://zookeeper.apache.org/doc/r3.6.0/apidocs/zookeeper-server/org/apache/zookeeper/server/LogFormatter.html
>
> For read access you may need to enable logging (log4j) or look at the
> connection level metrics.
>
> Patrick
>
> On Tue, Jun 23, 2020 at 3:32 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > Does the zookeeper have access logs, i wanted to see client reads and
> > writes information. Recently we had an incident where some of the clients
> > started pounding zk with a lot of reads and it was hard to find which
> > client was doing that.
> >
> > Ram
> >
>

Zookeeper access logs?

[rammohan ganapavarapu]

Hi,

Does the zookeeper have access logs, i wanted to see client reads and
writes information. Recently we had an incident where some of the clients
started pounding zk with a lot of reads and it was hard to find which
client was doing that.

Ram

Re: Side affects of setting quorumListenOnAllIPs to true

[rammohan ganapavarapu]

Thank you all for your suggestions.

On Tue, Jun 16, 2020, 5:56 AM Szalay-Bekő Máté 
wrote:

> :)
>
> just some info from https://zookeeper.apache.org/security.html
>
> " If you have any concern or believe you have uncovered a vulnerability, we
> suggest that you get in touch via the e-mail address
> secur...@zookeeper.apache.org. In the message, try to provide a
> description
> of the issue and ideally a way of reproducing it. (...) Please report any
> security problems to the project security address before disclosing it
> publicly. "
>
> Kind regards,
> Mate
>
> On Tue, Jun 16, 2020 at 1:36 PM ashish soni 
> wrote:
>
> > Good suggestions Mate. We are in progress to implement both (SSL AND
> SASL).
> > Will try to pan out some destructive cases to test it out :)
> >
> > On Tue, Jun 16, 2020, 4:07 AM Szalay-Bekő Máté <
> szalay.beko.m...@gmail.com
> > >
> > wrote:
> >
> > > Also the best is to use QuorumSASL or QuorumSSL to make sure the
> > ZooKeeper
> > > server-to-server communication is secure and noone who is not trusted
> can
> > > connect and gain access to the quorum.
> > >
> > > However, if one is using QuorumSASL or QuorumSSL then it is still
> > possible
> > > that a DOS attack can hit the ZooKeeper port causing problems. But that
> > can
> > > again be solved by firewalls I think.
> > >
> > > On Tue, Jun 16, 2020 at 12:49 PM Szalay-Bekő Máté <
> > > szalay.beko.m...@gmail.com> wrote:
> > >
> > > > > Mate, suppose we do set quorumListenOnAllIPs to true. Will the
> > > zookeeper
> > > > still connect and form a quorum with only the static or dynamic
> server
> > > > connection strings or it can connect and form a quorum with any IP
> > > address
> > > > outside the server connection strings as it is allowed to bind with a
> > > > 0.0.0.0 interface?
> > > >
> > > > This is a good question. I think there is a chance that one can
> > "intrude"
> > > > this way. Although I wouldn't give more tips on the mailing list. :)
> > > > The best is to protect the ZooKeeper internal network using
> firewalls.
> > > The
> > > > election port and leader port should be reachable only by other
> > ZooKeeper
> > > > server hosts.
> > > >
> > > > Regards,
> > > > Mate
> > > >
> > > > On Tue, Jun 16, 2020 at 12:24 PM ashish soni <
> > aishwarya.ash...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Mate, suppose we do set quorumListenOnAllIPs to true. Will the
> > zookeeper
> > > >> still connect and form a quorum with only the static or dynamic
> server
> > > >> connection strings or it can connect and form a quorum with any IP
> > > address
> > > >> outside the server connection strings as it is allowed to bind with
> a
> > > >> 0.0.0.0 interface?
> > > >>
> > > >> Ram, I think you don't need to add this if you have a static IP
> config
> > > or
> > > >> using 3.6+. If you feel it is a security issue for the organization,
> > try
> > > >> ZK
> > > >> 3.6.1 without setting that config.
> > > >>
> > > >> Regards,
> > > >> Aishwarya Soni
> > > >>
> > > >> On Tue, Jun 16, 2020 at 1:03 AM Szalay-Bekő Máté <
> > > >> szalay.beko.m...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Ram,
> > > >> >
> > > >> > > all i want to know is by enabling this property there are no
> side
> > > >> effects
> > > >> > or security risks.
> > > >> >
> > > >> > well, this is something for you (or for your security team) to
> > > evaluate.
> > > >> > E.g. if your hosts have multiple network interfaces with both
> > > "private"
> > > >> and
> > > >> > "public" networks attached, then I can consider setting
> > > >> > quorumListenOnAllIPs=true to be a security risk. Of course you can
> > > block
> > > >> > the public access with proper firewall rules.
> > > >> >
> > > >> > But usually ZooKeeper is deployed in some se

Re: Side affects of setting quorumListenOnAllIPs to true

[rammohan ganapavarapu]

Mate,

Thanks for explaining, all i want to know is by enabling this property
there are no side effects or security risks.

Ram

On Sun, Jun 14, 2020 at 11:48 PM Szalay-Bekő Máté <
szalay.beko.m...@gmail.com> wrote:

> Hi Ram,
>
> I am not sure I understand your question. The config quorumListenOnAllIPs
> is about to specify if the ports ZooKeeper uses for Server-to-server
> communication should bind on the specified address/IP
> (quorumListenOnAllIPs=false) or on 0.0.0.0 (quorumListenOnAllIPs=true).
>
> An example: You configure your server list using either static or dynamic
> configuration like:
> server.1=a.foo.com:2888:3888
> server.2=b.foo.com:2888:3888
> ...
>
> In this case when server.2 starts, it reads the config then initiates
> connection (for ZK internal leader election protocol) to server.1 by
> connecting to a.foo.com:3888 and sending it's own address (b.foo.com:3888)
> enabling server.1 to connect back. However, if server.2 is behind a proxy /
> using kubernetes / whatever, then it is possible that you can reach
> server.2 as b.foo.com but the ZK process on server.2 can not actually bind
> on b.foo.com:3888. In this case the easiest solution is to bind on
> 0.0.0.0:3888. However, you can not set 0.0.0.0:3888 in the config file of
> server 2, since in this case server.2 would send 0.0.0.0:3888 in the
> initial message to server.1 and server.1 would try to connect back to
> server.2 using 0.0.0.0:3888 what is a bad idea. So in this case it comes
> handy to set quorumListenOnAllIPs=true which will cause ZooKeeper to bind
> on 0.0.0.0:3888 and still send a 'valid' address in the initial message,
> an
> address where other servers can reach it.
>
> I hope the explanation made it more (and not less) clear :p
>
> Kind regards,
> Mate
>
>
> On Fri, Jun 12, 2020 at 7:42 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to see what are the pros and cons of setting
> > quorumListenOnAllIPs to true. Running zookeeper cluster in mtls or local
> > proxy environments is not working by keeping default value (false). So
> can
> > someone please explain?
> >
> > Any way zookeeper will form quorum with the servers list from the
> zoo.conf
> > static file right? so by enabling this property can any server or IP out
> of
> > the zoo.conf can join the quorum?
> >
> > Ram
> >
>

Side affects of setting quorumListenOnAllIPs to true

[rammohan ganapavarapu]

Hi,

I am trying to see what are the pros and cons of setting
quorumListenOnAllIPs to true. Running zookeeper cluster in mtls or local
proxy environments is not working by keeping default value (false). So can
someone please explain?

Any way zookeeper will form quorum with the servers list from the zoo.conf
static file right? so by enabling this property can any server or IP out of
the zoo.conf can join the quorum?

Ram

znode count

[rammohan ganapavarapu]

Hi,

I have restored zk from backup snapshot but i see znode count difference
before and after, any idea?

Before:

zk_version 3.4.5-1392090, built on 09/30/2012 17:52 GMT
zk_avg_latency 0
zk_max_latency 49784
zk_min_latency 0
zk_packets_received 106038250008
zk_packets_sent 106061041183
zk_num_alive_connections 1419
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 12876423
zk_watch_count 317602
zk_ephemerals_count 3016
zk_approximate_data_size 1675131049
zk_open_file_descriptor_count 1501
zk_max_file_descriptor_count 10
zk_followers 24
zk_synced_followers 4
zk_pending_syncs 0

After:

zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on
03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 22
zk_packets_sent 21
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 12876408
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 1675255490
zk_open_file_descriptor_count 54
zk_max_file_descriptor_count 10
zk_fsync_threshold_exceed_count 0
zk_followers 8
zk_synced_followers 4
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 36
zk_min_proposal_size 32

Ram

Re: Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

Andor,


It didnt work, my OS is resolving to new IP but for some reason zk
processes is still resolving to old IP address even with those two
properties. Any other pointers?

Thanks,
Ram

On Wed, Jan 22, 2020 at 1:18 PM Andor Molnar  wrote:

> Yep, give it a try.
>
>
> https://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/net/doc-files/net-properties.html
>
> I don't have personal experience with these settings.
>
> Andor
>
>
>
>
> On Wed, 2020-01-22 at 10:37 -0800, rammohan ganapavarapu wrote:
> > Hi Andor,
> >
> > On OS side the  hostname resolves to new IP, so it could be JVM is
> > the one
> > caching. Any setting son jvm to invalidate cache? In some other posts
> > i did
> > see some one recommending these but not sure it it works.
> >
> > -Dnetworkaddress.cache.ttl=0
> >
> > -Dnetworkaddress.cache.negative.ttl=0
> >
> > Ram
> >
> > On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar 
> > wrote:
> >
> > > Hi Ram,
> > >
> > > As far as I can see from the code, ZooKeeper uses the standard Java
> > > calls
> > > getByName() and getAllByName() every time it’s trying to connect to
> > > a
> > > server.
> > >
> > > 
> > > // zookeeper.ipReachableTimeout is not defined
> > > if (ipReachableTimeout <= 0) {
> > > address = InetAddress.getByName(this.hostname);
> > > } else {
> > > address = getReachableAddress(this.hostname,
> > > ipReachableTimeout);
> > > }
> > > 
> > >
> > > ZK doesn’t (and definitely should not) cache IP addresses. It’s
> > > either the
> > > cache of JVM or your DNS server.
> > >
> > > Dynamic reconfig is available in 3.5.x versions which are already
> > > stable
> > > now and I think with that you don’t need to reuse existing
> > > hostnames.
> > > Instead use reconfig commands to properly remove old nodes and add
> > > new
> > > ones. Sounds like more cumbersome, but maybe more reliable.
> > >
> > > Andor
> > >
> > >
> > >
> > >
> > > > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > > > But still happening for me, is there any config on zookeeper side
> > > > to make
> > > > this fix to work?
> > > >
> > > > Ram
> > > >
> > > > On Tue, Jan 21, 2020 at 2:12 PM Michael Han 
> > > > wrote:
> > > >
> > > > > Could be ZOOKEEPER-1506, though this should be fixed already in
> > > > > 3.4.14.
> > > > >
> > > > > On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> > > > > rammohanga...@gmail.com> wrote:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > I see same with both 3.4.5 and 3.4.14
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <
> > > > > > eolive...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > Which version of ZK are you using?
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> > > > > rammohanga...@gmail.com
> > > > > > > ha scritto:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Does zookeeper cache the host IP? if so how long does it
> > > > > > > > cache? I
> > > > > have
> > > > > > a
> > > > > > > > zk cluster in autoscaling groups and when a new node
> > > > > > > > comes up, other
> > > > > > > nodes
> > > > > > > > still resolving to old IP. Is there any setting to
> > > > > > > > invalidate dns
> > > > > cache
> > > > > > > for
> > > > > > > > zookeeper? or is it jvm dns cache? until other nodes
> > > > > > > > resolves to new
> > > > > > IP,
> > > > > > > > this node not able to join the cluster.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ram
> > > > > > > >
>
>

Reading snapshot

[rammohan ganapavarapu]

Hi,

What is the purpose of reading snapshot from disk while zk processes
starting up? i have a snapshot of ~2gb and it is taking time to read that
snapshot, hence  cluster recovery time. Is it to get the last zxid of that
server?


```2020-01-21 23:11:24,510 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:QuorumPeer@967] - OBSERVING
2020-01-21 23:11:24,510 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@174] - Created
server with tickTime 6 minSessionTimeout 12 maxSessionTimeout
120 datadir /opt/data/zookeeper/logs/version-2 snapdir
/opt/data/zookeeper/data/version-2
2020-01-21 23:11:24,510 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:ObserverZooKeeperServer@64] -
syncEnabled =true
2020-01-21 23:11:24,511 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@185] -
Resolved hostname: rzk100ea to address: rzk100ea/192.168.10.78
2020-01-21 23:11:24,512 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:Observer@65] - Observing
rzk100ea/192.168.10.78:2888
2020-01-21 23:11:24,711 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:FileSnap@86] - Reading snapshot
/opt/data/zookeeper/data/version-2/snapshot.5a
2020-01-21 23:12:31,954 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:Learner@336] - Getting a
snapshot from leader 0x5b
2020-01-21 23:18:32,497 [myid:106] - INFO
 [QuorumPeer[myid=106]/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@301] -
Snapshotting: 0x5b to
/opt/data/zookeeper/data/version-2/snapshot.5b```

Thanks,
Ram

Re: Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

Hi Andor,

On OS side the  hostname resolves to new IP, so it could be JVM is the one
caching. Any setting son jvm to invalidate cache? In some other posts i did
see some one recommending these but not sure it it works.

-Dnetworkaddress.cache.ttl=0

-Dnetworkaddress.cache.negative.ttl=0

Ram

On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar  wrote:

> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
> 
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
> 
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han  wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli 
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Which version of ZK are you using?
> >>>> Enrico
> >>>>
> >>>>
> >>>> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanga...@gmail.com
> >>>>
> >>>> ha scritto:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> >>>>> zk cluster in autoscaling groups and when a new node comes up, other
> >>>> nodes
> >>>>> still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> >>>> for
> >>>>> zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> >>>>> this node not able to join the cluster.
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

Andor,

Thank you, i will look into dynamic reconfig option.

Ram

On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar  wrote:

> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
> 
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
> 
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han  wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli 
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Which version of ZK are you using?
> >>>> Enrico
> >>>>
> >>>>
> >>>> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanga...@gmail.com
> >>>>
> >>>> ha scritto:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> >>>>> zk cluster in autoscaling groups and when a new node comes up, other
> >>>> nodes
> >>>>> still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> >>>> for
> >>>>> zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> >>>>> this node not able to join the cluster.
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [ANNOUNCE] Enrico Olivelli new ZooKeeper PMC Member

[rammohan ganapavarapu]

Congratulations Enrico!!

On Tue, Jan 21, 2020 at 1:41 PM Flavio Junqueira  wrote:

> I'm pleased to announce that Enrico Olivelli recently became the newest
> ZooKeeper PMC member. Enrico has contributed immensely to this community;
> he became a ZooKeeper committer in May 2019 and now he joins the PMC.
>
> Join me in congratulating him on the achievement. Congrats, Enrico!
>
> -Flavio on behalf of the Apache ZooKeeper PMC

Re: Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

But still happening for me, is there any config on zookeeper side to make
this fix to work?

Ram

On Tue, Jan 21, 2020 at 2:12 PM Michael Han  wrote:

> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
>
> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi Enrico,
> >
> > I see same with both 3.4.5 and 3.4.14
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli 
> > wrote:
> >
> > > Hi,
> > > Which version of ZK are you using?
> > > Enrico
> > >
> > >
> > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> rammohanga...@gmail.com
> > >
> > > ha scritto:
> > >
> > > > Hi,
> > > >
> > > > Does zookeeper cache the host IP? if so how long does it cache? I
> have
> > a
> > > > zk cluster in autoscaling groups and when a new node comes up, other
> > > nodes
> > > > still resolving to old IP. Is there any setting to invalidate dns
> cache
> > > for
> > > > zookeeper? or is it jvm dns cache? until other nodes resolves to new
> > IP,
> > > > this node not able to join the cluster.
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>

Re: Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

Hi Enrico,

I see same with both 3.4.5 and 3.4.14

Ram

On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli  wrote:

> Hi,
> Which version of ZK are you using?
> Enrico
>
>
> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu 
> ha scritto:
>
> > Hi,
> >
> > Does zookeeper cache the host IP? if so how long does it cache? I have a
> > zk cluster in autoscaling groups and when a new node comes up, other
> nodes
> > still resolving to old IP. Is there any setting to invalidate dns cache
> for
> > zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
> > this node not able to join the cluster.
> >
> > Thanks,
> > Ram
> >
>

Zookeeper resolving to old host IP addresses

[rammohan ganapavarapu]

Hi,

Does zookeeper cache the host IP? if so how long does it cache? I have a
zk cluster in autoscaling groups and when a new node comes up, other nodes
still resolving to old IP. Is there any setting to invalidate dns cache for
zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
this node not able to join the cluster.

Thanks,
Ram

Re: default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

OK, thank you!

On Tue, Dec 17, 2019 at 7:32 PM Rakesh Radhakrishnan 
wrote:

> As the name says, "quorum.auth.kerberos.servicePrincipal" property is
> specifically for Kerberos based quorum authentication and no need to set
> anything if you are enabling digest-md5.
>
> Like mentioned earlier, its default value is "zkquorum/localhost" and it
> will never be used if you configure/enable digest-md5.
>
> Thanks,
> Rakesh
>
> On Mon, Dec 16, 2019 at 7:14 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > "quorum.auth.kerberos.servicePrincipal" this one
> >
> > On Sun, Dec 15, 2019, 9:33 PM Rakesh Radhakrishnan 
> > wrote:
> >
> > > OK, got it.
> > >
> > > >>>> Even if i enable sasl but md5-diget what should be this property
> set
> > > to,
> > > Could you please name the specific property you are referring.
> > >
> > > Hope you are talking about "DIGEST-MD5" mechanism ? String[] mechs = {
> > > "DIGEST-MD5" };
> > >
> > > Presently the execution flow is that, if there is
> > > no subject.getPrincipals() in jaas config then it must not be GSSAPI
> and
> > > fallback to check DIGEST-MD5 details in jaas config.
> > > Whenever user want to enable DIGEST-MD5, they have to define the JAAS
> > > configuration file with DIGEST-MD5 configs like below and there is no
> > > default value for this mechanism.
> > >  QuorumServer {
> > >org.apache.zookeeper.server.auth.DigestLoginModule required
> > >user_test1="mypassword";
> > >  };
> > >
> > > QuorumLearner {
> > >org.apache.zookeeper.server.auth.DigestLoginModule required
> > >user_test2=" mypassword";
> > >  };
> > >
> > > Populate DIGEST-MD5 user -> password map for the "QuorumServer",
> > > "QuorumLearner" section.
> > > Usernames are distinguished from other options by prefixing the
> username
> > > with a "user_" prefix.
> > >
> > > Hope its clear to you.
> > >
> > > Thanks,
> > > Rakesh
> > >
> > > On Fri, Dec 13, 2019 at 9:45 PM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Hi Rakesh,
> > > >
> > > > Right now i am not enabling sasl but i am trying to define all
> default
> > > > properties and should be able to use them once sasl is enabled with
> > > > override values. So my question is for digest auth do we even need
> this
> > > > property? i remember seeing i don't set that property it was using
> the
> > > > default value "zkquorum/localhost".
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > > > On Thu, Dec 12, 2019 at 11:06 PM Rakesh Radhakrishnan <
> > > rake...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Ram,
> > > > >
> > > > > ZooKeeper Quorum authentication support two schemes, Kerberos or
> > > > > DIGEST-MD5. User has to configure either Kerb or digest
> configuration
> > > > > values. Both together not required.
> > > > >
> > > > > I'd recommend you to go through Kerberos, digest simulation unit
> test
> > > > cases
> > > > > where we have valid and invalid scenarios. Hope this would get idea
> > > about
> > > > > the required configs.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
> > > > >
> > > > > Could you describe the issues that troubles you in setting up
> quorum
> > > > auth,
> > > > > if any.
> > > > >
> > > > > Thanks,
> > > > > Rakesh
> > > > >
> > > > > On Fri, Dec 13, 2019 at 3:49 AM rammohan ganapavarapu <
> > > > > rammohanga...@gmail.com> wrote:
> > > > >
> > > > > > Hi,

Re: default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

"quorum.auth.kerberos.servicePrincipal" this one

On Sun, Dec 15, 2019, 9:33 PM Rakesh Radhakrishnan 
wrote:

> OK, got it.
>
> >>>> Even if i enable sasl but md5-diget what should be this property set
> to,
> Could you please name the specific property you are referring.
>
> Hope you are talking about "DIGEST-MD5" mechanism ? String[] mechs = {
> "DIGEST-MD5" };
>
> Presently the execution flow is that, if there is
> no subject.getPrincipals() in jaas config then it must not be GSSAPI and
> fallback to check DIGEST-MD5 details in jaas config.
> Whenever user want to enable DIGEST-MD5, they have to define the JAAS
> configuration file with DIGEST-MD5 configs like below and there is no
> default value for this mechanism.
>  QuorumServer {
>org.apache.zookeeper.server.auth.DigestLoginModule required
>user_test1="mypassword";
>  };
>
> QuorumLearner {
>org.apache.zookeeper.server.auth.DigestLoginModule required
>user_test2=" mypassword";
>  };
>
> Populate DIGEST-MD5 user -> password map for the "QuorumServer",
> "QuorumLearner" section.
> Usernames are distinguished from other options by prefixing the username
> with a "user_" prefix.
>
> Hope its clear to you.
>
> Thanks,
> Rakesh
>
> On Fri, Dec 13, 2019 at 9:45 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi Rakesh,
> >
> > Right now i am not enabling sasl but i am trying to define all default
> > properties and should be able to use them once sasl is enabled with
> > override values. So my question is for digest auth do we even need this
> > property? i remember seeing i don't set that property it was using the
> > default value "zkquorum/localhost".
> >
> > Thanks,
> > Ram
> >
> > On Thu, Dec 12, 2019 at 11:06 PM Rakesh Radhakrishnan <
> rake...@apache.org>
> > wrote:
> >
> > > Hi Ram,
> > >
> > > ZooKeeper Quorum authentication support two schemes, Kerberos or
> > > DIGEST-MD5. User has to configure either Kerb or digest configuration
> > > values. Both together not required.
> > >
> > > I'd recommend you to go through Kerberos, digest simulation unit test
> > cases
> > > where we have valid and invalid scenarios. Hope this would get idea
> about
> > > the required configs.
> > >
> > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
> > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
> > >
> > > Could you describe the issues that troubles you in setting up quorum
> > auth,
> > > if any.
> > >
> > > Thanks,
> > > Rakesh
> > >
> > > On Fri, Dec 13, 2019 at 3:49 AM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Even if i enable sasl but md5-diget what should be this property set
> > to,
> > > > this property only take effect for kerberos or for both?
> > > >
> > > > Ram
> > > >
> > > > On Fri, Dec 6, 2019 at 7:55 AM rammohan ganapavarapu <
> > > > rammohanga...@gmail.com> wrote:
> > > >
> > > > > Mate,
> > > > >
> > > > > Thank you, I did search source code found the same, I am trying to
> > > create
> > > > > a zoo conf with all default properties.
> > > > >
> > > > > Ram
> > > > >
> > > > > On Fri, Dec 6, 2019, 2:44 AM Mate Szalay-Beko
> > > > 
> > > > > wrote:
> > > > >
> > > > >> Hi Ram,
> > > > >>
> > > > >> this parameter is needed to be defined when you want to enable
> > secure
> > > > >> authentication in the communication between ZooKeeper servers. In
> > > > general,
> > > > >> the 'principal' is a 'username' what you want your ZooKeeper
> servers
> > > to
> > > > >> use
> > > > >> when they talk with each other. Ideally you have a central Kereros
> > > > service
> > > > >> somewhere where this principal is 

Re: default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

Hi Rakesh,

Right now i am not enabling sasl but i am trying to define all default
properties and should be able to use them once sasl is enabled with
override values. So my question is for digest auth do we even need this
property? i remember seeing i don't set that property it was using the
default value "zkquorum/localhost".

Thanks,
Ram

On Thu, Dec 12, 2019 at 11:06 PM Rakesh Radhakrishnan 
wrote:

> Hi Ram,
>
> ZooKeeper Quorum authentication support two schemes, Kerberos or
> DIGEST-MD5. User has to configure either Kerb or digest configuration
> values. Both together not required.
>
> I'd recommend you to go through Kerberos, digest simulation unit test cases
> where we have valid and invalid scenarios. Hope this would get idea about
> the required configs.
>
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/test/java/org/apache/zookeeper/server/quorum/auth/QuorumKerberosHostBasedAuthTest.java
>
> Could you describe the issues that troubles you in setting up quorum auth,
> if any.
>
> Thanks,
> Rakesh
>
> On Fri, Dec 13, 2019 at 3:49 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > Even if i enable sasl but md5-diget what should be this property set to,
> > this property only take effect for kerberos or for both?
> >
> > Ram
> >
> > On Fri, Dec 6, 2019 at 7:55 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Mate,
> > >
> > > Thank you, I did search source code found the same, I am trying to
> create
> > > a zoo conf with all default properties.
> > >
> > > Ram
> > >
> > > On Fri, Dec 6, 2019, 2:44 AM Mate Szalay-Beko
> > 
> > > wrote:
> > >
> > >> Hi Ram,
> > >>
> > >> this parameter is needed to be defined when you want to enable secure
> > >> authentication in the communication between ZooKeeper servers. In
> > general,
> > >> the 'principal' is a 'username' what you want your ZooKeeper servers
> to
> > >> use
> > >> when they talk with each other. Ideally you have a central Kereros
> > service
> > >> somewhere where this principal is already registered.
> > >> A kerberos principal is usually in the form of
> > >> "user_or_service_name/host@realm" (some more explanation:
> > >> https://ssimo.org/blog/id_016.html)
> > >>
> > >> According to the source code, the default value of
> > >> quorum.auth.kerberos.servicePrincipal is "zkquorum/localhost". But I
> > think
> > >> if you don't enable the quorum SASL in ZooKeeper, then this property
> > will
> > >> never be actually used.
> > >>
> > >> Please see this page about SASL in ZooKeeper:
> > >>
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL
> > >>
> > >> I also found a Cloudera blogpost on the topic:
> > >>
> > >>
> >
> https://blog.cloudera.com/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
> > >>
> > >> Cheers,
> > >> Mate
> > >>
> > >>
> > >> On Thu, Dec 5, 2019 at 11:50 PM rammohan ganapavarapu <
> > >> rammohanga...@gmail.com> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > What is the default value for this property, if i don't  enable sasl
> > >> and if
> > >> > i don't define what will be the value?
> > >> >
> > >> > quorum.auth.kerberos.servicePrincipal
> > >> >
> > >> > Also what does this means "servicename/_HOST"
> > >> >
> > >> > Thanks,
> > >> > Ram
> > >> >
> > >>
> > >
> >
>

Re: default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

Hi,

Even if i enable sasl but md5-diget what should be this property set to,
this property only take effect for kerberos or for both?

Ram

On Fri, Dec 6, 2019 at 7:55 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Mate,
>
> Thank you, I did search source code found the same, I am trying to create
> a zoo conf with all default properties.
>
> Ram
>
> On Fri, Dec 6, 2019, 2:44 AM Mate Szalay-Beko 
> wrote:
>
>> Hi Ram,
>>
>> this parameter is needed to be defined when you want to enable secure
>> authentication in the communication between ZooKeeper servers. In general,
>> the 'principal' is a 'username' what you want your ZooKeeper servers to
>> use
>> when they talk with each other. Ideally you have a central Kereros service
>> somewhere where this principal is already registered.
>> A kerberos principal is usually in the form of
>> "user_or_service_name/host@realm" (some more explanation:
>> https://ssimo.org/blog/id_016.html)
>>
>> According to the source code, the default value of
>> quorum.auth.kerberos.servicePrincipal is "zkquorum/localhost". But I think
>> if you don't enable the quorum SASL in ZooKeeper, then this property will
>> never be actually used.
>>
>> Please see this page about SASL in ZooKeeper:
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL
>>
>> I also found a Cloudera blogpost on the topic:
>>
>> https://blog.cloudera.com/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>>
>> Cheers,
>> Mate
>>
>>
>> On Thu, Dec 5, 2019 at 11:50 PM rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > What is the default value for this property, if i don't  enable sasl
>> and if
>> > i don't define what will be the value?
>> >
>> > quorum.auth.kerberos.servicePrincipal
>> >
>> > Also what does this means "servicename/_HOST"
>> >
>> > Thanks,
>> > Ram
>> >
>>
>

Re: quorum.cnxn.threads.size

[rammohan ganapavarapu]

No, my loglevel setting is INFO.

 ps -ef |grep zoo|awk '{print $10}'
-Dzookeeper.root.logger=INFO,CONSOLE

Just to clarify, i am upgrading from 3.4.5 to 3.4.14 and i am upgrading
observers firs then followers (yet to do) and see this flood of log
messages in observer logs.

Observer: 3.4.14 version
Leader: 3.4.5 version

On Thu, Dec 12, 2019 at 9:13 AM Flavio Junqueira  wrote:

> If you set up logging at DEBUG level and you see this message:
>"Not initializing connection executor as quorum sasl auth is disabled"
> then you are good.
>
> -Flavio
>
> > On 12 Dec 2019, at 18:06, rammohan ganapavarapu 
> wrote:
> >
> > I am assuming its only logging, is there any way i can verify its not
> > opening cnxn threads?
> >
> >
> >
> > On Thu, Dec 12, 2019 at 7:15 AM Flavio Junqueira  wrote:
> >
> >> If it is just logging the value of the property, then it is fine because
> >> the property is set, but it is not used if SASL is disabled.
> >>
> >> -Flavio
> >>
> >>> On 12 Dec 2019, at 16:01, rammohan ganapavarapu <
> rammohanga...@gmail.com>
> >> wrote:
> >>>
> >>> Flavio,
> >>>
> >>> Thanks for clarification, I have a cluster with sasl disabled but in
> >> start
> >>> up log I still see number of threads set to default 20.
> >>>
> >>> Thanks,
> >>> Ram
> >>> On Thu, Dec 12, 2019, 6:43 AM Flavio Junqueira  wrote:
> >>>
> >>>> Hi Ram,
> >>>>
> >>>> Yes, this parameter is only applicable when SASL is enabled, see in
> >>>> QuorumCnxManager#initializeAuth how it doesn't have a chance to use
> the
> >>>> parameter when SASL is disabled.
> >>>>
> >>>> Your assessment is correct about it being based on the number of
> >>>> participants, excluding observers. This parameter is used in
> >>>> QuorumCnxManager, which manages connections for leader election, and
> >>>> observers do not participate in the election.
> >>>>
> >>>> -Flavio
> >>>>
> >>>>> On 12 Dec 2019, at 14:41, rammohan ganapavarapu <
> >> rammohanga...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Any idea?
> >>>>>
> >>>>> On Wed, Dec 11, 2019, 2:23 PM rammohan ganapavarapu <
> >>>> rammohanga...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> And to add more on the same property, the description says
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *"For example, consider a 3-node cluster,# during quorum formation
> at
> >>>>>> least 3 outgoing connection requests and 3 incoming connection#
> >> requests
> >>>>>> will occur. So total 6 threads will be used. It is recommended to
> >>>> configure
> >>>>>> 2x# number of threads for smooth execution, where 'x' represents the
> >>>>>> cluster size"*
> >>>>>>
> >>>>>> Here* cluster size* is number of *followers* right not both
> followers
> >>>> and
> >>>>>> observers??
> >>>>>>
> >>>>>> Ram
> >>>>>>
> >>>>>> On Wed, Dec 11, 2019 at 2:19 PM rammohan ganapavarapu <
> >>>>>> rammohanga...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Is this property only applicable in case of sasl auth enabled? or
> >> does
> >>>> it
> >>>>>>> work even when we turn off authentication?
> >>>>>>>
> >>>>>>>
> >>>>>>> *quorum.cnxn.threads.size*
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Ram
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: Notification messages in log

[rammohan ganapavarapu]

Any thoughts?

On Wed, Dec 11, 2019 at 7:48 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi Enrico,
>
> Since i have log  rotation, i am not worried about size but i see ~14 such
> messages for a second which seems unusual for me.
>
> grep "Notification: 0 (message format version)"  zookeeper.log|awk '{print
> $1" "$2}'|awk -F "," '{print $1}'|sort |uniq -c
>   9 2019-12-12 03:11:14
>  14 2019-12-12 03:11:15
>  13 2019-12-12 03:11:16
>  14 2019-12-12 03:11:17
>  14 2019-12-12 03:11:18
>  14 2019-12-12 03:11:19
>  14 2019-12-12 03:11:20
>  13 2019-12-12 03:11:21
>  14 2019-12-12 03:11:22
>  14 2019-12-12 03:11:23
>  14 2019-12-12 03:11:24
>  14 2019-12-12 03:11:25
>  13 2019-12-12 03:11:26
>  14 2019-12-12 03:11:27
>  14 2019-12-12 03:11:28
>  14 2019-12-12 03:11:29
>  14 2019-12-12 03:11:30
>  13 2019-12-12 03:11:31
>  14 2019-12-12 03:11:32
>  14 2019-12-12 03:11:33
>  14 2019-12-12 03:11:34
>  14 2019-12-12 03:11:35
>
> FYI: I  am in upgrade processes where leader is still on 3.4.5 version and
> this observer is  3.4.14 version does it cause such behavior?
>
> Thanks,
> Ram
>
> On Wed, Dec 11, 2019 at 3:22 PM Enrico Olivelli 
> wrote:
>
>> Ram,
>> How many GB of logs do you have?
>> Those messages are not scary but they shouldn't be so many.
>>
>> Enrico
>>
>>
>> Il mer 11 dic 2019, 23:16 rammohan ganapavarapu 
>> ha scritto:
>>
>> > After 3.4.14 version upgrade i am seeing continuous messages as bellow
>> in
>> > logs, is it normal?
>> >
>> >
>> > 2019-12-11 22:12:44,036 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,108 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,180 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,252 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,323 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,395 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,467 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,538 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,610 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,682 [myid:104] - INFO
>> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
>> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
>> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
>> > 2019-12-11 22:12:44,754 [myid:104

Re: quorum.cnxn.threads.size

[rammohan ganapavarapu]

I am assuming its only logging, is there any way i can verify its not
opening cnxn threads?



On Thu, Dec 12, 2019 at 7:15 AM Flavio Junqueira  wrote:

> If it is just logging the value of the property, then it is fine because
> the property is set, but it is not used if SASL is disabled.
>
> -Flavio
>
> > On 12 Dec 2019, at 16:01, rammohan ganapavarapu 
> wrote:
> >
> > Flavio,
> >
> > Thanks for clarification, I have a cluster with sasl disabled but in
> start
> > up log I still see number of threads set to default 20.
> >
> > Thanks,
> > Ram
> > On Thu, Dec 12, 2019, 6:43 AM Flavio Junqueira  wrote:
> >
> >> Hi Ram,
> >>
> >> Yes, this parameter is only applicable when SASL is enabled, see in
> >> QuorumCnxManager#initializeAuth how it doesn't have a chance to use the
> >> parameter when SASL is disabled.
> >>
> >> Your assessment is correct about it being based on the number of
> >> participants, excluding observers. This parameter is used in
> >> QuorumCnxManager, which manages connections for leader election, and
> >> observers do not participate in the election.
> >>
> >> -Flavio
> >>
> >>> On 12 Dec 2019, at 14:41, rammohan ganapavarapu <
> rammohanga...@gmail.com>
> >> wrote:
> >>>
> >>> Any idea?
> >>>
> >>> On Wed, Dec 11, 2019, 2:23 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com>
> >>> wrote:
> >>>
> >>>> And to add more on the same property, the description says
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> *"For example, consider a 3-node cluster,# during quorum formation at
> >>>> least 3 outgoing connection requests and 3 incoming connection#
> requests
> >>>> will occur. So total 6 threads will be used. It is recommended to
> >> configure
> >>>> 2x# number of threads for smooth execution, where 'x' represents the
> >>>> cluster size"*
> >>>>
> >>>> Here* cluster size* is number of *followers* right not both followers
> >> and
> >>>> observers??
> >>>>
> >>>> Ram
> >>>>
> >>>> On Wed, Dec 11, 2019 at 2:19 PM rammohan ganapavarapu <
> >>>> rammohanga...@gmail.com> wrote:
> >>>>
> >>>>> Is this property only applicable in case of sasl auth enabled? or
> does
> >> it
> >>>>> work even when we turn off authentication?
> >>>>>
> >>>>>
> >>>>> *quorum.cnxn.threads.size*
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
> >>
> >>
>
>

Re: quorum.cnxn.threads.size

[rammohan ganapavarapu]

Flavio,

Thanks for clarification, I have a cluster with sasl disabled but in start
up log I still see number of threads set to default 20.

Thanks,
Ram
On Thu, Dec 12, 2019, 6:43 AM Flavio Junqueira  wrote:

> Hi Ram,
>
> Yes, this parameter is only applicable when SASL is enabled, see in
> QuorumCnxManager#initializeAuth how it doesn't have a chance to use the
> parameter when SASL is disabled.
>
> Your assessment is correct about it being based on the number of
> participants, excluding observers. This parameter is used in
> QuorumCnxManager, which manages connections for leader election, and
> observers do not participate in the election.
>
> -Flavio
>
> > On 12 Dec 2019, at 14:41, rammohan ganapavarapu 
> wrote:
> >
> > Any idea?
> >
> > On Wed, Dec 11, 2019, 2:23 PM rammohan ganapavarapu <
> rammohanga...@gmail.com>
> > wrote:
> >
> >> And to add more on the same property, the description says
> >>
> >>
> >>
> >>
> >>
> >> *"For example, consider a 3-node cluster,# during quorum formation at
> >> least 3 outgoing connection requests and 3 incoming connection# requests
> >> will occur. So total 6 threads will be used. It is recommended to
> configure
> >> 2x# number of threads for smooth execution, where 'x' represents the
> >> cluster size"*
> >>
> >> Here* cluster size* is number of *followers* right not both followers
> and
> >> observers??
> >>
> >> Ram
> >>
> >> On Wed, Dec 11, 2019 at 2:19 PM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Is this property only applicable in case of sasl auth enabled? or does
> it
> >>> work even when we turn off authentication?
> >>>
> >>>
> >>> *quorum.cnxn.threads.size*
> >>>
> >>> Thanks,
> >>> Ram
> >>>
> >>
>
>

Re: quorum.cnxn.threads.size

[rammohan ganapavarapu]

Any idea?

On Wed, Dec 11, 2019, 2:23 PM rammohan ganapavarapu 
wrote:

> And to add more on the same property, the description says
>
>
>
>
>
> *"For example, consider a 3-node cluster,# during quorum formation at
> least 3 outgoing connection requests and 3 incoming connection# requests
> will occur. So total 6 threads will be used. It is recommended to configure
> 2x# number of threads for smooth execution, where 'x' represents the
> cluster size"*
>
> Here* cluster size* is number of *followers* right not both followers and
> observers??
>
> Ram
>
> On Wed, Dec 11, 2019 at 2:19 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Is this property only applicable in case of sasl auth enabled? or does it
>> work even when we turn off authentication?
>>
>>
>> *quorum.cnxn.threads.size*
>>
>> Thanks,
>> Ram
>>
>

Re: Notification messages in log

[rammohan ganapavarapu]

Hi Enrico,

Since i have log  rotation, i am not worried about size but i see ~14 such
messages for a second which seems unusual for me.

grep "Notification: 0 (message format version)"  zookeeper.log|awk '{print
$1" "$2}'|awk -F "," '{print $1}'|sort |uniq -c
  9 2019-12-12 03:11:14
 14 2019-12-12 03:11:15
 13 2019-12-12 03:11:16
 14 2019-12-12 03:11:17
 14 2019-12-12 03:11:18
 14 2019-12-12 03:11:19
 14 2019-12-12 03:11:20
 13 2019-12-12 03:11:21
 14 2019-12-12 03:11:22
 14 2019-12-12 03:11:23
 14 2019-12-12 03:11:24
 14 2019-12-12 03:11:25
 13 2019-12-12 03:11:26
 14 2019-12-12 03:11:27
 14 2019-12-12 03:11:28
 14 2019-12-12 03:11:29
 14 2019-12-12 03:11:30
 13 2019-12-12 03:11:31
 14 2019-12-12 03:11:32
 14 2019-12-12 03:11:33
 14 2019-12-12 03:11:34
 14 2019-12-12 03:11:35

FYI: I  am in upgrade processes where leader is still on 3.4.5 version and
this observer is  3.4.14 version does it cause such behavior?

Thanks,
Ram

On Wed, Dec 11, 2019 at 3:22 PM Enrico Olivelli  wrote:

> Ram,
> How many GB of logs do you have?
> Those messages are not scary but they shouldn't be so many.
>
> Enrico
>
>
> Il mer 11 dic 2019, 23:16 rammohan ganapavarapu 
> ha scritto:
>
> > After 3.4.14 version upgrade i am seeing continuous messages as bellow in
> > logs, is it normal?
> >
> >
> > 2019-12-11 22:12:44,036 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,108 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,180 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,252 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,323 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,395 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,467 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,538 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,610 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,682 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,754 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
> > 2019-12-11 22:12:44,825 [myid:104] - INFO
> >  [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
> > (message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
> > LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (

Re: quorum.cnxn.threads.size

[rammohan ganapavarapu]

And to add more on the same property, the description says





*"For example, consider a 3-node cluster,# during quorum formation at least
3 outgoing connection requests and 3 incoming connection# requests will
occur. So total 6 threads will be used. It is recommended to configure 2x#
number of threads for smooth execution, where 'x' represents the cluster
size"*

Here* cluster size* is number of *followers* right not both followers and
observers??

Ram

On Wed, Dec 11, 2019 at 2:19 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Is this property only applicable in case of sasl auth enabled? or does it
> work even when we turn off authentication?
>
>
> *quorum.cnxn.threads.size*
>
> Thanks,
> Ram
>

quorum.cnxn.threads.size

[rammohan ganapavarapu]

Is this property only applicable in case of sasl auth enabled? or does it
work even when we turn off authentication?


*quorum.cnxn.threads.size*

Thanks,
Ram

Notification messages in log

[rammohan ganapavarapu]

After 3.4.14 version upgrade i am seeing continuous messages as bellow in
logs, is it normal?


2019-12-11 22:12:44,036 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,108 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,180 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,252 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,323 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,395 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,467 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,538 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,610 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,682 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,754 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,825 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,897 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:44,969 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:45,041 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:45,112 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:45,184 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)
2019-12-11 22:12:45,256 [myid:104] - INFO
 [WorkerReceiver[myid=104]:FastLeaderElection@595] - Notification: 0
(message format version), 100 (n.leader), 0x0 (n.zxid), 0x1 (n.round),
LOOKING (n.state), 100 (n.sid), 0x0 (n.peerEpoch) OBSERVING (my state)


But status looks ok though.

echo mntr |nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on
03/06/2019 16:18 GMT
zk_avg_latency 75
zk_max_latency 76
zk_min_latency 75
zk_packets_received 907
zk_packets_sent 906
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state observer
zk_znode_count 12168170

Re: default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

Mate,

Thank you, I did search source code found the same, I am trying to create a
zoo conf with all default properties.

Ram

On Fri, Dec 6, 2019, 2:44 AM Mate Szalay-Beko 
wrote:

> Hi Ram,
>
> this parameter is needed to be defined when you want to enable secure
> authentication in the communication between ZooKeeper servers. In general,
> the 'principal' is a 'username' what you want your ZooKeeper servers to use
> when they talk with each other. Ideally you have a central Kereros service
> somewhere where this principal is already registered.
> A kerberos principal is usually in the form of
> "user_or_service_name/host@realm" (some more explanation:
> https://ssimo.org/blog/id_016.html)
>
> According to the source code, the default value of
> quorum.auth.kerberos.servicePrincipal is "zkquorum/localhost". But I think
> if you don't enable the quorum SASL in ZooKeeper, then this property will
> never be actually used.
>
> Please see this page about SASL in ZooKeeper:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL
>
> I also found a Cloudera blogpost on the topic:
>
> https://blog.cloudera.com/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>
> Cheers,
> Mate
>
>
> On Thu, Dec 5, 2019 at 11:50 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > What is the default value for this property, if i don't  enable sasl and
> if
> > i don't define what will be the value?
> >
> > quorum.auth.kerberos.servicePrincipal
> >
> > Also what does this means "servicename/_HOST"
> >
> > Thanks,
> > Ram
> >
>

default value for quorum.auth.kerberos.servicePrincipal

[rammohan ganapavarapu]

Hi,

What is the default value for this property, if i don't  enable sasl and if
i don't define what will be the value?

quorum.auth.kerberos.servicePrincipal

Also what does this means "servicename/_HOST"

Thanks,
Ram

Re: Is there any tool to verify zookeeper snapshot file?

[rammohan ganapavarapu]

Thank you Andor,yes would be nice to have.

On Thu, Oct 31, 2019, 2:54 AM Andor Molnar  wrote:

> Try SnapshotFormatter:
>
>
> https://stackoverflow.com/questions/17894808/how-do-one-read-the-zookeeper-transaction-log
>
> We might need to have a SnapshotToolkit tool to verify snapshot files
> similar to TxnLogToolkit.
>
> Andor
>
>
>
> > On 2019. Oct 30., at 23:48, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > There are cases where zk will fail to start with invalid snapshot if
> > snapshot  file is corrupt or incomplete, i wanted to verify if the
> snapshot
> > is not corrupt or complete before restore.
> >
> > Ram
> >
> > On Wed, Oct 30, 2019 at 2:25 PM Enrico Olivelli 
> wrote:
> >
> >> Ram
> >>
> >> Il mer 30 ott 2019, 21:23 rammohan ganapavarapu <
> rammohanga...@gmail.com>
> >> ha scritto:
> >>
> >>> Hi,
> >>>
> >>> I am trying to see is there any tool available to verify the zk
> snapshot
> >>> file, any such thing exist?
> >>>
> >>
> >> What do you mean with 'verify'? To inspect the contents?
> >>
> >> Enrico
> >>
> >>>
> >>> Thanks,
> >>> Ram
> >>>
> >>
>
>

Re: Is there any tool to verify zookeeper snapshot file?

[rammohan ganapavarapu]

There are cases where zk will fail to start with invalid snapshot if
snapshot  file is corrupt or incomplete, i wanted to verify if the snapshot
is not corrupt or complete before restore.

Ram

On Wed, Oct 30, 2019 at 2:25 PM Enrico Olivelli  wrote:

> Ram
>
> Il mer 30 ott 2019, 21:23 rammohan ganapavarapu 
> ha scritto:
>
> > Hi,
> >
> > I am trying to see is there any tool available to verify the zk snapshot
> > file, any such thing exist?
> >
>
> What do you mean with 'verify'? To inspect the contents?
>
> Enrico
>
> >
> > Thanks,
> > Ram
> >
>

Is there any tool to verify zookeeper snapshot file?

[rammohan ganapavarapu]

Hi,

I am trying to see is there any tool available to verify the zk snapshot
file, any such thing exist?

Thanks,
Ram

Re: Zookeeper latency calculation

[rammohan ganapavarapu]

I do have a busy system total packet processed by zk is like 60k/s (30k
send and 30k received) and how about the max latencies? is ti always
constant ? for me avg latency is 0 but max latency is showing 40s how is it
possible?




On Wed, Jul 17, 2019 at 8:46 PM Michael Han  wrote:

> Typing send too soon...
>
> >> always give avg_latency "0"
>
> The latency metrics depends on workloads. Try hit your cluster hard with
> some artificially generated heavy read / write workloads, you will see the
> number deviates from 0.
>
> >> If I recall correctly avg_latency is an int, not float
>
> This is now a float on master branch, and the change was made in
> ZOOKEEPER-2641. I remember this because this actually breaks one of our
> internal metrics system, where our system expects an int (the old type).
>
> On Wed, Jul 17, 2019 at 8:43 PM Michael Han  wrote:
>
> > >> always give avg_latency "0"
> >
> > The latency metrics depends on workloads.
> >
> > On Wed, Jul 17, 2019 at 1:34 AM Enrico Olivelli 
> > wrote:
> >
> >> Il mar 16 lug 2019, 19:05 rammohan ganapavarapu <
> rammohanga...@gmail.com>
> >> ha scritto:
> >>
> >> > Hi,
> >> >
> >> > I am trying to understand how zookeeper latency calculated, mntr
> command
> >> > always give avg_latency "0", can some one help how to calculate avg
> >> request
> >> > latency in zookeeper?
> >> >
> >>
> >>
> >> Are you also taking metrics on the client?
> >> Maybe on the server side the value is biased by very fast ops
> >>
> >>
> >> Enrico
> >>
> >> >
> >> >
> >> > Thanks,
> >> > Ram
> >> >
> >>
> >
>

Zookeeper latency calculation

[rammohan ganapavarapu]

Hi,

I am trying to understand how zookeeper latency calculated, mntr command
always give avg_latency "0", can some one help how to calculate avg request
latency in zookeeper?


Thanks,
Ram

Re: ACL implimentation

[rammohan ganapavarapu]

Can you guys share your learnings or exp so that i dont have to go through
that pain if i want to enable ACL ?

Ram

On Mon, Jun 10, 2019 at 2:00 PM Andor Molnár  wrote:

> Agreed. I had to dig a bunch of Hortonworks / Stackoverflow docs to
> learn how ACLs work.
>
>
> Andor
>
>
>
> On 2019. 06. 09. 17:03, Patrick Hunt wrote:
> > I had to deal with some ACL issues myself recently and noticed the lack
> of
> > docs we have, both generally and best practices. I spent a bunch of time
> > when testing the recent ACL changes from Andor just re-learning the shell
> > commands and config necessary to exercise the patches. This would be a
> > great area for contributions.
> >
> > Patrick
> >
> > On Fri, Jun 7, 2019 at 8:04 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> >> Enrico,
> >>
> >> Thank you.
> >>
> >> Ram
> >>
> >> On Fri, Jun 7, 2019 at 5:30 AM Enrico Olivelli 
> >> wrote:
> >>
> >>> Ram
> >>> Can you describe better your problem ?
> >>> Usually you are activating auth on clients and then you apply the ACLs
> >> this
> >>> way all clients will be able to access data.
> >>>
> >>> Try our procedure in a staging environment before doing in in
> production
> >>>
> >>> Enrico
> >>>
> >>> Il gio 6 giu 2019, 23:56 rammohan ganapavarapu <
> rammohanga...@gmail.com>
> >>> ha
> >>> scritto:
> >>>
> >>>> Hi,
> >>>>
> >>>> Is there any recommendations or best practices on implementing ACL on
> >>>> existing zookeeper cluster with production data with out downtime?
> >>>>
> >>>> Thanks,
> >>>> Ram
> >>>>
>

Re: Zookeeper client connection

[rammohan ganapavarapu]

All i want to do is avoid static IP as we have zk cluster in autoscalling
environment and IP is ephemeral there.

On Fri, Jun 7, 2019, 2:38 PM Enrico Olivelli  wrote:

> A round robin DNS name should work
>
>
> Enrico
>
> Il ven 7 giu 2019, 22:39 rammohan ganapavarapu 
> ha
> scritto:
>
> > Hi,
> >
> > Since with 3.5.5 version clients can watch on zoookeeper/confg znode for
> > the active zk servers, do we still need to give all the IPs of zk servers
> > to client? or can i give one IP or rr dns name to a client and client can
> > get  server's list from the config znode to balance the connections?
> >
> > Thanks,
> > Ram
> >
>

Zookeeper client connection

[rammohan ganapavarapu]

Hi,

Since with 3.5.5 version clients can watch on zoookeeper/confg znode for
the active zk servers, do we still need to give all the IPs of zk servers
to client? or can i give one IP or rr dns name to a client and client can
get  server's list from the config znode to balance the connections?

Thanks,
Ram

Re: ACL implimentation

[rammohan ganapavarapu]

Enrico,

Thank you.

Ram

On Fri, Jun 7, 2019 at 5:30 AM Enrico Olivelli  wrote:

> Ram
> Can you describe better your problem ?
> Usually you are activating auth on clients and then you apply the ACLs this
> way all clients will be able to access data.
>
> Try our procedure in a staging environment before doing in in production
>
> Enrico
>
> Il gio 6 giu 2019, 23:56 rammohan ganapavarapu 
> ha
> scritto:
>
> > Hi,
> >
> > Is there any recommendations or best practices on implementing ACL on
> > existing zookeeper cluster with production data with out downtime?
> >
> > Thanks,
> > Ram
> >
>

ACL implimentation

[rammohan ganapavarapu]

Hi,

Is there any recommendations or best practices on implementing ACL on
existing zookeeper cluster with production data with out downtime?

Thanks,
Ram

Re: Dynamic Config

[rammohan ganapavarapu]

Hi Alex,

Thanks, that helps a lot.

Ram

On Thu, May 30, 2019 at 11:27 AM Alexander Shraer  wrote:

> Hi,
>
> 1. Right - only the configuration parameters that live in the dynamic file
> are controlled by dynamic reconfig. The dynamic files are
> kept in sync across all the ZK servers, whereas the static files may not be
> the same.
>
> There is a backward compatibility mode, where you start up a server without
> a dynamic file, and ZK copies over whatever
> it can from the static to the dynamic file. From that point, you're not
> supposed to manually change the dynamic file - ZK
> manages that for you, and you only affect the configuration via reconfig
> commands.
>
> 2. Dynamic files are written out upon commit of new configurations created
> via reconfig, or, more precisely, when a server learns about such a commit.
> The number is the zxid of the commit.
>
> 3. I don't think there's any purge job that was implemented, so the old
> copies will
>
> 4. There is a fixed set of things that can live in the dynamic file. You
> can't just put anything there, because ZK still looks for other config
> parameters in the static file.
>
> Please see details in the manual:
> https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html
>
> Thanks,
> Alex
>
> On Thu, May 30, 2019 at 10:49 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > One more question
> >
> > 4. Is there any list of configs that should be only in static file to
> boot
> > up the zookeeper? or can i have some thing like this?
> >
> > cat zoo.cfg
> > dynamicConfigFile=/opt/zookeeper/conf/dynamic.cfg
> >
> > cat dynamic.cfg
> > # All zookeeper configurations
> >
> > will this work?
> >
> > On Thu, May 30, 2019 at 9:59 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have  few questions regarding dynamic reconfig feature,
> > >
> > > 1. this feature can only reconfigure the properties or configuration
> > > defined in dynamic configuration file and not the configs in static
> > default
> > > zoo.cfg file?
> > > 2. What is the criteria to create version extension for dynamic config
> > > file? ex: zoo.cfg.dynamic.1, i mean when does zk create new
> > version
> > > file, i have change a property in static file and restarted zk but it
> > didnt
> > > create new version file so it will only create new version when a
> config
> > in
> > > dynamic file changes?
> > > 3. How many copies/versions of these dynamic config files will get
> > created
> > > and is there a purge task that zk runs to cleanup older version files?
> > >
> > >
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: Dynamic Config

[rammohan ganapavarapu]

Hi,

One more question

4. Is there any list of configs that should be only in static file to boot
up the zookeeper? or can i have some thing like this?

cat zoo.cfg
dynamicConfigFile=/opt/zookeeper/conf/dynamic.cfg

cat dynamic.cfg
# All zookeeper configurations

will this work?

On Thu, May 30, 2019 at 9:59 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> I have  few questions regarding dynamic reconfig feature,
>
> 1. this feature can only reconfigure the properties or configuration
> defined in dynamic configuration file and not the configs in static default
> zoo.cfg file?
> 2. What is the criteria to create version extension for dynamic config
> file? ex: zoo.cfg.dynamic.1, i mean when does zk create new version
> file, i have change a property in static file and restarted zk but it didnt
> create new version file so it will only create new version when a config in
> dynamic file changes?
> 3. How many copies/versions of these dynamic config files will get created
> and is there a purge task that zk runs to cleanup older version files?
>
>
>
> Thanks,
> Ram
>

Dynamic Config

[rammohan ganapavarapu]

Hi,

I have  few questions regarding dynamic reconfig feature,

1. this feature can only reconfigure the properties or configuration
defined in dynamic configuration file and not the configs in static default
zoo.cfg file?
2. What is the criteria to create version extension for dynamic config
file? ex: zoo.cfg.dynamic.1, i mean when does zk create new version
file, i have change a property in static file and restarted zk but it didnt
create new version file so it will only create new version when a config in
dynamic file changes?
3. How many copies/versions of these dynamic config files will get created
and is there a purge task that zk runs to cleanup older version files?



Thanks,
Ram

Re: Admin Server Authentication/SSL

[rammohan ganapavarapu]

Hi,

let me re-phrase my question, Is there any way to disable non-ssl
connections for adminserver? i want admin server accept only ssl
connections.

Ram

On Tue, May 28, 2019 at 11:40 AM Martin Gainty  wrote:

> for 3.5..5 since the default security protocol is TLSv1.2 :
> ssl.protocol and ssl.quorum.protocol : (Java system properties:
> zookeeper.ssl.protocol and zookeeper.ssl.quorum.protocol) New in 3.5.5:
> Specifies to protocol to be used in client and quorum TLS negotiation.
> Default: TLSv1.2
>
> any reason (other than political) why you cannot implement TLS (v1.2)
> instead of SSLv3 for 3.5.5 ?
>
> https://zookeeper.apache.org/doc/current/zookeeperAdmin.html
> ZooKeeper: Because Coordinating Distributed Systems is a Zoo - Apache
> ZooKeeper - Home<
> https://zookeeper.apache.org/doc/current/zookeeperAdmin.html>
> For any operating system not explicitly mentioned as supported in the
> matrix, components may or may not work. The ZooKeeper community will fix
> obvious bugs that are reported for other platforms, but there is no full
> support.
> zookeeper.apache.org
>
> ?
>
> 
> From: rammohan ganapavarapu 
> Sent: Tuesday, May 28, 2019 12:56 PM
> To: user@zookeeper.apache.org
> Subject: Admin Server Authentication/SSL
>
> Hi,
>
> Does zk AdminServer in  3.5 support authentication and SSL?
>
> Thanks,
> Ram
>

Admin Server Authentication/SSL

[rammohan ganapavarapu]

Hi,

Does zk AdminServer in  3.5 support authentication and SSL?

Thanks,
Ram

Re: Updated SSL guide fro 3.5.5

[rammohan ganapavarapu]

Hi,

I did find doc for quorum tls
https://zookeeper.apache.org/doc/r3.5.5/zookeeperAdmin.html#Quorum+TLS but
looking for client - server tls (for existing client server upgrade)

Ram

On Wed, May 22, 2019 at 1:54 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> Since 3.5.5 is out is there any updated guide to configure SSL both for
> server-server and client-server?
>
> Thanks,
> Ram
>

Updated SSL guide fro 3.5.5

[rammohan ganapavarapu]

Hi,

Since 3.5.5 is out is there any updated guide to configure SSL both for
server-server and client-server?

Thanks,
Ram

Re: [ANNOUNCE] Apache ZooKeeper 3.5.5

[rammohan ganapavarapu]

Hi Jordan,

Thank you, will do let you know.

Ram

On Tue, May 21, 2019 at 8:58 AM Jordan Zimmerman 
wrote:

> AFAIK, the version of ZooKeeper 3.5.5 being released is fully compatible
> with Curator. Please let us know if you find any issues.
>
> -Jordan
>
> > On May 21, 2019, at 10:44 AM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > I am curious to know what are the compatible "Apache Curator" client
> > version for 3.5.5 version.
> > Also is there any upgrade path from 3.4.5 to 3.5.5. Can i add a 3.5.5
> > follower/observer for a 3.4.5 leader and vice versa?
> >
> > Thanks,
> > Ram
> >
> > On Tue, May 21, 2019 at 1:34 AM Tamas Penzes  >
> > wrote:
> >
> >> Congratulations!
> >>
> >> We waited for this release for a really long time. I'm looking forward
> to
> >> use it.
> >> ZooKeeper arrived to a new Era.
> >>
> >> Regards, Tamaas
> >>
> >> On Tue, May 21, 2019 at 4:18 AM Zili Chen  wrote:
> >>
> >>> Congratulations!
> >>>
> >>> rammohan ganapavarapu  于2019年5月21日周二
> 上午7:25写道：
> >>>
> >>>> Congratulations, finally it's out 😊
> >>>>
> >>>> On Mon, May 20, 2019, 11:59 AM Enrico Olivelli 
> >>>> wrote:
> >>>>
> >>>>> Congratulations!
> >>>>>
> >>>>> Enrico
> >>>>>
> >>>>> Il lun 20 mag 2019, 19:28 Lars Francke  ha
> >>>>> scritto:
> >>>>>
> >>>>>> Congratulations on this release! It looks great and I'm looking
> >>> forward
> >>>>> to
> >>>>>> using all those new features.
> >>>>>>
> >>>>>> Thank you, everyone, for your work on this.
> >>>>>>
> >>>>>> On Mon, May 20, 2019 at 7:06 PM Andor Molnar 
> >>> wrote:
> >>>>>>
> >>>>>>> The Apache ZooKeeper team is proud to announce Apache ZooKeeper
> >>>> version
> >>>>>>> 3.5.5
> >>>>>>>
> >>>>>>> ZooKeeper is a high-performance coordination service for
> >>> distributed
> >>>>>>> applications. It exposes common services - such as naming,
> >>>>>>> configuration management, synchronization, and group services -
> >> in
> >>> a
> >>>>>>> simple interface so you don't have to write them from scratch.
> >> You
> >>>> can
> >>>>>>> use it off-the-shelf to implement consensus, group management,
> >>> leader
> >>>>>>> election, and presence protocols. And you can build on it for
> >> your
> >>>>>>> own, specific needs.
> >>>>>>>
> >>>>>>> For ZooKeeper release details and downloads, visit:
> >>>>>>> https://zookeeper.apache.org/releases.html
> >>>>>>>
> >>>>>>> ZooKeeper 3.5.5 Release Notes are at:
> >>>>>>> https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html
> >>>>>>>
> >>>>>>> We would like to thank the contributors that made the release
> >>>> possible.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> The ZooKeeper Team
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.5

[rammohan ganapavarapu]

I am curious to know what are the compatible "Apache Curator" client
version for 3.5.5 version.
Also is there any upgrade path from 3.4.5 to 3.5.5. Can i add a 3.5.5
follower/observer for a 3.4.5 leader and vice versa?

Thanks,
Ram

On Tue, May 21, 2019 at 1:34 AM Tamas Penzes 
wrote:

> Congratulations!
>
> We waited for this release for a really long time. I'm looking forward to
> use it.
> ZooKeeper arrived to a new Era.
>
> Regards, Tamaas
>
> On Tue, May 21, 2019 at 4:18 AM Zili Chen  wrote:
>
> > Congratulations!
> >
> > rammohan ganapavarapu  于2019年5月21日周二 上午7:25写道：
> >
> > > Congratulations, finally it's out 😊
> > >
> > > On Mon, May 20, 2019, 11:59 AM Enrico Olivelli 
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Enrico
> > > >
> > > > Il lun 20 mag 2019, 19:28 Lars Francke  ha
> > > > scritto:
> > > >
> > > > > Congratulations on this release! It looks great and I'm looking
> > forward
> > > > to
> > > > > using all those new features.
> > > > >
> > > > > Thank you, everyone, for your work on this.
> > > > >
> > > > > On Mon, May 20, 2019 at 7:06 PM Andor Molnar 
> > wrote:
> > > > >
> > > > > > The Apache ZooKeeper team is proud to announce Apache ZooKeeper
> > > version
> > > > > > 3.5.5
> > > > > >
> > > > > > ZooKeeper is a high-performance coordination service for
> > distributed
> > > > > > applications. It exposes common services - such as naming,
> > > > > > configuration management, synchronization, and group services -
> in
> > a
> > > > > > simple interface so you don't have to write them from scratch.
> You
> > > can
> > > > > > use it off-the-shelf to implement consensus, group management,
> > leader
> > > > > > election, and presence protocols. And you can build on it for
> your
> > > > > > own, specific needs.
> > > > > >
> > > > > > For ZooKeeper release details and downloads, visit:
> > > > > > https://zookeeper.apache.org/releases.html
> > > > > >
> > > > > > ZooKeeper 3.5.5 Release Notes are at:
> > > > > > https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html
> > > > > >
> > > > > > We would like to thank the contributors that made the release
> > > possible.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > The ZooKeeper Team
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Apache ZooKeeper 3.5.5

[rammohan ganapavarapu]

Congratulations, finally it's out 😊

On Mon, May 20, 2019, 11:59 AM Enrico Olivelli  wrote:

> Congratulations!
>
> Enrico
>
> Il lun 20 mag 2019, 19:28 Lars Francke  ha
> scritto:
>
> > Congratulations on this release! It looks great and I'm looking forward
> to
> > using all those new features.
> >
> > Thank you, everyone, for your work on this.
> >
> > On Mon, May 20, 2019 at 7:06 PM Andor Molnar  wrote:
> >
> > > The Apache ZooKeeper team is proud to announce Apache ZooKeeper version
> > > 3.5.5
> > >
> > > ZooKeeper is a high-performance coordination service for distributed
> > > applications. It exposes common services - such as naming,
> > > configuration management, synchronization, and group services - in a
> > > simple interface so you don't have to write them from scratch. You can
> > > use it off-the-shelf to implement consensus, group management, leader
> > > election, and presence protocols. And you can build on it for your
> > > own, specific needs.
> > >
> > > For ZooKeeper release details and downloads, visit:
> > > https://zookeeper.apache.org/releases.html
> > >
> > > ZooKeeper 3.5.5 Release Notes are at:
> > > https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html
> > >
> > > We would like to thank the contributors that made the release possible.
> > >
> > > Regards,
> > >
> > > The ZooKeeper Team
> > >
> > >
> > >
> >
>

SASL configuration

[rammohan ganapavarapu]

Hi,

I have a question regarding enabling server-server and client-server SASL
using DIGEST-MD5, from the zk wiki i see below for server-server
authentication

QuorumServer {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   user_test="test";
};

QuorumLearner {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="test"
   password="test";
};
And for client-server below config on server side

Server {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   user_super="adminsecret"
   user_bob="bobsecret";
};

So if i am enabling both server-server and client-server i need to have
both Server and QuorumServer contexts in jaas.conf file or can i just use
the same QuorumServer for both server-server and client-server setup?

Thanks,
Ram

Re: RR DNS name instead of list of server

[rammohan ganapavarapu]

ok, got it thank you!

On Tue, Feb 12, 2019 at 8:49 AM Andor Molnar 
wrote:

> Not sure what do you mean by 'static'?
> ZK instance cannot change myid, it's tight to the database.
>
> Andor
>
>
> On Tue, Feb 12, 2019 at 5:18 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Andor,
> >
> > Thanks you, do we have to have a static myid? any alternatives to it?
> >
> > Ram
> >
> > On Tue, Feb 12, 2019 at 3:44 AM Andor Molnar  >
> > wrote:
> >
> > > Hi Ram / Alan,
> > >
> > > I quite like the idea of implementing some kind of autoconfiguration
> for
> > > ZooKeeper, because currently it's entirely based on static config files
> > > which is not 100% cloud-friendly. Starting the project with an initial
> > > support for EC2 instances based on Alan's approach would be awesome.
> > > There's no concept of "seed nodes" in ZK, like Cassandra, e.g. neither
> > > clients nor servers are able to learn cluster topology from each other
> > > (that could be another improvement). In order to start a participant we
> > > have to provide "myid" (from instance tag), server IP addresses
> > > (autoscaling group), election and quorum port numbers and participant
> > type.
> > > Basically replacing the "server.X" section of the config.
> > >
> > > RR DNS might not be a good option, because as Alan mentioned the order
> of
> > > returning IPs is not guaranteed, so myid config would be cumbersome.
> > >
> > > Need to think about it more, but I believe it's definitely worth to
> > raise a
> > > Jira.
> > >
> > > Cool stuff.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Feb 11, 2019 at 5:48 PM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Jürgen,
> > > >
> > > > I have zk clusters in dynamic environment like Autoscalling groups
> and
> > as
> > > > you know in ASG it is quite common for a instance to get terminate
> and
> > > new
> > > > one comes up right, so in that case if i rely on static config it
> will
> > be
> > > > little bit hard to manage the cluster, i was thinking if we have RR
> DNS
> > > > name atleast i can update the DNS entry when new nodes comes up or
> old
> > > one
> > > > terminate. I have not played with dynamic config option yet but if
> that
> > > > solves the problem we see in dynamic environments i am good. And i am
> > not
> > > > comparing with consul but just pointing out the existing example.
> > > >
> > > >
> > > >
> > > > Alan,
> > > >
> > > > Yes i am looking for the similar solution.
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > > > On Mon, Feb 11, 2019 at 6:52 AM Alan Scherger <
> alan.scher...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Jürgen,
> > > > >
> > > > > My intent was to simply suggest a more programmatic means for
> dynamic
> > > > > configuration. In particular, the detecting of seed nodes and their
> > > > > appropriate id numbers. One might imagine provisioning 3 nodes with
> > > tags
> > > > > like:
> > > > >
> > > > > zk_cluster=thebestcluster
> > > > > zk_myid={1,2,3}
> > > > >
> > > > > and then in the zk configuration we might have:
> > > > >
> > > > > discovery=ec2Tags
> > > > > discovery.ec2Tags.tagCluster=zk_cluster
> > > > > discovery.ec2Tags.tagMyid=zk_myid
> > > > >
> > > > > This would allow a little code to parse the tags out of ec2 and
> build
> > > the
> > > > > seed node configurations.
> > > > >
> > > > > Similarly we could build and maintain a custom auth provider that
> > could
> > > > use
> > > > > the AWS Certificate Manager Private CA APIs or Hashicorp Vault PKI
> > APIs
> > > > to
> > > > > automatically create and fetch the appropriate certificates and
> > > > > configurations.
> > > > >
> > > > > To your point, the security of introducing autoconfiguration

Re: RR DNS name instead of list of server

[rammohan ganapavarapu]

Andor,

Thanks you, do we have to have a static myid? any alternatives to it?

Ram

On Tue, Feb 12, 2019 at 3:44 AM Andor Molnar 
wrote:

> Hi Ram / Alan,
>
> I quite like the idea of implementing some kind of autoconfiguration for
> ZooKeeper, because currently it's entirely based on static config files
> which is not 100% cloud-friendly. Starting the project with an initial
> support for EC2 instances based on Alan's approach would be awesome.
> There's no concept of "seed nodes" in ZK, like Cassandra, e.g. neither
> clients nor servers are able to learn cluster topology from each other
> (that could be another improvement). In order to start a participant we
> have to provide "myid" (from instance tag), server IP addresses
> (autoscaling group), election and quorum port numbers and participant type.
> Basically replacing the "server.X" section of the config.
>
> RR DNS might not be a good option, because as Alan mentioned the order of
> returning IPs is not guaranteed, so myid config would be cumbersome.
>
> Need to think about it more, but I believe it's definitely worth to raise a
> Jira.
>
> Cool stuff.
>
> Regards,
> Andor
>
>
>
>
>
> On Mon, Feb 11, 2019 at 5:48 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Jürgen,
> >
> > I have zk clusters in dynamic environment like Autoscalling groups and as
> > you know in ASG it is quite common for a instance to get terminate and
> new
> > one comes up right, so in that case if i rely on static config it will be
> > little bit hard to manage the cluster, i was thinking if we have RR DNS
> > name atleast i can update the DNS entry when new nodes comes up or old
> one
> > terminate. I have not played with dynamic config option yet but if that
> > solves the problem we see in dynamic environments i am good. And i am not
> > comparing with consul but just pointing out the existing example.
> >
> >
> >
> > Alan,
> >
> > Yes i am looking for the similar solution.
> >
> > Thanks,
> > Ram
> >
> > On Mon, Feb 11, 2019 at 6:52 AM Alan Scherger 
> > wrote:
> >
> > > Hey Jürgen,
> > >
> > > My intent was to simply suggest a more programmatic means for dynamic
> > > configuration. In particular, the detecting of seed nodes and their
> > > appropriate id numbers. One might imagine provisioning 3 nodes with
> tags
> > > like:
> > >
> > > zk_cluster=thebestcluster
> > > zk_myid={1,2,3}
> > >
> > > and then in the zk configuration we might have:
> > >
> > > discovery=ec2Tags
> > > discovery.ec2Tags.tagCluster=zk_cluster
> > > discovery.ec2Tags.tagMyid=zk_myid
> > >
> > > This would allow a little code to parse the tags out of ec2 and build
> the
> > > seed node configurations.
> > >
> > > Similarly we could build and maintain a custom auth provider that could
> > use
> > > the AWS Certificate Manager Private CA APIs or Hashicorp Vault PKI APIs
> > to
> > > automatically create and fetch the appropriate certificates and
> > > configurations.
> > >
> > > To your point, the security of introducing autoconfiguration of
> settings
> > > like these might not be appropriate for all folks or installations, but
> > > environments where things like instance level IAM exist help mitigate
> > some
> > > risk assuming the proper access controls have been put in place.
> >  > > rant :) >
> > >
> > > I believe it's the lack of autoconfiguration in Zookeeper that has led
> to
> > > the creation of tools like Exhibitor or other tools that have never
> been
> > > open sourced for one reason or another. The introduction of Dynamic
> > > Reconfiguration is quite great, but the 'Re' part might imply we still
> > have
> > > some initial work left to be done.
> > >
> > > I'm also not sure how a RR DNS record mechanism would satisfy the id
> > > assignment requirement since typically the value of such a record is
> that
> > > the results never return in a guaranteed order. Historically, I've seen
> > > tools, Netflix's Eureka, over come such a challenge by use of TXT
> records
> > > instead. One might argue an SRV record with appropriate priority values
> > > could also ensure the ordering. However, personally none of this is
> > > particularly my cup of tea, but I do enjoy coming across and existence
> of
> > > such sy

Re: RR DNS name instead of list of server

[rammohan ganapavarapu]

Jürgen,

I have zk clusters in dynamic environment like Autoscalling groups and as
you know in ASG it is quite common for a instance to get terminate and new
one comes up right, so in that case if i rely on static config it will be
little bit hard to manage the cluster, i was thinking if we have RR DNS
name atleast i can update the DNS entry when new nodes comes up or old one
terminate. I have not played with dynamic config option yet but if that
solves the problem we see in dynamic environments i am good. And i am not
comparing with consul but just pointing out the existing example.



Alan,

Yes i am looking for the similar solution.

Thanks,
Ram

On Mon, Feb 11, 2019 at 6:52 AM Alan Scherger 
wrote:

> Hey Jürgen,
>
> My intent was to simply suggest a more programmatic means for dynamic
> configuration. In particular, the detecting of seed nodes and their
> appropriate id numbers. One might imagine provisioning 3 nodes with tags
> like:
>
> zk_cluster=thebestcluster
> zk_myid={1,2,3}
>
> and then in the zk configuration we might have:
>
> discovery=ec2Tags
> discovery.ec2Tags.tagCluster=zk_cluster
> discovery.ec2Tags.tagMyid=zk_myid
>
> This would allow a little code to parse the tags out of ec2 and build the
> seed node configurations.
>
> Similarly we could build and maintain a custom auth provider that could use
> the AWS Certificate Manager Private CA APIs or Hashicorp Vault PKI APIs to
> automatically create and fetch the appropriate certificates and
> configurations.
>
> To your point, the security of introducing autoconfiguration of settings
> like these might not be appropriate for all folks or installations, but
> environments where things like instance level IAM exist help mitigate some
> risk assuming the proper access controls have been put in place.  rant :) >
>
> I believe it's the lack of autoconfiguration in Zookeeper that has led to
> the creation of tools like Exhibitor or other tools that have never been
> open sourced for one reason or another. The introduction of Dynamic
> Reconfiguration is quite great, but the 'Re' part might imply we still have
> some initial work left to be done.
>
> I'm also not sure how a RR DNS record mechanism would satisfy the id
> assignment requirement since typically the value of such a record is that
> the results never return in a guaranteed order. Historically, I've seen
> tools, Netflix's Eureka, over come such a challenge by use of TXT records
> instead. One might argue an SRV record with appropriate priority values
> could also ensure the ordering. However, personally none of this is
> particularly my cup of tea, but I do enjoy coming across and existence of
> such systems.
>
> Hopefully this helps? I'm certainly not trying to advocate for busy work,
> extensive feature design, or russle any jimmies.
>
> Alan Scherger
>
>
>
> On Mon, Feb 11, 2019, 12:43 AM Jürgen Wagner (DVT)
> 
> > ...and come to think of it, there is another question. Cloud instance
> > tags are simply labels. There is no real semantics associated or
> > mandated by tags.
> >
> > In particular, there is no guarantee that a Zk instance is running on,
> > e.g., an EC2 instance labelled as "Zookeeper". Tags don't make services.
> >
> > If you want to use auto-scaling to create more Zk nodes and reconfigure
> > an existing cluster, the cluster will take care of discovering its
> > members, so only clients would be affected by the changes. They, however
> > could start with a well-known set of Zk nodes (e.g., the initial three),
> > inquire about the actual configuration, and subscribe to configuration
> > changes. There is no need for a tag- or DNS-based grouping this way.
> >
> > If you wanted to say, "hey, all of you Zk instances in my VPC, form a
> > cluster right now", you could do this indeed with tagging to seed the
> > server list initially. However, keep in mind that Zk is often a
> > security-relevant component and you don't really want ANY new Zk server
> > out there to be able to join your precious cluster - only the ones you
> > know about already.
> >
> > The fact that Consul may support something like it, doesn't mean it also
> > makes sense for Zookeeper. Consul and Zookeeper protocols and
> > architectures are quite different.
> >
> > I still don't understand what the precise requirement is that lead to
> > this question.
> >
> > I hope you'll enlighten me :-)
> >
> > Cheers,
> >
> > --Jürgen
> >
> >
> > On 11.02.2019 01:20, rammohan ganapavarapu wrote:
> >
> > > Should I open a feature request? For bot

Re: RR DNS name instead of list of server

[rammohan ganapavarapu]

Should I open a feature request? For both cloud auto discovery and use DNS
end point to form a quorum.

On Sun, Feb 10, 2019, 3:56 PM Alan Scherger  wrote:

> We might look at something like this: https://github.com/hekate-io/hekate
> for inspiration (or adoption). In the Golang community Hashicorp has built
> something similar: https://github.com/hashicorp/go-discover -- this
> problem
> set itself probably warrants a multilingual Apache project to help drive
> some standards and interoperability.
>
> Alan
>
> On Sun, Feb 10, 2019, 5:42 PM rammohan ganapavarapu <
> rammohanga...@gmail.com
> wrote:
>
> > Clod providers have api to query instance IP based in tags, actually
> consul
> > is doing that to form a cluster.
> >
> > On Sun, Feb 10, 2019, 11:40 AM Andor Molnar 
> > wrote:
> >
> > > Hi Ram!
> > >
> > > What exactly do you mean by "auto-discovery on cloud instance tags"?
> > > Is there a standard way of doing that?
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Sat, Feb 9, 2019 at 4:07 PM Norbert Kalmar
> >  > > >
> > > wrote:
> > >
> > > > Hi Ram,
> > > >
> > > > Unfortunately ZK does not support RR DNS name.
> > > > As for plans on discovery based on cloud tags, I am not aware of any
> > > plans.
> > > > You can create a jira for it if you'd like, but I can't tell you when
> > > that
> > > > would make it into a release.
> > > >
> > > > Regards,
> > > > Norbert
> > > >
> > > > On Fri, Feb 8, 2019 at 11:53 PM rammohan ganapavarapu <
> > > > rammohanga...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Does zookeper support RR DNS name in the config instead of giving
> > each
> > > > > server name/ip like what consul does to join the cluster?
> > > > >
> > > > >
> > > > > server.1=server1
> > > > > server.2=server2
> > > > > server.3=server3
> > > > >
> > > > > vs
> > > > > server=example.com
> > > > > where example.com is RR of server1, server2 and server3
> > > > >
> > > > > And does any one know if zk team has any plans to add cloud
> > > autodiscovery
> > > > > based on cloud instance tags?
> > > > >
> > > > > Thanks,
> > > > > Ram
> > > > >
> > > >
> > >
> >
>

Re: RR DNS name instead of list of server

[rammohan ganapavarapu]

Clod providers have api to query instance IP based in tags, actually consul
is doing that to form a cluster.

On Sun, Feb 10, 2019, 11:40 AM Andor Molnar 
wrote:

> Hi Ram!
>
> What exactly do you mean by "auto-discovery on cloud instance tags"?
> Is there a standard way of doing that?
>
> Regards,
> Andor
>
>
>
> On Sat, Feb 9, 2019 at 4:07 PM Norbert Kalmar  >
> wrote:
>
> > Hi Ram,
> >
> > Unfortunately ZK does not support RR DNS name.
> > As for plans on discovery based on cloud tags, I am not aware of any
> plans.
> > You can create a jira for it if you'd like, but I can't tell you when
> that
> > would make it into a release.
> >
> > Regards,
> > Norbert
> >
> > On Fri, Feb 8, 2019 at 11:53 PM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Does zookeper support RR DNS name in the config instead of giving each
> > > server name/ip like what consul does to join the cluster?
> > >
> > >
> > > server.1=server1
> > > server.2=server2
> > > server.3=server3
> > >
> > > vs
> > > server=example.com
> > > where example.com is RR of server1, server2 and server3
> > >
> > > And does any one know if zk team has any plans to add cloud
> autodiscovery
> > > based on cloud instance tags?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: Enable authentication per client

[rammohan ganapavarapu]

Thank you all.

On Sat, Feb 9, 2019, 9:56 AM Enrico Olivelli  wrote:

> Ram
> It depends on your use case.
> Auth is required mostly in conjunction with ACLs, so you can have client
> which perform auth and clients without auth.
> This is useful during the transition from no-ACL to an ACL based database.
>
> Hope that helps
> Enrico
>
> Il giorno sab 9 feb 2019, 16:13 Norbert Kalmar
> 
> ha scritto:
>
> > Hi Ram,
> >
> > ZooKeeper only knows IP addresses. You either require authentication from
> > all clients, or turn it off completely.
> > At least I couldn't think up anything that would achieve what you want.
> >
> > Regards,
> > Norbert
> >
> > On Sat, Feb 9, 2019 at 12:52 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Is it possible to enable authentication for specific clients in
> > zookeeper?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: Enable authentication per client

[rammohan ganapavarapu]

One more question, when I enable client server auth for the first time what
will happen to the existing connection that are established without
authentication?

On Sat, Feb 9, 2019, 9:56 AM Enrico Olivelli  wrote:

> Ram
> It depends on your use case.
> Auth is required mostly in conjunction with ACLs, so you can have client
> which perform auth and clients without auth.
> This is useful during the transition from no-ACL to an ACL based database.
>
> Hope that helps
> Enrico
>
> Il giorno sab 9 feb 2019, 16:13 Norbert Kalmar
> 
> ha scritto:
>
> > Hi Ram,
> >
> > ZooKeeper only knows IP addresses. You either require authentication from
> > all clients, or turn it off completely.
> > At least I couldn't think up anything that would achieve what you want.
> >
> > Regards,
> > Norbert
> >
> > On Sat, Feb 9, 2019 at 12:52 AM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Is it possible to enable authentication for specific clients in
> > zookeeper?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Enable authentication per client

[rammohan ganapavarapu]

Hi,

Is it possible to enable authentication for specific clients in zookeeper?

Thanks,
Ram

RR DNS name instead of list of server

[rammohan ganapavarapu]

Hi,

Does zookeper support RR DNS name in the config instead of giving each
server name/ip like what consul does to join the cluster?


server.1=server1
server.2=server2
server.3=server3

vs
server=example.com
where example.com is RR of server1, server2 and server3

And does any one know if zk team has any plans to add cloud autodiscovery
based on cloud instance tags?

Thanks,
Ram

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Hi,

After a long time i have tried this again, i have removed the observer type
but still failing.

Ram

On Sat, Sep 29, 2018 at 11:50 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> I will try number 1 and yes there is no such entry in host file.
>
> On Sat, Sep 29, 2018, 10:37 AM Rakesh Radhakrishnan 
> wrote:
>
>> OK, it looks to me some common networking related issue.
>>
>> 1) To confirm, can you remove the Observer type and simply try to join zk
>> server to quorum like participant?
>>
>> 2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1
>> line in /etc/hosts. Something like,
>>
>>127.0.0.1   node203ea localhost localhost.localdomain
>> localhost4
>> localhost4.localdomain4
>>::1 localhost localhost.localdomain localhost6
>> localhost6.localdomain6
>>
>> http://ccl.cse.nd.edu/operations/condor/hostname.shtml
>>
>> On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>> > Any thoughts on what could be the reason for observers not able to
>> connect
>> > to followers/leader?
>> >
>> > Ram
>> >
>> > On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu <
>> > rammohanga...@gmail.com> wrote:
>> >
>> >> Incase if you have not received my previous logs files.
>> >>
>> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
>> >> rammohanga...@gmail.com> wrote:
>> >>
>> >>> Rakesh,
>> >>>
>> >>> Thank you, i have 3 floower and 3 observers in two different DC's
>> >>> followers came up fine with SASL but for some reasons observers are
>> not
>> >>> coming up with the following error but i dont see any network issues,
>> i was
>> >>> able to telnet to 2181 and 3888 ports.
>> >>>
>> >>>
>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] -
>> Queue
>> >>> size: 1
>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] -
>> Queue
>> >>> size: 1
>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] -
>> Queue
>> >>> size: 1
>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] -
>> >>> Opening channel to server 1
>> >>> 2018-09-24 17:55:34,151 [myid:6] - WARN
>> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] -
>> Cannot
>> >>> open channel to 1 at election address zk-server1/10.16.1.102:3888
>> >>> java.net.SocketTimeoutException: connect timed out
>> >>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> >>> at
>> >>> java.net
>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> >>> at
>> >>> java.net
>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> >>> at
>> >>> java.net
>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> >>> at java.net.Socket.connect(Socket.java:589)
>> >>> at
>> >>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
>> >>> at
>> >>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
>> >>> at
>> >>>
>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
>> >>> at
>> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
>> >>>
>> >>>
>> >>> server.1=zk-server1:2888:3888
>> >>> server.2=zk-server2:2888:3888
>> >>> server.3=zk-server3:2888:3888
>> >>> server.4=zk-server4:2888:3888:observer
>> >>> server.5=zk-server5:2888:3888:observer
>> >>> server.6=zk-server6:2888:3888:observer
>> >>> peerType=observer
>> >>>
>> >>> What could be the reason?
>> >>>
>> >

Re: Zookeeper 3.5.X beta to stable versoin

[rammohan ganapavarapu]

I think lot of people wait for 3.5  as me :)

Ram

On Tue, Nov 20, 2018 at 6:22 AM Norbert Kalmar 
wrote:

> Hi Gopi,
>
> I (and pretty much everyone) can only give you estimates, unfortunately.
> So, my guess: January.
> But I am not the release manager, nor a "binding voter".
> I'm just also closely monitoring 3.5 stable release as well, and working on
> one of the blocking issues.
>
> Regards,
> Norbert
>
> On Tue, Nov 20, 2018 at 2:14 PM Testing Ideas 
> wrote:
>
> > Hi,
> >
> > Do you have a time of when we can expect the stable release of Zookeeper
> > 3.5? I got from your community that you are working on to release the
> > stable version ASAP, but a gross time would help us to communicate with
> my
> > stakeholders on the release and plan our migration.
> >
> > Many thanks,
> > Gopi
> >
>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

I will try number 1 and yes there is no such entry in host file.

On Sat, Sep 29, 2018, 10:37 AM Rakesh Radhakrishnan 
wrote:

> OK, it looks to me some common networking related issue.
>
> 1) To confirm, can you remove the Observer type and simply try to join zk
> server to quorum like participant?
>
> 2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1
> line in /etc/hosts. Something like,
>
>127.0.0.1   node203ea localhost localhost.localdomain localhost4
> localhost4.localdomain4
>::1 localhost localhost.localdomain localhost6
> localhost6.localdomain6
>
> http://ccl.cse.nd.edu/operations/condor/hostname.shtml
>
> On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Any thoughts on what could be the reason for observers not able to
> connect
> > to followers/leader?
> >
> > Ram
> >
> > On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> >> Incase if you have not received my previous logs files.
> >>
> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
> >> rammohanga...@gmail.com> wrote:
> >>
> >>> Rakesh,
> >>>
> >>> Thank you, i have 3 floower and 3 observers in two different DC's
> >>> followers came up fine with SASL but for some reasons observers are not
> >>> coming up with the following error but i dont see any network issues,
> i was
> >>> able to telnet to 2181 and 3888 ports.
> >>>
> >>>
> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> >>> size: 1
> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> >>> size: 1
> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> >>> size: 1
> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] -
> >>> Opening channel to server 1
> >>> 2018-09-24 17:55:34,151 [myid:6] - WARN
> >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] -
> Cannot
> >>> open channel to 1 at election address zk-server1/10.16.1.102:3888
> >>> java.net.SocketTimeoutException: connect timed out
> >>> at java.net.PlainSocketImpl.socketConnect(Native Method)
> >>> at
> >>> java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> >>> at
> >>> java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> >>> at
> >>> java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>> at java.net.Socket.connect(Socket.java:589)
> >>> at
> >>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
> >>> at
> >>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
> >>> at
> >>>
> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
> >>> at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
> >>>
> >>>
> >>> server.1=zk-server1:2888:3888
> >>> server.2=zk-server2:2888:3888
> >>> server.3=zk-server3:2888:3888
> >>> server.4=zk-server4:2888:3888:observer
> >>> server.5=zk-server5:2888:3888:observer
> >>> server.6=zk-server6:2888:3888:observer
> >>> peerType=observer
> >>>
> >>> What could be the reason?
> >>>
> >>> Ram
> >>>
> >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan <
> >>> rake...@apache.org> wrote:
> >>>
> >>>> Thanks Ram for the interest on this feature.
> >>>>
> >>>> Yes, user can enable SASL for Observer nodes as well. In general,
> >>>> QuorumLearner will send authentication packet to peer QuorumServer.
> >>>> Observer is a learner which follows the same quorum authentication
> protocol
> >>>> and auth logic will work fine.
> >>>>
> >>>> FYI, hope you are referring below links for configurations,
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
> >>>>
> >>>>
> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
> >>>>
> >>>> Please let us know if you are facing any issues.
> >>>>
> >>>> Thanks,
> >>>> Rakesh
> >>>>
> >>>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
> >>>> rammohanga...@gmail.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Do we need to configure any thing on observer nodes for SASL
> >>>>> authentication?
> >>>>>
> >>>>> tcpKeepAlive=true ( this is not for sasl but just asking )
> >>>>>
> >>>>> quorum.auth.enableSasl=true
> >>>>> quorum.auth.learnerRequireSasl=true
> >>>>> quorum.auth.serverRequireSasl=true
> >>>>>
> >>>>> What will happen if i set these properties on observers nodes as
> well ?
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Any thoughts on what could be the reason for observers not able to connect
to followers/leader?

Ram

On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Incase if you have not received my previous logs files.
>
> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Rakesh,
>>
>> Thank you, i have 3 floower and 3 observers in two different DC's
>> followers came up fine with SASL but for some reasons observers are not
>> coming up with the following error but i dont see any network issues, i was
>> able to telnet to 2181 and 3888 ports.
>>
>>
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening
>> channel to server 1
>> 2018-09-24 17:55:34,151 [myid:6] - WARN
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
>> open channel to 1 at election address zk-server1/10.16.1.102:3888
>> java.net.SocketTimeoutException: connect timed out
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
>> at
>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
>>
>>
>> server.1=zk-server1:2888:3888
>> server.2=zk-server2:2888:3888
>> server.3=zk-server3:2888:3888
>> server.4=zk-server4:2888:3888:observer
>> server.5=zk-server5:2888:3888:observer
>> server.6=zk-server6:2888:3888:observer
>> peerType=observer
>>
>> What could be the reason?
>>
>> Ram
>>
>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan 
>> wrote:
>>
>>> Thanks Ram for the interest on this feature.
>>>
>>> Yes, user can enable SASL for Observer nodes as well. In general,
>>> QuorumLearner will send authentication packet to peer QuorumServer.
>>> Observer is a learner which follows the same quorum authentication protocol
>>> and auth logic will work fine.
>>>
>>> FYI, hope you are referring below links for configurations,
>>>
>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>
>>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>>>
>>> Please let us know if you are facing any issues.
>>>
>>> Thanks,
>>> Rakesh
>>>
>>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
>>> rammohanga...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Do we need to configure any thing on observer nodes for SASL
>>>> authentication?
>>>>
>>>> tcpKeepAlive=true ( this is not for sasl but just asking )
>>>>
>>>> quorum.auth.enableSasl=true
>>>> quorum.auth.learnerRequireSasl=true
>>>> quorum.auth.serverRequireSasl=true
>>>>
>>>> What will happen if i set these properties on observers nodes as well ?
>>>>
>>>> Thanks,
>>>> Ram
>>>>
>>>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Any help?

On Tue, Sep 25, 2018 at 2:20 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> And observer never joining the cluster its keep saying  "Cannot open
> channel to"  in the logs.
>
> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Rakesh,
>>
>> Thank you, i have 3 floower and 3 observers in two different DC's
>> followers came up fine with SASL but for some reasons observers are not
>> coming up with the following error but i dont see any network issues, i was
>> able to telnet to 2181 and 3888 ports.
>>
>>
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>> size: 1
>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening
>> channel to server 1
>> 2018-09-24 17:55:34,151 [myid:6] - WARN
>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
>> open channel to 1 at election address zk-server1/10.16.1.102:3888
>> java.net.SocketTimeoutException: connect timed out
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
>> at
>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
>>
>>
>> server.1=zk-server1:2888:3888
>> server.2=zk-server2:2888:3888
>> server.3=zk-server3:2888:3888
>> server.4=zk-server4:2888:3888:observer
>> server.5=zk-server5:2888:3888:observer
>> server.6=zk-server6:2888:3888:observer
>> peerType=observer
>>
>> What could be the reason?
>>
>> Ram
>>
>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan 
>> wrote:
>>
>>> Thanks Ram for the interest on this feature.
>>>
>>> Yes, user can enable SASL for Observer nodes as well. In general,
>>> QuorumLearner will send authentication packet to peer QuorumServer.
>>> Observer is a learner which follows the same quorum authentication protocol
>>> and auth logic will work fine.
>>>
>>> FYI, hope you are referring below links for configurations,
>>>
>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>
>>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>>>
>>> Please let us know if you are facing any issues.
>>>
>>> Thanks,
>>> Rakesh
>>>
>>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
>>> rammohanga...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Do we need to configure any thing on observer nodes for SASL
>>>> authentication?
>>>>
>>>> tcpKeepAlive=true ( this is not for sasl but just asking )
>>>>
>>>> quorum.auth.enableSasl=true
>>>> quorum.auth.learnerRequireSasl=true
>>>> quorum.auth.serverRequireSasl=true
>>>>
>>>> What will happen if i set these properties on observers nodes as well ?
>>>>
>>>> Thanks,
>>>> Ram
>>>>
>>>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

And observer never joining the cluster its keep saying  "Cannot open
channel to"  in the logs.

On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Rakesh,
>
> Thank you, i have 3 floower and 3 observers in two different DC's
> followers came up fine with SASL but for some reasons observers are not
> coming up with the following error but i dont see any network issues, i was
> able to telnet to 2181 and 3888 ports.
>
>
> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> size: 1
> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> size: 1
> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
> size: 1
> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening
> channel to server 1
> 2018-09-24 17:55:34,151 [myid:6] - WARN
> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
> open channel to 1 at election address zk-server1/10.16.1.102:3888
> java.net.SocketTimeoutException: connect timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
> at
> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
>
>
> server.1=zk-server1:2888:3888
> server.2=zk-server2:2888:3888
> server.3=zk-server3:2888:3888
> server.4=zk-server4:2888:3888:observer
> server.5=zk-server5:2888:3888:observer
> server.6=zk-server6:2888:3888:observer
> peerType=observer
>
> What could be the reason?
>
> Ram
>
> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan 
> wrote:
>
>> Thanks Ram for the interest on this feature.
>>
>> Yes, user can enable SASL for Observer nodes as well. In general,
>> QuorumLearner will send authentication packet to peer QuorumServer.
>> Observer is a learner which follows the same quorum authentication protocol
>> and auth logic will work fine.
>>
>> FYI, hope you are referring below links for configurations,
>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>
>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>>
>> Please let us know if you are facing any issues.
>>
>> Thanks,
>> Rakesh
>>
>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Do we need to configure any thing on observer nodes for SASL
>>> authentication?
>>>
>>> tcpKeepAlive=true ( this is not for sasl but just asking )
>>>
>>> quorum.auth.enableSasl=true
>>> quorum.auth.learnerRequireSasl=true
>>> quorum.auth.serverRequireSasl=true
>>>
>>> What will happen if i set these properties on observers nodes as well ?
>>>
>>> Thanks,
>>> Ram
>>>
>>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Rakesh,

Thank you, i have 3 floower and 3 observers in two different DC's followers
came up fine with SASL but for some reasons observers are not coming up
with the following error but i dont see any network issues, i was able to
telnet to 2181 and 3888 ports.


2018-09-24 17:55:34,145 [myid:6] - DEBUG
[QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
size: 1
2018-09-24 17:55:34,145 [myid:6] - DEBUG
[QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
size: 1
2018-09-24 17:55:34,145 [myid:6] - DEBUG
[QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
size: 1
2018-09-24 17:55:34,145 [myid:6] - DEBUG
[QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - Opening
channel to server 1
2018-09-24 17:55:34,151 [myid:6] - WARN
[QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
open channel to 1 at election address zk-server1/10.16.1.102:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
at
org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)


server.1=zk-server1:2888:3888
server.2=zk-server2:2888:3888
server.3=zk-server3:2888:3888
server.4=zk-server4:2888:3888:observer
server.5=zk-server5:2888:3888:observer
server.6=zk-server6:2888:3888:observer
peerType=observer

What could be the reason?

Ram

On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan 
wrote:

> Thanks Ram for the interest on this feature.
>
> Yes, user can enable SASL for Observer nodes as well. In general,
> QuorumLearner will send authentication packet to peer QuorumServer.
> Observer is a learner which follows the same quorum authentication protocol
> and auth logic will work fine.
>
> FYI, hope you are referring below links for configurations,
>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>
> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>
> Please let us know if you are facing any issues.
>
> Thanks,
> Rakesh
>
> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Hi,
>>
>> Do we need to configure any thing on observer nodes for SASL
>> authentication?
>>
>> tcpKeepAlive=true ( this is not for sasl but just asking )
>>
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=true
>> quorum.auth.serverRequireSasl=true
>>
>> What will happen if i set these properties on observers nodes as well ?
>>
>> Thanks,
>> Ram
>>
>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Ok, thanks

On Mon, Sep 24, 2018 at 11:29 AM Norbert Kalmar
 wrote:

> Unfortunately I'm not entirely sure on this one, and I can't test it out
> right now, but shouldn't be any different then a normal follower. So you
> should configure SASL the same way. The only difference basically is that
> they are non-voters. Everything else works the same. Clients connect and
> can send read / write commands. So it would be a huge security hole if an
> observer is not configured as well.
>
> Regards,
> Norbert
>
> On Mon, Sep 24, 2018 at 10:59 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Any thoughts?
> >
> > On Sun, Sep 23, 2018 at 8:00 PM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Do we need to configure any thing on observer nodes for SASL
> > > authentication?
> > >
> > > tcpKeepAlive=true ( this is not for sasl but just asking )
> > >
> > > quorum.auth.enableSasl=true
> > > quorum.auth.learnerRequireSasl=true
> > > quorum.auth.serverRequireSasl=true
> > >
> > > What will happen if i set these properties on observers nodes as well ?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Any thoughts?

On Sun, Sep 23, 2018 at 8:00 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> Do we need to configure any thing on observer nodes for SASL
> authentication?
>
> tcpKeepAlive=true ( this is not for sasl but just asking )
>
> quorum.auth.enableSasl=true
> quorum.auth.learnerRequireSasl=true
> quorum.auth.serverRequireSasl=true
>
> What will happen if i set these properties on observers nodes as well ?
>
> Thanks,
> Ram
>

Re: Kerberos based authentication

[rammohan ganapavarapu]

Thank you  Norbert!, i was not sure if we need one or not, it may be good
to capture that in doc as pre requisites? i know lot of people know that
but it may help people like me.

Ram

On Mon, Sep 24, 2018 at 10:45 AM Norbert Kalmar
 wrote:

> Hi Ram,
>
> Yes, you will need a Kerberos instance, ZooKeeper doesn't have it bundled (
> I don't think it's even possible especially in terms of security reasons).
> Then you will have to configure SASL in ZooKeeper, as an additional layer
> over Kerberos.
>
> I'd say Kerberos is more secure, as for example you don't have your
> password stored. But it is more complex to setup and you require a 3rd
> party Kerberos instance. Lot of tutorial on it though. Like for example:
> https://blog.bluesoftglobal.com/3-steps-to-apache-zookeeper-authentication/
> https://github.com/ekoontz/zookeeper/wiki
>
> At the end, I think it comes down to preference.
>
> Regards,
> Norbert
>
> On Sat, Sep 22, 2018 at 1:40 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Hi,
> >
> > To configure "Kerberos based authentication" on zookeeper server-server
> or
> > client-server do we need to install any additional packages? do we need
> to
> > setup kerberos server? or zookeeper embedded the kerberos server? and
> what
> > is the recommended authentication mechanism, kerberos or digest-md5?
> >
> > Thanks,
> > Ram
> >
>

Observer properties for SASL authentication in 3.4.13 version

[rammohan ganapavarapu]

Hi,

Do we need to configure any thing on observer nodes for SASL
authentication?

tcpKeepAlive=true ( this is not for sasl but just asking )

quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true

What will happen if i set these properties on observers nodes as well ?

Thanks,
Ram

Kerberos based authentication

[rammohan ganapavarapu]

Hi,

To configure "Kerberos based authentication" on zookeeper server-server or
client-server do we need to install any additional packages? do we need to
setup kerberos server? or zookeeper embedded the kerberos server? and what
is the recommended authentication mechanism, kerberos or digest-md5?

Thanks,
Ram

Re: Have smaller server identifier, so dropping the connection

[rammohan ganapavarapu]

Andor,

Yes you are right, i miss understood the log message.

Thanks,
Ram

On Fri, Sep 14, 2018 at 3:23 AM Andor Molnar  wrote:

> Hi Ram,
>
> I might be missing something from your explanation, but that error message
> alone is not an issue. All ZK nodes open connection to each other, but
> having 2 connections between the same nodes is redundant and one of them
> has to be closed. To decide which one to close ZK use the server
> identifiers: node with smaller id closes the initiated connection. That’s
> the rule and decision is shown in the logs.
>
> Andor
>
>
>
> > On 2018. Sep 12., at 3:20, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
> >
> > Is this issue got fixed in 3.4.13 ? i thought it got fixed but i am still
> > seeing this when leader nodes is with lower myid and reboot a follower
> with
> > higher myid.
> >
> > Have smaller server identifier, so dropping the connection: (3, 2)
> >
> > Thanks,
> > Ram
>
>

Re: Upgrade from 3.4.5 to 3.4.13

[rammohan ganapavarapu]

Thank you

On Wed, Sep 12, 2018, 9:57 AM Norbert Kalmar 
wrote:

> A rolling restart is enough. Just stop one server ,replace jar, restart,
> and it will join back the quorum. Then do this for the rest of the servers.
>
> Update requires a restart, but you can save yourself a full restart this
> way. You will have no downtime.
>
> Regards,
> Norbert
>
> On Wed, Sep 12, 2018 at 3:05 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Norbert,
> >
> > Thank you, do we need to restart process after jar replacement?
> >
> > Ram
> >
> > On Wed, Sep 12, 2018, 12:56 AM Norbert Kalmar
>  > >
> > wrote:
> >
> > > Hi,
> > >
> > > We had a discussion about this not long ago, it was 3.4.8 to 3.4.13,
> but
> > > same things have to be done in this case.
> > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/How-to-upgrade-zookeeper-from-3-4-8-to-3-4-13-td7583831.html
> > >
> > > If you just replace the jar, you don't even have to copy anything else.
> > >
> > >
> > > Regards,
> > > Norbert
> > >
> > > On Wed, Sep 12, 2018 at 12:29 AM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > I have zk cluster with 3.4.5 version and planing to upgrade to 3.4.13
> > > > version, can some one point me to a upgrade steps? When i copy
> snapshot
> > > > from 3.4.5 node to 3.4.13 node it seems to be working but not sure
> what
> > > is
> > > > the proper way of upgrading existing cluster.
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>

Re: Upgrade from 3.4.5 to 3.4.13

[rammohan ganapavarapu]

Norbert,

Thank you, do we need to restart process after jar replacement?

Ram

On Wed, Sep 12, 2018, 12:56 AM Norbert Kalmar 
wrote:

> Hi,
>
> We had a discussion about this not long ago, it was 3.4.8 to 3.4.13, but
> same things have to be done in this case.
>
> http://zookeeper-user.578899.n2.nabble.com/How-to-upgrade-zookeeper-from-3-4-8-to-3-4-13-td7583831.html
>
> If you just replace the jar, you don't even have to copy anything else.
>
>
> Regards,
> Norbert
>
> On Wed, Sep 12, 2018 at 12:29 AM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > I have zk cluster with 3.4.5 version and planing to upgrade to 3.4.13
> > version, can some one point me to a upgrade steps? When i copy snapshot
> > from 3.4.5 node to 3.4.13 node it seems to be working but not sure what
> is
> > the proper way of upgrading existing cluster.
> >
> > Thanks,
> > Ram
> >
>

Have smaller server identifier, so dropping the connection

[rammohan ganapavarapu]

Is this issue got fixed in 3.4.13 ? i thought it got fixed but i am still
seeing this when leader nodes is with lower myid and reboot a follower with
higher myid.

Have smaller server identifier, so dropping the connection: (3, 2)

Thanks,
Ram

Upgrade from 3.4.5 to 3.4.13

[rammohan ganapavarapu]

I have zk cluster with 3.4.5 version and planing to upgrade to 3.4.13
version, can some one point me to a upgrade steps? When i copy snapshot
from 3.4.5 node to 3.4.13 node it seems to be working but not sure what is
the proper way of upgrading existing cluster.

Thanks,
Ram

Re: How to use these system properties

[rammohan ganapavarapu]

just have to have those properties in zoo.conf file.

4lw.commands.whitelist=*
leaderServes=yes
globalOutstandingLimit=1000
preAllocSize=65536
syncEnabled=true
readonlymode.enabled=false
tcpKeepAlive=false
ipReachableTimeout=0


On Tue, Sep 11, 2018 at 6:56 AM Steph van Schalkwyk 
wrote:

> Kindly post the solution for others to not have to re-invent the wheel.
>
>
> +1.314.452.2896 (Tel/SMS)
>
> On Mon, Sep 10, 2018 at 11:18 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > NVM, i think i got it.
> >
> > Thanks,
> > Ram
> >
> > On Mon, Sep 10, 2018 at 3:35 PM rammohan ganapavarapu <
> > rammohanga...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am trying to use the new properties available in 3.4.13 version but i
> > > couldn't find any examples, can some one help me on how to set those
> > > properties.
> > >
> > >
> > > *zookeeper.ipReachableTimeout*
> > > *zookeeper.tcpKeepAlive*
> > > *zookeeper.observer.syncEnabled*
> > >
> > >
> > > Do i ned to set them in zoo.conf or should i pass them as extra args as
> > "-Dzookeeper.tcpKeepAlive=true"
> > > to startup command ?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>

Re: How to use these system properties

[rammohan ganapavarapu]

NVM, i think i got it.

Thanks,
Ram

On Mon, Sep 10, 2018 at 3:35 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> Hi,
>
> I am trying to use the new properties available in 3.4.13 version but i
> couldn't find any examples, can some one help me on how to set those
> properties.
>
>
> *zookeeper.ipReachableTimeout*
> *zookeeper.tcpKeepAlive*
> *zookeeper.observer.syncEnabled*
>
>
> Do i ned to set them in zoo.conf or should i pass them as extra args as 
> "-Dzookeeper.tcpKeepAlive=true"
> to startup command ?
>
> Thanks,
> Ram
>

How to use these system properties

[rammohan ganapavarapu]

Hi,

I am trying to use the new properties available in 3.4.13 version but i
couldn't find any examples, can some one help me on how to set those
properties.


*zookeeper.ipReachableTimeout*
*zookeeper.tcpKeepAlive*
*zookeeper.observer.syncEnabled*


Do i ned to set them in zoo.conf or should i pass them as extra args
as "-Dzookeeper.tcpKeepAlive=true"
to startup command ?

Thanks,
Ram

Re: Understanding zookeeper quorum

[rammohan ganapavarapu]

Thank you

On Thu, Aug 2, 2018 at 10:50 AM John Sirois  wrote:

> The full ensemble size is still known (5); so operations still take 3
> successful node writes to work.
>
> On Thu, Aug 2, 2018, 11:46 AM rammohan ganapavarapu <
> rammohanga...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Lets say i have 5 node zk cluster and if one node went down then how does
> > quorum work as i have only 4 nodes now and it may run into split-brain
> > scenario right?
> >
> >
> > Thanks,
> > Ram
> >
>

Understanding zookeeper quorum

[rammohan ganapavarapu]

Hi,

Lets say i have 5 node zk cluster and if one node went down then how does
quorum work as i have only 4 nodes now and it may run into split-brain
scenario right?


Thanks,
Ram

Re: Observer went down with Read timed out exception

[rammohan ganapavarapu]

Andor,

Thanks for your time, i am waiting for 3.5 stable version to upgrade. Log
says read timeout right, what kind of packet or data is it reading from
leader?

Ram

On Wed, Jul 4, 2018, 12:24 AM Andor Molnar 
wrote:

> Unfortunately I cannot imagine anything other than what Norbert already
> mentioned. If the followers were stable, a problem in the DC-DC link could
> explain why all the observers have gone in a moment. If it had been a
> problem with leader overloading, even the followers would have gone with
> the observers too.
>
> If none of these cases happened, I'm afraid I cannot help more. I'm not
> aware of a similar, existing issue. Maybe more senior devs can comment.
>
> However, your version is quite old. Most production clusters are running
> 3.4.6 or 3.4.9 as far as I'm concerned. You might want to upgrade to the
> latest stable version though which is 3.4.12 at the moment. 3.4.13 will be
> out soon as well.
>
> Regards,
> Andor
>
>
>
>
> On Tue, Jul 3, 2018 at 8:13 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Andor,
> >
> > Zk  version that i use is zk_version 3.4.5-1392090, built on 09/30/2012
> > 17:52 GMT
> > No Auth or encryption config
> > None my of network graphs showing any dip or unusual pattern thats why i
> am
> > thinking there may not be any n/w issue. I have those nodes in cloud so
> > checking with them to see if any n/w issue between regions.
> >
> > Thanks,
> > Ram
> >
> >
> > On Tue, Jul 3, 2018 at 6:29 AM Andor Molnar 
> > wrote:
> >
> > > Hi Rammohan,
> > >
> > > Would you please elaborate on the details of your cluster setup?
> > > Which ZooKeeper version do you use?
> > > Do you use authentication / encryption?
> > > Would you please attach config files and log files of other nodes like
> > > leader and followers?
> > >
> > > How did you make sure that there was no network problem at the time
> when
> > > issue happened?
> > > Would you please attach graphs / diagrams on the network traffic
> > including
> > > latency and bandwidth usage between the affected data centers?
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > >
> > > On Tue, Jul 3, 2018 at 2:56 PM, rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > Yes I am sure there is no network issues, if leader is busy in GC
> > > followers
> > > > on the same DC would have been shutdown as we right but it wasn't the
> > > case.
> > > >
> > > > On Tue, Jul 3, 2018, 1:56 AM Norbert Kalmar
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi Ram,
> > > > >
> > > > > Are you sure there were no network error? For me, this looks like
> it
> > > > could
> > > > > be due to failed heartbeats (as shutdown was called after the
> > timeout).
> > > > >
> > > > > It is also possible the leader was busy (maybe garbage collection
> > > caused
> > > > > pause?) - especially if you store big(ish) chunks of data in
> > ZooKeeper.
> > > > > (There is plan to integrate JVMPauseMonitor to ZooKeeper for this
> > > reason
> > > > > actually).
> > > > >
> > > > > Regards,
> > > > > Norbert
> > > > >
> > > > > On Mon, Jul 2, 2018 at 9:13 PM rammohan ganapavarapu <
> > > > > rammohanga...@gmail.com> wrote:
> > > > >
> > > > > > All,
> > > > > >
> > > > > > I have multi data-center ldap cluster setup with other
> data-center
> > > with
> > > > > all
> > > > > > observers all of sudden all the observer threads went down with
> the
> > > > > > following message, any idea why they went down? We don't see any
> > > > network
> > > > > > related issues between data-centers.
> > > > > >
> > > > > >
> > > > > > 2018-06-29 05:32:59,036 [myid:222] - WARN
> > > > > > [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@79] -
> > Exception
> > > > when
> > > > > > observing the leader
> > > > > > java.net.SocketTimeoutException: Read timed out
> > > > > > at java.net.SocketInputStream.socketRead0(Native Method)
> &g

Re: Observer went down with Read timed out exception

[rammohan ganapavarapu]

Andor,

Zk  version that i use is zk_version 3.4.5-1392090, built on 09/30/2012
17:52 GMT
No Auth or encryption config
None my of network graphs showing any dip or unusual pattern thats why i am
thinking there may not be any n/w issue. I have those nodes in cloud so
checking with them to see if any n/w issue between regions.

Thanks,
Ram


On Tue, Jul 3, 2018 at 6:29 AM Andor Molnar 
wrote:

> Hi Rammohan,
>
> Would you please elaborate on the details of your cluster setup?
> Which ZooKeeper version do you use?
> Do you use authentication / encryption?
> Would you please attach config files and log files of other nodes like
> leader and followers?
>
> How did you make sure that there was no network problem at the time when
> issue happened?
> Would you please attach graphs / diagrams on the network traffic including
> latency and bandwidth usage between the affected data centers?
>
> Regards,
> Andor
>
>
>
>
> On Tue, Jul 3, 2018 at 2:56 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > Yes I am sure there is no network issues, if leader is busy in GC
> followers
> > on the same DC would have been shutdown as we right but it wasn't the
> case.
> >
> > On Tue, Jul 3, 2018, 1:56 AM Norbert Kalmar  >
> > wrote:
> >
> > > Hi Ram,
> > >
> > > Are you sure there were no network error? For me, this looks like it
> > could
> > > be due to failed heartbeats (as shutdown was called after the timeout).
> > >
> > > It is also possible the leader was busy (maybe garbage collection
> caused
> > > pause?) - especially if you store big(ish) chunks of data in ZooKeeper.
> > > (There is plan to integrate JVMPauseMonitor to ZooKeeper for this
> reason
> > > actually).
> > >
> > > Regards,
> > > Norbert
> > >
> > > On Mon, Jul 2, 2018 at 9:13 PM rammohan ganapavarapu <
> > > rammohanga...@gmail.com> wrote:
> > >
> > > > All,
> > > >
> > > > I have multi data-center ldap cluster setup with other data-center
> with
> > > all
> > > > observers all of sudden all the observer threads went down with the
> > > > following message, any idea why they went down? We don't see any
> > network
> > > > related issues between data-centers.
> > > >
> > > >
> > > > 2018-06-29 05:32:59,036 [myid:222] - WARN
> > > > [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception
> > when
> > > > observing the leader
> > > > java.net.SocketTimeoutException: Read timed out
> > > > at java.net.SocketInputStream.socketRead0(Native Method)
> > > > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> > > > at java.net.SocketInputStream.read(SocketInputStream.java:170)
> > > > at java.net.SocketInputStream.read(SocketInputStream.java:141)
> > > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> > > > at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> > > > at java.io.DataInputStream.readInt(DataInputStream.java:387)
> > > > at org.apache.jute.BinaryInputArchive.readInt(
> > BinaryInputArchive.java:63)
> > > > at
> > > >
> > > >
> > > org.apache.zookeeper.server.quorum.QuorumPacket.
> > deserialize(QuorumPacket.java:83)
> > > > at
> > > >
> > > org.apache.jute.BinaryInputArchive.readRecord(
> > BinaryInputArchive.java:108)
> > > > at
> > > org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
> > > > at
> > > >
> > > org.apache.zookeeper.server.quorum.Observer.observeLeader(
> > Observer.java:75)
> > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> > QuorumPeer.java:727)
> > > > 2018-06-29 05:32:59,244 [myid:222] - INFO
> > > > [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown
> > > called
> > > > java.lang.Exception: shutdown Observer
> > > > at
> > > org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137)
> > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> > QuorumPeer.java:731)
> > > >
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>

Re: Observer went down with Read timed out exception

[rammohan ganapavarapu]

Yes I am sure there is no network issues, if leader is busy in GC followers
on the same DC would have been shutdown as we right but it wasn't the case.

On Tue, Jul 3, 2018, 1:56 AM Norbert Kalmar 
wrote:

> Hi Ram,
>
> Are you sure there were no network error? For me, this looks like it could
> be due to failed heartbeats (as shutdown was called after the timeout).
>
> It is also possible the leader was busy (maybe garbage collection caused
> pause?) - especially if you store big(ish) chunks of data in ZooKeeper.
> (There is plan to integrate JVMPauseMonitor to ZooKeeper for this reason
> actually).
>
> Regards,
> Norbert
>
> On Mon, Jul 2, 2018 at 9:13 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
> > All,
> >
> > I have multi data-center ldap cluster setup with other data-center with
> all
> > observers all of sudden all the observer threads went down with the
> > following message, any idea why they went down? We don't see any network
> > related issues between data-centers.
> >
> >
> > 2018-06-29 05:32:59,036 [myid:222] - WARN
> > [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception when
> > observing the leader
> > java.net.SocketTimeoutException: Read timed out
> > at java.net.SocketInputStream.socketRead0(Native Method)
> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> > at java.net.SocketInputStream.read(SocketInputStream.java:170)
> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> > at java.io.DataInputStream.readInt(DataInputStream.java:387)
> > at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> > at
> >
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> > at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
> > at
> >
> org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:75)
> > at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
> > 2018-06-29 05:32:59,244 [myid:222] - INFO
> > [QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown
> called
> > java.lang.Exception: shutdown Observer
> > at
> org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137)
> > at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:731)
> >
> >
> > Thanks,
> > Ram
> >
>

Observer went down with Read timed out exception

[rammohan ganapavarapu]

All,

I have multi data-center ldap cluster setup with other data-center with all
observers all of sudden all the observer threads went down with the
following message, any idea why they went down? We don't see any network
related issues between data-centers.


2018-06-29 05:32:59,036 [myid:222] - WARN
[QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception when
observing the leader
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
at
org.apache.zookeeper.server.quorum.Observer.observeLeader(Observer.java:75)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
2018-06-29 05:32:59,244 [myid:222] - INFO
[QuorumPeer[myid=222]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown called
java.lang.Exception: shutdown Observer
at org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:731)


Thanks,
Ram

Re: How to find the actual size of zk data

[rammohan ganapavarapu]

Mark,

Thanks, yes it would be nice to have as part of JMX. Please let me know if
its already there or any tool that does.

Ram

On Tue, Mar 20, 2018 at 2:27 AM, Mark Fenes  wrote:

> Ok, understood. Maybe the best way of calculating the ZK data size would
> be to count/estimate the number of znodes and their average data size per
> node. This would eliminate file size differences by file format.
> Maybe we could write a simple utility to read a snapshot file and
> calculate these statistics. Also transactions after the latest snapshot
> file can cause file size differences. Or perhaps the ZK server itself could
> publish online statistics about node count and average node size via JMX -
> I'll check if this is already implemented or not.
>
> Mark
>
> On Mon, Mar 19, 2018 at 4:05 PM, rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> zkTreeutil has option to dump every thing into a directory and create
>> same tree in filesystem under that directory, i am talking about that size.
>>
>> Ram
>>
>> On Mon, Mar 19, 2018 at 5:42 AM, Mark Fenes  wrote:
>>
>>>
>>> Hi Ram,
>>>
>>> If the format of the snapshot file and the zktreeutil dump file is
>>> different, they will have different sizes. Does zktreeutil export the
>>> zk data in XML format?
>>>
>>> Mark
>>>
>>>
>>> On Fri, Mar 16, 2018 at 2:58 AM, rammohan ganapavarapu <
>>> rammohanga...@gmail.com> wrote:
>>>
>>>> Jordan,
>>>>
>>>> Thank you, even though you have multiple snapshot files in dataDir
>>>> zookeeper uses only one latest file right? If my latest snapshot file is
>>>> 1GB means my total zk data side is 1GB ?  But when I use zktreeutil and
>>>> dump to filesystem i am getting lot more GB space then snapshot size.
>>>>
>>>> Ram
>>>>
>>>> On Thu, Mar 15, 2018, 4:02 PM Jordan Zimmerman <
>>>> jor...@jordanzimmerman.com>
>>>> wrote:
>>>>
>>>> > > How to find  actual size of zookeeper data since it store and
>>>> operate
>>>> > from
>>>> > > in-memory? and snapshot may not give me actual size?
>>>> >
>>>> > ZooKeeper transactions/snapshots are stored on disk in a series of
>>>> files.
>>>> > You can use your file system to get the combined sizes of these
>>>> files. The
>>>> > directory is specified by the "dataDir" value in zoo.cfg.
>>>> >
>>>> > > One more question, since its a in-memory datastore, what if my zk
>>>> data
>>>> > > grows beyond available RAM? does it swap?
>>>> >
>>>> > No. ZooKeeper does not swap. You will get OutOfMemory exceptions if
>>>> store
>>>> > too much data. The ZooKeeper database is always limited by memory.
>>>> >
>>>> > -JZ
>>>> >
>>>> > > On Mar 15, 2018, at 5:57 PM, rammohan ganapavarapu <
>>>> > rammohanga...@gmail.com> wrote:
>>>> > >
>>>> > > Hi,
>>>> > >
>>>> > > How to find  actual size of zookeeper data since it store and
>>>> operate
>>>> > from
>>>> > > in-memory? and snapshot may not give me actual size? I used
>>>> zkteeUtils to
>>>> > > dump the data is it the actual size ?
>>>> > >
>>>> > > One more question, since its a in-memory datastore, what if my zk
>>>> data
>>>> > > grows beyond available RAM? does it swap?
>>>> > >
>>>> > > Thanks,
>>>> > > Ram
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>