Re: zookeeper discnnected every minutes when i am getting bulk data

[AjeetSingh]

Hi Rakesh,

I am using 3 Kafka/Zookeepers nodes and that is running cluster mode and i
am getting 100 GB data per day in one topics and i have ELK cluster.

So i am not getting zookeepers disconnected in each mins and getting lag
when zookeepers disconnected and how can i see the jute.maxbuffer  and
define the exact value.

Please help me.

Cheers,
Ajeet S






-
Cheers,
Ajeet S

--
View this message in context: 
http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817p7582820.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: AvgRequestLatency metric always zero

[Rakesh Radhakrishnan]

Thanks Arshad for the good analysis.

 How can we go about getting these fixes?

Please feel free to raise an improvement task under ZK project issue
tracker, URL: https://issues.apache.org/jira/browse/ZOOKEEPER
Also, I'd appreciate if you can put a patch to fix it. Probably you can
refer how to contribute to the project section,
https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute

On Fri, Dec 2, 2016 at 3:25 AM, allen chan 
wrote:

> Arshad,
> How can we go about getting these fixes?
>
>
> On Thu, Dec 1, 2016 at 12:01 PM, Arshad Mohammad <
> arshad.mohamma...@gmail.com> wrote:
>
> > Hi Alen
> >
> > AvgRequestLatency 0 implies that on an average time taken by the server
> to
> > serve a request is less than one milli second. May be it is really 0 or
> it
> > may be 0.99 ms.
> >
> > This behaviour is not specific to 3.4.7 and 3.4.9 version but the same
> > behaviour is there in all versions
> >
> > I find two reason whys AvgRequestLatency is almost all the time 0
> >
> > 1) Ping requests are counted the most:
> >
> > AvgRequestLatency is calculated as
> >
> > AvgRequestLatency=totalLatency/count
> >
> > Ping requests come very often and complete very fast, these request add
> > nothing to totalLatency but add one to count.
> >
> > 2) Wrong data type is chosen to store AvgRequestLatency:
> >
> > AvgRequestLatency is calculated and store as the long value instead of
> > double vlaue.
> >
> >
> >
> > In my opinion ZooKeeper code should be modified to improve this metrics
> >
> > i) Ping request should be ignored while recording the statistics or at
> > least should be configurable whether to ignore or not. If ping request is
> > not counted even other metrics will be more meaningful.
> >
> > ii)  AvgRequestLatency should be of double type.
> >
> >
> >
> > -Arshad
> >
> > On Thu, Dec 1, 2016 at 4:50 AM, allen chan  >
> > wrote:
> >
> > > Anyone seeing this issue? I am experiencing it on 3.4.7 and 3.4.9
> > >
> > > JMX metric name: AvgRequestLatency
> > > JMX Location:
> > > org.apache.ZooKeeperService:name0=ReplicatedServer_id#,
> > > name1=replica.#,name2=[Leader|Follower]:AvgRequestLatency
> > >
> > > It always has a value of zero. The MaxRequestLatency is non-zero and
> > > changes while the AvgRequestLatency always is zero.
> > >
> > > The zk_avg_latency metric in mntr is also zero. Is this metric not
> > tracked
> > > anymore?
> > >
> > > Thanks
> > > --
> > > Allen Michael Chan
> > >
> >
>
>
>
> --
> Allen Michael Chan
>

Re: zookeeper discnnected every minutes when i am getting bulk data

[Rakesh Radhakrishnan]

Hi Ajeet,

Are you talking about putting more data bytes(bulk data) into a single
zNode?

I hope you know that, the data size (or packet size) is controlled through
configuration property - Java system property: jute.maxbuffer
Reference: https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html

ZooKeeper will throw exception if the data size cross this value. Could you
check ZooKeeper server logs to see the chances of exceptions and due to
that the client requests are failing.

Thanks,
Rakesh

On Fri, Dec 2, 2016 at 10:38 AM, AjeetSingh 
wrote:

> Hi Folks,
>
> I am facing some challenging issue in my cluster and that is related with
> zookeeper request timeout.
>
> A:-  3 Kafka cluster with zookeeper
> B:-  Zookeeper Veriosn is zookeeper-3.4.6
> C:- Java Version java version "1.7.0_111"
> OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-0ubuntu0.14.04.3)
> OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
> D:- Kafka Version kafka_2.10-0.10.0.1
>
> When i am running the zookeeper and getting timeout in every minutes,
> please
> folle the below error.
>
> [2016-12-02 09:46:30,869] INFO Client session timed out, have not heard
> from
> server in 4002ms for sessionid 0x158b9f7c720, closing socket connection
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:30,971] INFO zookeeper state changed (Disconnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-12-02 09:46:31,922] INFO Opening socket connection to server
> 10.0.2.3/10.0.2.3:2181. Will not attempt to authenticate using SASL
> (unknown
> error) (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:31,924] INFO Socket connection established to
> 10.0.2.3/10.0.2.3:2181, initiating session (org.apache.zookeeper.
> ClientCnxn)
> [2016-12-02 09:46:31,926] INFO Session establishment complete on server
> 10.0.2.3/10.0.2.3:2181, sessionid = 0x158b9f7c720, negotiated timeout
> =
> 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-12-02 09:46:31,926] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2016-12-02 09:50:15,163] INFO [Group Metadata Manager on Broker 2]:
> Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.GroupMetadataManager)
> 
>
> Please help me and resolve this problem.
>
>
> Cheers,
> Ajeet S
>
>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.
> n2.nabble.com/zookeeper-discnnected-every-minutes-
> when-i-am-getting-bulk-data-tp7582817.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>

zookeeper discnnected every minutes when i am getting bulk data

[AjeetSingh]

Hi Folks,

I am facing some challenging issue in my cluster and that is related with
zookeeper request timeout.

A:-  3 Kafka cluster with zookeeper
B:-  Zookeeper Veriosn is zookeeper-3.4.6
C:- Java Version java version "1.7.0_111"
OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-0ubuntu0.14.04.3)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
D:- Kafka Version kafka_2.10-0.10.0.1

When i am running the zookeeper and getting timeout in every minutes, please
folle the below error.

[2016-12-02 09:46:30,869] INFO Client session timed out, have not heard from
server in 4002ms for sessionid 0x158b9f7c720, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:30,971] INFO zookeeper state changed (Disconnected)
(org.I0Itec.zkclient.ZkClient)
[2016-12-02 09:46:31,922] INFO Opening socket connection to server
10.0.2.3/10.0.2.3:2181. Will not attempt to authenticate using SASL (unknown
error) (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,924] INFO Socket connection established to
10.0.2.3/10.0.2.3:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,926] INFO Session establishment complete on server
10.0.2.3/10.0.2.3:2181, sessionid = 0x158b9f7c720, negotiated timeout =
6000 (org.apache.zookeeper.ClientCnxn)
[2016-12-02 09:46:31,926] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
[2016-12-02 09:50:15,163] INFO [Group Metadata Manager on Broker 2]: Removed
0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
 

Please help me and resolve this problem.


Cheers,
Ajeet S






--
View this message in context: 
http://zookeeper-user.578899.n2.nabble.com/zookeeper-discnnected-every-minutes-when-i-am-getting-bulk-data-tp7582817.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Setting ZOO_DATADIR

[Martin Gainty]

apparently when ZOO_DATADIR is empty mkdir fubars mkdir ''

when ZOO_DATADIR is empty add this export ZOO_DATADIR="zoo_datadir"


if [ -z "$ZOOPIDFILE" ]; then
ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
if [ ! -d "$ZOO_DATADIR" ]; then
echo "ZOO_DATADIR="

export ZOO_DATADIR="zoo_datadir"
echo $ZOO_DATADIR

mkdir -p "$ZOO_DATADIR"
fi

HTH
M-



From: Zac Harvey 
Sent: Thursday, December 1, 2016 11:03 AM
To: Martin Gainty; user@zookeeper.apache.org
Subject: Re: Messages intermittently get lost


Thanks everyone


The script can be found like so:


wget 
http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

tar xfz zookeeper-3.4.6.tar.gz

cd zookeeper-3.4.6/bin

zkServer.sh


Let me know if you have any other questions/concerns!


Best,

Zac


From: Martin Gainty 
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost


absolutely..lets get this resolved asap


Zac...can you send me the abending script?


Molte Grazie Flavio!

Martini
__




From: Flavio Junqueira 
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: us...@kafka.apache.org
Subject: Re: Messages intermittently get lost

Hey Martin,

Do you want to create a jira for this and report the issue?

-Flavio

> On 30 Nov 2016, at 18:33, Martin Gainty  wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected 
> (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> 
> From: Martin Gainty 
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and 
> Conquer each part of this problem individually starting at the source of the 
> data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a 
> Kafka producer fail to be received by a Kafka consumer?
>
> 
> From: Zac Harvey 
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I 
> ask why? What is it about my problem description that makes you think this is 
> an issue with ZooKeeper?
>
> 
> From: Martin Gainty 
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> 
> From: Zac Harvey 
> Sent: Friday, November 25, 2016 6:17 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - 
IBM
www.ibm.com
Use these solutions to resolve the problems that you might encounter with 
Apache ZooKeeper.



> ZooKeeper problems and solutions - 
> IBM
ZooKeeper problems and solutions - 
IBM

Re: AvgRequestLatency metric always zero

[allen chan]

Arshad,
How can we go about getting these fixes?


On Thu, Dec 1, 2016 at 12:01 PM, Arshad Mohammad <
arshad.mohamma...@gmail.com> wrote:

> Hi Alen
>
> AvgRequestLatency 0 implies that on an average time taken by the server to
> serve a request is less than one milli second. May be it is really 0 or it
> may be 0.99 ms.
>
> This behaviour is not specific to 3.4.7 and 3.4.9 version but the same
> behaviour is there in all versions
>
> I find two reason whys AvgRequestLatency is almost all the time 0
>
> 1) Ping requests are counted the most:
>
> AvgRequestLatency is calculated as
>
> AvgRequestLatency=totalLatency/count
>
> Ping requests come very often and complete very fast, these request add
> nothing to totalLatency but add one to count.
>
> 2) Wrong data type is chosen to store AvgRequestLatency:
>
> AvgRequestLatency is calculated and store as the long value instead of
> double vlaue.
>
>
>
> In my opinion ZooKeeper code should be modified to improve this metrics
>
> i) Ping request should be ignored while recording the statistics or at
> least should be configurable whether to ignore or not. If ping request is
> not counted even other metrics will be more meaningful.
>
> ii)  AvgRequestLatency should be of double type.
>
>
>
> -Arshad
>
> On Thu, Dec 1, 2016 at 4:50 AM, allen chan 
> wrote:
>
> > Anyone seeing this issue? I am experiencing it on 3.4.7 and 3.4.9
> >
> > JMX metric name: AvgRequestLatency
> > JMX Location:
> > org.apache.ZooKeeperService:name0=ReplicatedServer_id#,
> > name1=replica.#,name2=[Leader|Follower]:AvgRequestLatency
> >
> > It always has a value of zero. The MaxRequestLatency is non-zero and
> > changes while the AvgRequestLatency always is zero.
> >
> > The zk_avg_latency metric in mntr is also zero. Is this metric not
> tracked
> > anymore?
> >
> > Thanks
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

Re: AvgRequestLatency metric always zero

[Arshad Mohammad]

Hi Alen

AvgRequestLatency 0 implies that on an average time taken by the server to
serve a request is less than one milli second. May be it is really 0 or it
may be 0.99 ms.

This behaviour is not specific to 3.4.7 and 3.4.9 version but the same
behaviour is there in all versions

I find two reason whys AvgRequestLatency is almost all the time 0

1) Ping requests are counted the most:

AvgRequestLatency is calculated as

AvgRequestLatency=totalLatency/count

Ping requests come very often and complete very fast, these request add
nothing to totalLatency but add one to count.

2) Wrong data type is chosen to store AvgRequestLatency:

AvgRequestLatency is calculated and store as the long value instead of
double vlaue.



In my opinion ZooKeeper code should be modified to improve this metrics

i) Ping request should be ignored while recording the statistics or at
least should be configurable whether to ignore or not. If ping request is
not counted even other metrics will be more meaningful.

ii)  AvgRequestLatency should be of double type.



-Arshad

On Thu, Dec 1, 2016 at 4:50 AM, allen chan 
wrote:

> Anyone seeing this issue? I am experiencing it on 3.4.7 and 3.4.9
>
> JMX metric name: AvgRequestLatency
> JMX Location:
> org.apache.ZooKeeperService:name0=ReplicatedServer_id#,
> name1=replica.#,name2=[Leader|Follower]:AvgRequestLatency
>
> It always has a value of zero. The MaxRequestLatency is non-zero and
> changes while the AvgRequestLatency always is zero.
>
> The zk_avg_latency metric in mntr is also zero. Is this metric not tracked
> anymore?
>
> Thanks
> --
> Allen Michael Chan
>

Re: Messages intermittently get lost

[Zac Harvey]

Thanks everyone


The script can be found like so:


wget 
http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

tar xfz zookeeper-3.4.6.tar.gz

cd zookeeper-3.4.6/bin

zkServer.sh


Let me know if you have any other questions/concerns!


Best,

Zac


From: Martin Gainty 
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost


absolutely..lets get this resolved asap


Zac...can you send me the abending script?


Molte Grazie Flavio!

Martini
__




From: Flavio Junqueira 
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: us...@kafka.apache.org
Subject: Re: Messages intermittently get lost

Hey Martin,

Do you want to create a jira for this and report the issue?

-Flavio

> On 30 Nov 2016, at 18:33, Martin Gainty  wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected 
> (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> 
> From: Martin Gainty 
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and 
> Conquer each part of this problem individually starting at the source of the 
> data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a 
> Kafka producer fail to be received by a Kafka consumer?
>
> 
> From: Zac Harvey 
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I 
> ask why? What is it about my problem description that makes you think this is 
> an issue with ZooKeeper?
>
> 
> From: Martin Gainty 
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> 
> From: Zac Harvey 
> Sent: Friday, November 25, 2016 6:17 AM
> To: us...@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - 
IBM
www.ibm.com
Use these solutions to resolve the problems that you might encounter with 
Apache ZooKeeper.



> ZooKeeper problems and solutions - 
> IBM
ZooKeeper problems and solutions - 
IBM
www.ibm.com
Use these solutions to resolve the problems that you might encounter with 
Apache ZooKeeper.



> www.ibm.com
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]

IBM - United States
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to 
creating innovations that matter for the world



> Use these solutions to resolve the problems that you migh