[
https://issues.apache.org/jira/browse/ZOOKEEPER-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917661#action_12917661
]
Patrick Hunt commented on ZOOKEEPER-885:
----------------------------------------
I suspect this is being caused by ZOOKEEPER-568 which was fixed in 3.3 releases
and later. (you are using 3.2.2 where it's not fixed)
ZK provides very strong durability guarantees, and needs to wait for fsync to
complete before it responds to a write request with "success". I've seen the
fsync take significant amounts of time depending on IO load, kernel, filesyste,
etc...
Even though your client isn't doing work, pings are still sent in the b/g.
These should not cause a forced disk sync, but they are forcing a sync due to
this bug 568. As a result we need to wait for the disk sync to complete before
replying to the ping, as you are loading the disk this is taking time. If it
takes longer than 3.3 seconds (1/3 the timeout) the client will timeout. (10
sec will cause expiration)
Can you retry this using zookeeper 3.3.1?
BTW, You can monitor server latency using the following:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_zkCommands
specifically the "stat" command. Check the server avg/max latency before and
during your "dd ..." test.
> Zookeeper drops connections under moderate IO load
> --------------------------------------------------
>
> Key: ZOOKEEPER-885
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885
> Project: Zookeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.2.2
> Environment: Debian (Lenny)
> 1Gb RAM
> swap disabled
> 100Mb heap for zookeeper
> Reporter: Alexandre Hardy
> Priority: Critical
> Attachments: WatcherTest.java
>
>
> A zookeeper server under minimum load, with a number of clients watching
> exactly one node will fail to maintain the connection when the machine is
> subjected to moderate IO load.
> In a specific test example we had three zookeeper servers running on
> dedicated machines with 45 clients connected, watching exactly one node. The
> clients would disconnect after moderate load was added to each of the
> zookeeper servers with the command:
> {noformat}
> dd if=/dev/urandom of=/dev/mapper/nimbula-test
> {noformat}
> The {{dd}} command transferred data at a rate of about 4Mb/s.
> The same thing happens with
> {noformat}
> dd if=/dev/zero of=/dev/mapper/nimbula-test
> {noformat}
> It seems strange that such a moderate load should cause instability in the
> connection.
> Very few other processes were running, the machines were setup to test the
> connection instability we have experienced. Clients performed no other read
> or mutation operations.
> Although the documents state that minimal competing IO load should present on
> the zookeeper server, it seems reasonable that moderate IO should not cause
> problems in this case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
