[
https://issues.apache.org/jira/browse/ZOOKEEPER-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728996#comment-14728996
]
nijel commented on ZOOKEEPER-2251:
----------------------------------
hi [~suda]
i got this issue from one of our site. The issue is from ResourceManager.
I could get very few info. The thread which got blocked is as follows
{code}
StandByTransitionThread Handler" daemon prio=10 tid=0x000000000119a800
nid=0x641e3 in Object.wait() [0x00007ff3f67ab000]
1836 java.lang.Thread.State: WAITING (on object monitor)
1837 at java.lang.Object.wait(Native Method)
1838 - waiting on <0x00000007855f2010> (a
org.apache.zookeeper.ClientCnxn$Packet)
1839 at java.lang.Object.wait(Object.java:503)
1840 at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1414)
1841 - locked <0x00000007855f2010> (a
org.apache.zookeeper.ClientCnxn$Packet)
1842 at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1386)
1843 at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:677)
1844 - locked <0x000000078660daf0> (a org.apache.zookeeper.ZooKeeper)
1845 at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.closeZkClients(ZKRMStateStore.java:360)
1846 - locked <0x0000000781087940> (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
1847 at
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.closeInternal(ZKRMStateStore.java:382)
1848 - locked <0x0000000781087940> (a
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore)
1849 at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStop(RMStateStore.java:486)
1850 at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
1851 - locked <0x00000007823b2a80> (a java.lang.Object)
1852 at
org.apache.hadoop.service.AbstractService.close(AbstractService.java:250)
1853 at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:549)
1854 at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
1855 - locked <0x00000007823d0610> (a java.lang.Object)
1856 at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:958)
1857 at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1018)
1858 - locked <0x0000000780059c60> (a
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager)
1859 at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandBy(ResourceManager.java:703)
1860 at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$StandByTransitionThread.run(RMStateStore.java:884)
{code}
My analysis is that it is waiting for the packet response
Hope this helps
> Add Client side packet response timeout to avoid infinite wait.
> ---------------------------------------------------------------
>
> Key: ZOOKEEPER-2251
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2251
> Project: ZooKeeper
> Issue Type: Bug
> Reporter: nijel
>
> I came across one issue related to Client side packet response timeout In my
> cluster many packet drops happened for some time.
> One observation is the zookeeper client got hanged. As per the thread dump it
> is waiting for the response/ACK for the operation performed (synchronous API
> used here).
> I am using
> zookeeper.serverCnxnFactory=org.apache.zookeeper.server.NIOServerCnxnFactory
> Since only few packets missed there is no DISCONNECTED event occurred.
> Need add a "response time out" for the operations or packets.
> *Comments from [~rakeshr]*
> My observation about the problem:-
> * Can use tools like 'Wireshark' to simulate the artificial packet loss.
> * Assume there is only one packet in the 'outgoingQueue' and unfortunately
> the server response packet lost. Now, client will enter into infinite
> waiting.
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1515
> * Probably we can discuss more about this problem and possible solutions(add
> packet ACK timeout or another better approach) in the jira.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
