[jira] [Updated] (HBASE-15805) Canary test timeout if there is failed or shutdown zookeeper node in a quorum
[ https://issues.apache.org/jira/browse/HBASE-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15805: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98. Due to the JIRA lockdown and the size of our contributor list I'm unable to assign this issue to you at this time [~vishk]. Browser is not cooperating. We'll fix the assignment later. The commit message includes proper attribution. > Canary test timeout if there is failed or shutdown zookeeper node in a quorum > - > > Key: HBASE-15805 > URL: https://issues.apache.org/jira/browse/HBASE-15805 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 0.98.20 >Reporter: Vishal Khandelwal > Fix For: 0.98.20 > > Attachments: HBASE-15805.0.98.00.patch, HBASE-15805.0.98.01.patch, > HBASE-15805_0.98.patch > > > Canary tools creates HBaseadmin object which which starts a thread to keep > pinging zookeeper on a regular interval. This cause delay when one of the > node in zookeeper is down and test timeouts. > Canary tool is creating HBaseadmin whose isEnabled and tableExists calls > starts a catalog tacker --> retries for zookeeper connection. This zookeeper > (6s) retry happens for multiple times for scanning 390+ region it is causing > lot of sleeps. By removing the failed zookeeper from the test it actually > took not more than 9 sec where it was taking almost 8 mins. Since "sniff" > calls each table one by one thus causes the delay in the test. > {code} > 2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG > zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input > java.net.SocketException: Socket is not connected > at sun.nio.ch.Net.translateToSocketException(Net.java:123) > at sun.nio.ch.Net.translateException(Net.java:157) > at sun.nio.ch.Net.translateException(Net.java:163) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) > at > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) > Caused by: java.nio.channels.NotYetConnectedException > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) > ... 4 more > 2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG > zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output > java.net.SocketException: Socket is not connected > at sun.nio.ch.Net.translateToSocketException(Net.java:123) > at sun.nio.ch.Net.translateException(Net.java:157) > at sun.nio.ch.Net.translateException(Net.java:163) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) > at > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) > Caused by: java.nio.channels.NotYetConnectedException > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15805) Canary test timeout if there is failed or shutdown zookeeper node in a quorum
[ https://issues.apache.org/jira/browse/HBASE-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15805: --- Summary: Canary test timeout if there is failed or shutdown zookeeper node in a quorum (was: Canary test timesout if there is failed\shutdown zookeeper node in a quorum) > Canary test timeout if there is failed or shutdown zookeeper node in a quorum > - > > Key: HBASE-15805 > URL: https://issues.apache.org/jira/browse/HBASE-15805 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 0.98.20 >Reporter: Vishal Khandelwal > Fix For: 0.98.20 > > Attachments: HBASE-15805.0.98.00.patch, HBASE-15805.0.98.01.patch, > HBASE-15805_0.98.patch > > > Canary tools creates HBaseadmin object which which starts a thread to keep > pinging zookeeper on a regular interval. This cause delay when one of the > node in zookeeper is down and test timeouts. > Canary tool is creating HBaseadmin whose isEnabled and tableExists calls > starts a catalog tacker --> retries for zookeeper connection. This zookeeper > (6s) retry happens for multiple times for scanning 390+ region it is causing > lot of sleeps. By removing the failed zookeeper from the test it actually > took not more than 9 sec where it was taking almost 8 mins. Since "sniff" > calls each table one by one thus causes the delay in the test. > {code} > 2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG > zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input > java.net.SocketException: Socket is not connected > at sun.nio.ch.Net.translateToSocketException(Net.java:123) > at sun.nio.ch.Net.translateException(Net.java:157) > at sun.nio.ch.Net.translateException(Net.java:163) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) > at > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) > Caused by: java.nio.channels.NotYetConnectedException > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) > ... 4 more > 2016-04-29 09:52:13 [Thread-6-SendThread()] DEBUG > zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output > java.net.SocketException: Socket is not connected > at sun.nio.ch.Net.translateToSocketException(Net.java:123) > at sun.nio.ch.Net.translateException(Net.java:157) > at sun.nio.ch.Net.translateException(Net.java:163) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) > at > org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) > at > org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1236) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1169) > Caused by: java.nio.channels.NotYetConnectedException > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)