[ https://issues.apache.org/jira/browse/HBASE-23731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bharath Vissapragada resolved HBASE-23731. ------------------------------------------ Fix Version/s: HBASE-18095 Resolution: Fixed Thanks [~stack] and [~janh] for the review. > TestFromClientSide flakey after junit upgrade > --------------------------------------------- > > Key: HBASE-23731 > URL: https://issues.apache.org/jira/browse/HBASE-23731 > Project: HBase > Issue Type: Bug > Components: test > Affects Versions: HBASE-18095 > Reporter: Bharath Vissapragada > Assignee: Bharath Vissapragada > Priority: Major > Labels: flaky-test > Fix For: HBASE-18095 > > > The cluster shutdown and restart between parameterized runs causes a thread > leak with stacks like follows. > {noformat} > 2020-01-22 09:52:10,900 WARN [LeaseRenewer:hostname@localhost:62252] > impl.LeaseRenewer(438): Failed to renew lease for > [DFSClient_NONMAPREDUCE_-1123200667_19] for 73 seconds. Will retry shortly > ... > java.net.ConnectException: Call From xxxxx/xxxx to localhost:62252 failed on > connection exception: java.net.ConnectException: Connection refused; For more > details see: http://wiki.apache.org/hadoop/ConnectionRefused > at sun.reflect.GeneratedConstructorAccessor64.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493) > at org.apache.hadoop.ipc.Client.call(Client.java:1435) > at org.apache.hadoop.ipc.Client.call(Client.java:1345) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy29.renewLease(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:613) > at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) > at com.sun.proxy.$Proxy30.renewLease(Unknown Source) > at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) > at com.sun.proxy.$Proxy33.renewLease(Unknown Source) > at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) > at com.sun.proxy.$Proxy33.renewLease(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:560) > at > org.apache.hadoop.hdfs.client.impl.LeaseRenewer.renew(LeaseRenewer.java:395) > at > org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:415) > at > org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) > at > org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) > at java.lang.Thread.run(Thread.java:748) > {noformat} > While the issue existed even before, it looks like junit 4.12 somehow was > masking it. Upgrade to 4.13 made it worse. The host:port refers to already > shutdown hdfs cluster, so ideally this thread shouldn't be there in the first > place. A culprit could be leaking filesystem / dfsclient objects. -- This message was sent by Atlassian Jira (v8.3.4#803005)