[ https://issues.apache.org/jira/browse/HBASE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656459#comment-16656459 ]
Guanghao Zhang commented on HBASE-21325: ---------------------------------------- Write a ut for this case. And found the regionserver not hang in waitOnAllRegionsToClose. As we will break the loop even there are online regions. {code:java} // No regions in RIT, we could stop waiting now. if (this.regionsInTransitionInRS.isEmpty()) { if (!isOnlineRegionsEmpty()) { LOG.info("We were exiting though online regions are not empty," + " because some regions failed closing"); } break; } {code} 2018-10-19 16:26:28,449 INFO [RS:1;hao-OptiPlex-7050:37602] regionserver.HRegionServer(1426): We were exiting though online regions are not empty, because some regions failed closing But the regionserver still hang in shutdown wal when stop. {code:java} "RS:1;hao-OptiPlex-7050:37602" daemon prio=5 tid=380 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:224) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.shutdown(AbstractFSWAL.java:821) at org.apache.hadoop.hbase.wal.SyncReplicationWALProvider.shutdown(SyncReplicationWALProvider.java:225) at org.apache.hadoop.hbase.wal.WALFactory.shutdown(WALFactory.java:246) at org.apache.hadoop.hbase.regionserver.HRegionServer.shutdownWAL(HRegionServer.java:1459) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1115) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) {code} > Add a max wait time for waitOnAllRegionsToClose > ----------------------------------------------- > > Key: HBASE-21325 > URL: https://issues.apache.org/jira/browse/HBASE-21325 > Project: HBase > Issue Type: Improvement > Reporter: Duo Zhang > Assignee: Guanghao Zhang > Priority: Major > > When testing sync replication, I found that, if I transit the remote cluster > to DA, while the local cluster is still in A, the region server will hang > when shutdown. As the fsOk flag only test the local cluster(which is > reasonable), we will enter the waitOnAllRegionsToClose, and since the WAL is > broken(the remote wal directory is gone) so we will never succeed. And this > lead to an infinite wait inside waitOnAllRegionsToClose. > So I think here we should have an upper bound for the wait time in > waitOnAllRegionsToClose method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)