[ https://issues.apache.org/jira/browse/HBASE-25905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452162#comment-17452162 ]
Hudson commented on HBASE-25905: -------------------------------- Results for branch branch-2.4 [build #248 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/248/]: (x) *{color:red}-1 overall{color}* ---- details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/248/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/248/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/248/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/248/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Shutdown of WAL stuck at waitForSafePoint > ----------------------------------------- > > Key: HBASE-25905 > URL: https://issues.apache.org/jira/browse/HBASE-25905 > Project: HBase > Issue Type: Bug > Components: regionserver, wal > Affects Versions: 3.0.0-alpha-1, 2.0.0 > Reporter: Xiaolin Ha > Assignee: Duo Zhang > Priority: Blocker > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9 > > Attachments: rs-jstack1, rs-jstack2, wal-stuck-error-logs.png > > > We use the fan-out HDFS OutputStream and AsyncFSWAL on our clusters, but met > the problem than RS can not exit completely for several hours util manual > interventions. > The two jstacks below show that the regionserver thread can waiting > unlimitedly in both > AsyncFSWAL#waitForSafePoint() > {code:java} > "regionserver/gh-data-hbase-finance08.mt/10.22.179.24:16020" #29 prio=5 > os_prio=0 tid=0x00007fb2feb5c000 nid=0xa92b waiting on condition > [0x00007f9ccb992000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007faea229a9d0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.waitForSafePoint(AsyncFSWAL.java:687) > at > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.doShutdown(AsyncFSWAL.java:743) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.shutdown(AbstractFSWAL.java:900) > at > org.apache.hadoop.hbase.wal.AbstractFSWALProvider.shutdown(AbstractFSWALProvider.java:182) > at > org.apache.hadoop.hbase.wal.RegionGroupingProvider.shutdown(RegionGroupingProvider.java:232) > at > org.apache.hadoop.hbase.wal.WALFactory.shutdown(WALFactory.java:271) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.shutdownWAL(HRegionServer.java:1405) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1147) > at java.lang.Thread.run(Thread.java:745) > {code} > and the log roller stuck at waiting for lock > {code:java} > "regionserver/gh-data-hbase-finance08.mt/10.22.179.24:16020.logRoller" #322 > daemon prio=5 os_prio=0 tid=0x00007fb2e11a4000 nid=0xa953 waiting on > condition [0x00007f9cbd9f1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007faea1217048> (a > java.util.concurrent.locks.ReentrantLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:224) > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:822) > at > org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:269) > at > org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:186){code} > -- This message was sent by Atlassian Jira (v8.20.1#820001)