[jira] [Resolved] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently

[Ayush Saxena (Jira)]

     [ 
https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved HDFS-15456.
---------------------------------
    Fix Version/s: 3.4.0
     Hadoop Flags: Reviewed
       Resolution: Fixed

Merged PR to trunk. Thanx Everyone

> TestExternalStoragePolicySatisfier fails intermittently
> -------------------------------------------------------
>
>                 Key: HDFS-15456
>                 URL: https://issues.apache.org/jira/browse/HDFS-15456
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ahmed Hussein
>            Assignee: Leon Gao
>            Priority: Major
>              Labels: pull-request-available, test
>             Fix For: 3.4.0
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk 
> {code:bash}
> [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 421.443 s <<< FAILURE! - in 
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
> [ERROR] 
> testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier)
>   Time elapsed: 43.983 s  <<< ERROR!
> java.util.concurrent.TimeoutException: 
> Timed out waiting for condition. Thread diagnostics:
> Timestamp: 2020-07-07 07:51:10,267
> "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1217 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
>         at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "pool-90-thread-1"  prio=5 tid=1069 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@69d6b99d" daemon 
> prio=5 tid=1181 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:113)
>         at 
> org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:85)
>         at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:226)
>         at java.lang.Thread.run(Thread.java:748)
> "LeaseRenewer:jenkins@localhost:33273" daemon prio=5 tid=261 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:412)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:308)
>         at java.lang.Thread.run(Thread.java:748)
> "process reaper" daemon prio=10 tid=204 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@2984014d" daemon prio=5 
> tid=1123 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
>         at java.lang.Thread.run(Thread.java:748)
> "Session-HouseKeeper-5896c743"  prio=5 tid=1155 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1073" daemon prio=5 tid=1073 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data5)"
>  daemon prio=5 tid=1242 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "Signal Dispatcher" daemon prio=9 tid=4 runnable
> java.lang.Thread.State: RUNNABLE
> "qtp1442156271-1074-acceptor-0@3d9b409f-ServerConnector@6001a0b2{HTTP/1.1,[http/1.1]}{localhost:34409}"
>  daemon prio=3 tid=1074 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "java.util.concurrent.ThreadPoolExecutor$Worker@42139ef8[State = -1, empty 
> queue]" daemon prio=5 tid=1241 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "nioEventLoopGroup-14-1"  prio=10 tid=1122 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
>         at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>         at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>         at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:748)
> "StorageLocationChecker thread 0" daemon prio=5 tid=1110 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 38495" daemon prio=5 tid=1202 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "pool-89-thread-1"  prio=5 tid=1059 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "StorageLocationChecker thread 1" daemon prio=5 tid=1145 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1062" daemon prio=5 tid=1062 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1065" daemon prio=5 tid=1065 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "pool-106-thread-1"  prio=5 tid=1150 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "ForkJoinPool-2-worker-29" daemon prio=5 tid=229 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "GcTimeMonitor obsWindow = 60000, sleepInterval = 5000, maxGcTimePerc = 100" 
> daemon prio=5 tid=1058 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.util.GcTimeMonitor.run(GcTimeMonitor.java:155)
> "IPC Server handler 6 on default port 38495" daemon prio=5 tid=1206 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2982dc5" daemon 
> prio=5 tid=1114 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:113)
>         at 
> org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:85)
>         at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:226)
>         at java.lang.Thread.run(Thread.java:748)
> "StorageLocationChecker thread 1" daemon prio=5 tid=1111 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1067" daemon prio=5 tid=1067 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "DataNode DiskChecker thread 1" daemon prio=5 tid=1254 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 4 on default port 38495" daemon prio=5 tid=1204 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "StorageLocationChecker thread 0" daemon prio=5 tid=1177 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server idle connection scanner for port 43691" daemon prio=5 tid=1086 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         at java.util.TimerThread.run(Timer.java:505)
> "IPC Server handler 5 on default port 37995" daemon prio=5 tid=1172 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server Responder" daemon prio=5 tid=1194 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1512)
>         at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1495)
> "IPC Server handler 9 on default port 43691" daemon prio=5 tid=1102 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server handler 3 on default port 43691" daemon prio=5 tid=1096 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "nioEventLoopGroup-18-1"  prio=10 tid=1189 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
>         at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>         at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>         at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:748)
> "BlockMoverTask-0" daemon prio=5 tid=711 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp203019717-1119" daemon prio=5 tid=1119 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "BlocksStorageMovementAttemptMonitor" daemon prio=5 tid=1262 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.sps.BlockStorageMovementAttemptedItems$BlocksStorageMovementAttemptMonitor.run(BlockStorageMovementAttemptedItems.java:233)
>         at java.lang.Thread.run(Thread.java:748)
> "pool-95-thread-1" daemon prio=5 tid=1222 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 44933" daemon prio=5 tid=1136 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@701fb2e8"
>  daemon prio=5 tid=1108 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4562)
>         at java.lang.Thread.run(Thread.java:748)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1232 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 9 on default port 44933" daemon prio=5 tid=1143 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-8" daemon prio=5 tid=232 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "IPC Server handler 7 on default port 43691" daemon prio=5 tid=1100 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "Block report processor" daemon prio=5 tid=1081 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:5299)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5288)
> "IPC Server listener on 0" daemon prio=5 tid=1191 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1335)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data6)"
>  daemon prio=5 tid=1243 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "pool-111-thread-1" daemon prio=5 tid=1253 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server idle connection scanner for port 37995" daemon prio=5 tid=1160 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         at java.util.TimerThread.run(Timer.java:505)
> "datanode DomainSocketWatcher" daemon prio=5 tid=1149 runnable
> java.lang.Thread.State: RUNNABLE
>         at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native 
> Method)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:503)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1490000461-1152-acceptor-0@758525bc-ServerConnector@29d326dc{HTTP/1.1,[http/1.1]}{localhost:38691}"
>  daemon prio=3 tid=1152 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "DataNode DiskChecker thread 0" daemon prio=5 tid=1236 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1066" daemon prio=5 tid=1066 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 2 on default port 43691" daemon prio=5 tid=1095 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "pool-107-thread-1"  prio=5 tid=1166 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 0 on default port 44933" daemon prio=5 tid=1134 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server handler 5 on default port 44933" daemon prio=5 tid=1139 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "qtp203019717-1118-acceptor-0@5429938-ServerConnector@3775d9c7{HTTP/1.1,[http/1.1]}{localhost:36417}"
>  daemon prio=3 tid=1118 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "Session-HouseKeeper-4cf7dd4d"  prio=5 tid=1188 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 6 on default port 43691" daemon prio=5 tid=1099 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "StorageLocationChecker thread 0" daemon prio=5 tid=1144 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "StoragePolicySatisfier" daemon prio=5 tid=1261 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier.run(StoragePolicySatisfier.java:316)
>         at java.lang.Thread.run(Thread.java:748)
> "datanode DomainSocketWatcher" daemon prio=5 tid=1182 runnable
> java.lang.Thread.State: RUNNABLE
>         at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native 
> Method)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:503)
>         at java.lang.Thread.run(Thread.java:748)
> "main"  prio=5 tid=1 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:204)
>         at 
> org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141)
>         at 
> org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127)
>         at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>         at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>         at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>         at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>         at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>         at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>         at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>         at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>         at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> "IPC Server handler 1 on default port 38495" daemon prio=5 tid=1201 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-25" daemon prio=5 tid=225 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4)"
>  daemon prio=5 tid=1227 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "StorageLocationChecker thread 1" daemon prio=5 tid=1178 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1063" daemon prio=5 tid=1063 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "BlockMoverTask-1" daemon prio=5 tid=712 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1076-acceptor-2@7ea21e77-ServerConnector@6001a0b2{HTTP/1.1,[http/1.1]}{localhost:34409}"
>  daemon prio=3 tid=1076 blocked
> java.lang.Thread.State: BLOCKED
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:231)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "BlockStorageMovementTracker" daemon prio=5 tid=256 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
>         at 
> org.apache.hadoop.hdfs.server.common.sps.BlockStorageMovementTracker.run(BlockStorageMovementTracker.java:63)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Client (1194919396) connection to localhost/127.0.0.1:43691 from 
> jenkins" daemon prio=5 tid=1258 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1051)
>         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1095)
> "IPC Server handler 0 on default port 37995" daemon prio=5 tid=1167 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "Session-HouseKeeper-679230f0"  prio=5 tid=1121 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@1a120c28"
>  daemon prio=5 tid=1106 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4424)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" 
> daemon prio=5 tid=87 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
>         at 
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3890)
>         at java.lang.Thread.run(Thread.java:748)
> "Finalizer" daemon prio=8 tid=3 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1218 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp749170575-1187" daemon prio=5 tid=1187 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1061" daemon prio=5 tid=1061 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "pool-92-thread-1"  prio=5 tid=1103 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@315a2da3"
>  daemon prio=5 tid=1079 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:542)
>         at java.lang.Thread.run(Thread.java:748)
> "DataNode DiskChecker thread 0" daemon prio=5 tid=1221 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 8 on default port 37995" daemon prio=5 tid=1175 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "java.util.concurrent.ThreadPoolExecutor$Worker@67d0e27e[State = -1, empty 
> queue]" daemon prio=5 tid=1225 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 5 on default port 43691" daemon prio=5 tid=1098 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "DataNode DiskChecker thread 1" daemon prio=5 tid=1238 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "ForkJoinPool-2-worker-18" daemon prio=5 tid=226 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "qtp1442156271-1075-acceptor-1@465c6d21-ServerConnector@6001a0b2{HTTP/1.1,[http/1.1]}{localhost:34409}"
>  daemon prio=3 tid=1075 blocked
> java.lang.Thread.State: BLOCKED
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:231)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "pool-99-thread-1"  prio=5 tid=1132 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor@4f3d6f2"
>  daemon prio=5 tid=1090 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor.run(PendingReconstructionBlocks.java:259)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 7 on default port 44933" daemon prio=5 tid=1141 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-26" daemon prio=5 tid=234 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1)"
>  daemon prio=5 tid=1211 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "qtp1490000461-1153" daemon prio=5 tid=1153 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "BlockMoverTask-0" daemon prio=5 tid=2779 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 1 on default port 37995" daemon prio=5 tid=1168 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "qtp1442156271-1071" daemon prio=5 tid=1071 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@3584946" daemon prio=5 
> tid=1057 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server Responder" daemon prio=5 tid=1087 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1512)
>         at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1495)
> "GcTimeMonitor obsWindow = 60000, sleepInterval = 5000, maxGcTimePerc = 100" 
> daemon prio=5 tid=34 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.util.GcTimeMonitor.run(GcTimeMonitor.java:155)
> "IPC Server handler 9 on default port 37995" daemon prio=5 tid=1176 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1233 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b137f60" daemon 
> prio=5 tid=1148 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:113)
>         at 
> org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:85)
>         at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:226)
>         at java.lang.Thread.run(Thread.java:748)
> "CacheReplicationMonitor(1144697033)" daemon prio=5 tid=1109 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181)
> "IPC Server listener on 0" daemon prio=5 tid=1158 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1335)
> "BP-912129709-172.17.0.2-1594151429636 heartbeating to 
> localhost/127.0.0.1:43691" daemon prio=5 tid=1131 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:158)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:878)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 8 on default port 38495" daemon prio=5 tid=1208 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server handler 1 on default port 44933" daemon prio=5 tid=1135 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2)"
>  daemon prio=5 tid=1212 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "surefire-forkedjvm-ping-30s" daemon prio=5 tid=19 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1072" daemon prio=5 tid=1072 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp749170575-1186" daemon prio=5 tid=1186 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server idle connection scanner for port 38495" daemon prio=5 tid=1193 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         at java.util.TimerThread.run(Timer.java:505)
> "Session-HouseKeeper-48385a80"  prio=5 tid=1068 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@7651ea64"
>  daemon prio=5 tid=1107 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4466)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 0 on default port 43691" daemon prio=5 tid=1093 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "qtp1490000461-1151" daemon prio=5 tid=1151 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 7 on default port 37995" daemon prio=5 tid=1174 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server Responder" daemon prio=5 tid=1127 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1512)
>         at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1495)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1159 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
>         at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "Command processor" daemon prio=5 tid=1164 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1324)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
> "GcTimeMonitor obsWindow = 60000, sleepInterval = 5000, maxGcTimePerc = 100" 
> daemon prio=5 tid=557 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.util.GcTimeMonitor.run(GcTimeMonitor.java:155)
> "IPC Server handler 4 on default port 37995" daemon prio=5 tid=1171 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Client (1194919396) connection to localhost/127.0.0.1:43691 from 
> jenkins" daemon prio=5 tid=1133 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1051)
>         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1095)
> "BlockStorageMovementTracker" daemon prio=5 tid=1260 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
>         at 
> org.apache.hadoop.hdfs.server.common.sps.BlockStorageMovementTracker.run(BlockStorageMovementTracker.java:63)
>         at java.lang.Thread.run(Thread.java:748)
> "BP-912129709-172.17.0.2-1594151429636 heartbeating to 
> localhost/127.0.0.1:43691" daemon prio=5 tid=1165 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:158)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:878)
>         at java.lang.Thread.run(Thread.java:748)
> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@4bd47e90" daemon prio=5 
> tid=1157 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
>         at java.lang.Thread.run(Thread.java:748)
> "surefire-forkedjvm-command-thread" daemon prio=5 tid=18 runnable
> java.lang.Thread.State: RUNNABLE
>         at java.io.FileInputStream.readBytes(Native Method)
>         at java.io.FileInputStream.read(FileInputStream.java:255)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>         at 
> org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
>         at 
> org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:390)
>         at java.lang.Thread.run(Thread.java:748)
> "Reference Handler" daemon prio=10 tid=2 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data6/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1249 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 8 on default port 43691" daemon prio=5 tid=1101 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "qtp1442156271-1060" daemon prio=5 tid=1060 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1070" daemon prio=5 tid=1070 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "Listener at localhost/38495" daemon prio=5 tid=1048 runnable
> java.lang.Thread.State: RUNNABLE
>         at java.lang.Thread.dumpThreads(Native Method)
>         at java.lang.Thread.getAllStackTraces(Thread.java:1610)
>         at 
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
>         at 
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
>         at 
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:401)
>         at 
> org.apache.hadoop.hdfs.DFSTestUtil.waitExpectedStorageType(DFSTestUtil.java:2536)
>         at 
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier.testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(TestExternalStoragePolicySatisfier.java:1064)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>         at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>         at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>         at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 4 on default port 43691" daemon prio=5 tid=1097 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3)"
>  daemon prio=5 tid=1226 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:656)
> "java.util.concurrent.ThreadPoolExecutor$Worker@7436c62f[State = -1, empty 
> queue]" daemon prio=5 tid=1257 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "Command processor" daemon prio=5 tid=1197 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1324)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
> "RedundancyMonitor" daemon prio=5 tid=1077 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at java.lang.Thread.sleep(Thread.java:340)
>         at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4941)
>         at java.lang.Thread.run(Thread.java:748)
> "ForkJoinPool-2-worker-4" daemon prio=5 tid=228 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data5/current/BP-912129709-172.17.0.2-1594151429636"
>  daemon prio=5 tid=1248 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205)
>         at java.lang.Thread.run(Thread.java:748)
> "AsyncAppender-Dispatcher-Thread-47" daemon prio=5 tid=79 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at 
> org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1490000461-1154" daemon prio=5 tid=1154 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 3 on default port 37995" daemon prio=5 tid=1170 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1085 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
>         at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "Timer for 'NameNode' metrics system" daemon prio=5 tid=1056 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         at java.util.TimerThread.run(Timer.java:505)
> "IPC Server Responder" daemon prio=5 tid=1161 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1512)
>         at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1495)
> "StorageInfoMonitor" daemon prio=5 tid=1078 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.run(BlockManager.java:4976)
>         at java.lang.Thread.run(Thread.java:748)
> "LeaseRenewer:jenkins@localhost:43691" daemon prio=5 tid=1265 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:412)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
>         at 
> org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:308)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 0 on default port 38495" daemon prio=5 tid=1200 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@12e80396" daemon prio=5 
> tid=1190 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
>         at java.lang.Thread.run(Thread.java:748)
> "process reaper" daemon prio=10 tid=202 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp749170575-1184" daemon prio=5 tid=1184 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 1 on default port 43691" daemon prio=5 tid=1094 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "pool-115-thread-1"  prio=5 tid=1199 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "DatanodeAdminMonitor-0" daemon prio=5 tid=1091 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server listener on 0" daemon prio=5 tid=1124 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1335)
> "qtp203019717-1120" daemon prio=5 tid=1120 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:851)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:890)
>         at java.lang.Thread.run(Thread.java:748)
> "BlockStorageMovementTracker" daemon prio=5 tid=611 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
>         at 
> org.apache.hadoop.hdfs.server.common.sps.BlockStorageMovementTracker.run(BlockStorageMovementTracker.java:63)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 9 on default port 38495" daemon prio=5 tid=1209 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server handler 3 on default port 38495" daemon prio=5 tid=1203 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "pool-103-thread-1" daemon prio=5 tid=1237 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 3 on default port 44933" daemon prio=5 tid=1137 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Parameter Sending Thread #0" daemon prio=5 tid=158 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Parameter Sending Thread #1" daemon prio=5 tid=615 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "ForkJoinPool-2-worker-1" daemon prio=5 tid=233 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "pool-98-thread-1"  prio=5 tid=1116 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "AsyncAppender-Dispatcher-Thread-68" daemon prio=5 tid=110 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at 
> org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
>         at java.lang.Thread.run(Thread.java:748)
> "client DomainSocketWatcher" daemon prio=5 tid=251 runnable
> java.lang.Thread.State: RUNNABLE
>         at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native 
> Method)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:503)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 6 on default port 44933" daemon prio=5 tid=1140 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Server handler 6 on default port 37995" daemon prio=5 tid=1173 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "IPC Client (1194919396) connection to localhost/127.0.0.1:43691 from 
> jenkins" daemon prio=5 tid=1210 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1051)
>         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1095)
> "DataNode DiskChecker thread 0" daemon prio=5 tid=1252 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>         at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>         at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server listener on 0" daemon prio=5 tid=1084 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1335)
> "SPSPathIdProcessor" daemon prio=5 tid=1263 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.sps.BlockStorageMovementNeeded$SPSPathIdProcessor.run(BlockStorageMovementNeeded.java:243)
>         at java.lang.Thread.run(Thread.java:748)
> "datanode DomainSocketWatcher" daemon prio=5 tid=1115 runnable
> java.lang.Thread.State: RUNNABLE
>         at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native 
> Method)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
>         at 
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:503)
>         at java.lang.Thread.run(Thread.java:748)
> "Command processor" daemon prio=5 tid=1130 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1324)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
> "qtp749170575-1185-acceptor-0@66f7d789-ServerConnector@5ad06c66{HTTP/1.1,[http/1.1]}{localhost:43865}"
>  daemon prio=3 tid=1185 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:419)
>         at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:247)
>         at 
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
>         at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:664)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server idle connection scanner for port 44933" daemon prio=5 tid=1126 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at java.util.TimerThread.mainLoop(Timer.java:552)
>         at java.util.TimerThread.run(Timer.java:505)
> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@4bb65341" daemon 
> prio=5 tid=1105 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Thread.sleep(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:545)
>         at java.lang.Thread.run(Thread.java:748)
> "FSEditLogAsync" daemon prio=5 tid=1083 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at 
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>         at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.dequeueEdit(FSEditLogAsync.java:221)
>         at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:229)
>         at java.lang.Thread.run(Thread.java:748)
> "pool-114-thread-1"  prio=5 tid=1183 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp1442156271-1064" daemon prio=5 tid=1064 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 5 on default port 38495" daemon prio=5 tid=1205 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "BP-912129709-172.17.0.2-1594151429636 heartbeating to 
> localhost/127.0.0.1:43691" daemon prio=5 tid=1198 timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at java.lang.Object.wait(Native Method)
>         at 
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:158)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:878)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 7 on default port 38495" daemon prio=5 tid=1207 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "nioEventLoopGroup-16-1"  prio=10 tid=1156 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
>         at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
>         at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>         at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>         at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:748)
> "qtp203019717-1117" daemon prio=5 tid=1117 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:466)
>         at 
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:403)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>         at 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
>         at 
> org.eclipse.jetty.io.ManagedSelector$$Lambda$52/778240797.run(Unknown Source)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
>         at java.lang.Thread.run(Thread.java:748)
> "IPC Server handler 8 on default port 44933" daemon prio=5 tid=1142 
> timed_waiting
> java.lang.Thread.State: TIMED_WAITING
>         at sun.misc.Unsafe.park(Native Method)
>         at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
>         at 
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918)
> "ForkJoinPool-2-worker-22" daemon prio=5 tid=230 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> "Socket Reader #1 for port 0" daemon prio=5 tid=1125 runnable
> java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at 
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273)
>         at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252)
> "ForkJoinPool-2-worker-11" daemon prio=5 tid=227 in Object.wait()
> java.lang.Thread.State: WAITING (on object monitor)
>         at sun.misc.Unsafe.park(Native Method)
>         at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
>         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
>         at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
>       at 
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:401)
>       at 
> org.apache.hadoop.hdfs.DFSTestUtil.waitExpectedStorageType(DFSTestUtil.java:2536)
>       at 
> org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier.testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(TestExternalStoragePolicySatisfier.java:1064)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:498)
>       at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>       at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>       at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>       at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>       at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>       at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>       at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org