[ https://issues.apache.org/jira/browse/SPARK-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabiano Francesconi closed SPARK-15558. --------------------------------------- Resolution: Fixed Tried with Spark-2.0.0 compiled from the 2.x branch and this problem seems to be gone. Thank you! > Deadlock when retreiving shuffled cached data > --------------------------------------------- > > Key: SPARK-15558 > URL: https://issues.apache.org/jira/browse/SPARK-15558 > Project: Spark > Issue Type: Bug > Affects Versions: 1.6.1 > Reporter: Fabiano Francesconi > Attachments: screenshot-1.png > > > Spark-1.6.1-bin-hadoop2.6 hangs when trying to retrieving shuffled cached > data from another host. The job I am currently executing is fetching data > using async actions and persisting these RDDs into main memory (they all > fit). Later on, at the point in which it is currently hanging, the > application is retrieving this cached data but it hangs. The application, > once the timeout set in the Await.results call is met, crashes. > This problem is reproducible at every executing, although the point in which > it hangs it is not. > I have also tried activating: > {code} > spark.memory.useLegacyMode=true > {code} > as mentioned in SPARK-13566 guessing a similar deadlock as the one given > between MemoryStore and BlockManager. Unfortunately, this didn't help. > The only plausible (albeit debatable) solution would be to use speculation > mode. > Configuration: > {code} > /usr/local/tl/spark-latest/bin/spark-submit \ > --executor-memory 80G \ > --total-executor-cores 90 \ > --driver-memory 8G \ > {code} > Stack trace: > {code} > "sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-55" #293 > daemon prio=5 os_prio=0 tid=0x00007f99d4004000 nid=0x4e80 waiting on > condition [0x00007f9946bfb000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b541a6570> (a > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) > at > scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > "sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-54" #292 > daemon prio=5 os_prio=0 tid=0x00007f99d4002000 nid=0x4e6d waiting on > condition [0x00007f98c86b6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b541a6570> (a > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > "Executor task launch worker-43" #236 daemon prio=5 os_prio=0 > tid=0x00007f9950001800 nid=0x4acc waiting on condition [0x00007f9a2c4be000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007fab3f081300> (a > scala.concurrent.impl.Promise$CompletionLatch) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at > scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202) > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:102) > at > org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:588) > at > org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:585) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.storage.BlockManager.doGetRemote(BlockManager.scala:585) > at org.apache.spark.storage.BlockManager.getRemote(BlockManager.scala:570) > at org.apache.spark.storage.BlockManager.get(BlockManager.scala:630) > at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:44) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "ShortCircuitCache_SlotReleaser" #218 daemon prio=5 os_prio=0 > tid=0x00007f989c001000 nid=0x4a7b waiting on condition [0x00007f98c8fbf000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b578ee248> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-3" #130 daemon prio=5 os_prio=0 tid=0x00007f98c0022000 > nid=0x4a31 runnable [0x00007f98cadcc000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b543d0aa8> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b543d4b60> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b543d09c0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-3" #138 daemon prio=5 os_prio=0 tid=0x00007f98dc004800 > nid=0x4992 runnable [0x00007f99468f8000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b5407a858> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b54095710> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b5407a770> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-2" #129 daemon prio=5 os_prio=0 tid=0x00007f9888003000 > nid=0x4979 runnable [0x00007f98c8ebe000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b5428fb58> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b54293c10> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b5428fa70> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-2" #137 daemon prio=5 os_prio=0 tid=0x00007f98dc003000 > nid=0x4974 runnable [0x00007f98ca3c4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b54098010> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b540ada48> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b54097f28> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "ShortCircuitCache_Cleaner" #164 daemon prio=5 os_prio=0 > tid=0x00007f993414f000 nid=0x496e waiting on condition [0x00007f98ca9ca000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b578edd80> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "org.apache.hadoop.hdfs.PeerCache@7f02ca" #163 daemon prio=5 os_prio=0 > tid=0x00007f9934110000 nid=0x496d waiting on condition [0x00007f98caacb000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) > at java.lang.Thread.run(Thread.java:745) > "Thread-6" #158 daemon prio=5 os_prio=0 tid=0x00007f9930014800 nid=0x4966 > runnable [0x00007f98cb3d0000] > java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52) > at > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:459) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-1" #136 daemon prio=5 os_prio=0 tid=0x00007f98dc001800 > nid=0x4961 runnable [0x00007f9a2c8c3000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540b0348> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b540bc388> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540b0260> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-1" #128 daemon prio=5 os_prio=0 tid=0x00007f98d4025000 > nid=0x4960 runnable [0x00007f9a2c6c1000] > java.lang.Thread.State: RUNNABLE > at sun.misc.Unsafe.setMemory(Native Method) > at sun.misc.Unsafe.setMemory(Unsafe.java:529) > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:132) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) > at > io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651) > at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:215) > at io.netty.buffer.PoolArena.allocate(PoolArena.java:132) > at > io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271) > at > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > at > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) > at > io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1345) > at > io.netty.buffer.CompositeByteBuf.consolidateIfNeeded(CompositeByteBuf.java:276) > at > io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:116) > at > org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148) > at > org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:82) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-0" #127 daemon prio=5 os_prio=0 tid=0x00007f994c13b800 > nid=0x495e runnable [0x00007f9a2c5c0000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b51fca258> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b51fca288> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b51fca1e0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "driver-heartbeater" #143 daemon prio=5 os_prio=0 tid=0x00007f99581a7000 > nid=0x4953 waiting on condition [0x00007f99445ea000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b540b0528> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-0" #135 daemon prio=5 os_prio=0 tid=0x00007f995819e000 > nid=0x4952 runnable [0x00007f99446eb000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540a8778> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b540a87a8> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540a8700> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "BROADCAST_VARS cleanup timer" #126 daemon prio=5 os_prio=0 > tid=0x00007faf609b9800 nid=0x4951 in Object.wait() [0x00007f99449ec000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00007f9b54081928> (a java.util.TaskQueue) > at java.lang.Object.wait(Object.java:502) > at java.util.TimerThread.mainLoop(Timer.java:526) > - locked <0x00007f9b54081928> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > "BLOCK_MANAGER cleanup timer" #125 daemon prio=5 os_prio=0 > tid=0x00007faf609bc000 nid=0x4950 in Object.wait() [0x00007f9944aed000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00007f9b54091148> (a java.util.TaskQueue) > at java.lang.Object.wait(Object.java:502) > at java.util.TimerThread.mainLoop(Timer.java:526) > - locked <0x00007f9b54091148> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > "shuffle-client-0" #100 daemon prio=5 os_prio=0 tid=0x00007f98f8001000 > nid=0x494f runnable [0x00007f9945def000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b519d9ca8> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00007f9b519ddd60> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b519d9bc0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "netty-rpc-env-timeout" #123 daemon prio=5 os_prio=0 tid=0x00007faf6097e800 > nid=0x494e waiting on condition [0x00007f9945ef0000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b5095e620> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O server boss #6" #121 daemon prio=5 os_prio=0 tid=0x00007f990c0f1800 > nid=0x494c runnable [0x00007f99460f2000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540b08f0> (a sun.nio.ch.Util$2) > - locked <0x00007f9b540b0908> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540b0878> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at > org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O worker #5" #120 daemon prio=5 os_prio=0 tid=0x00007f990c07e800 > nid=0x494b runnable [0x00007f99461f3000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b54082388> (a sun.nio.ch.Util$2) > - locked <0x00007f9b540823a0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b54082310> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O worker #4" #119 daemon prio=5 os_prio=0 tid=0x00007f990c07d800 > nid=0x494a runnable [0x00007f99462f4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540b0d48> (a sun.nio.ch.Util$2) > - locked <0x00007f9b540b0d60> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540b0cd0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O boss #3" #118 daemon prio=5 os_prio=0 tid=0x00007f990c07a800 > nid=0x4949 runnable [0x00007f99463f5000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540b1888> (a sun.nio.ch.Util$2) > - locked <0x00007f9b540b18a0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540b1810> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O worker #2" #116 daemon prio=5 os_prio=0 tid=0x00007f990c01b800 > nid=0x4948 runnable [0x00007f99464f6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b541790e0> (a sun.nio.ch.Util$2) > - locked <0x00007f9b541790f8> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b54179068> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "New I/O worker #1" #115 daemon prio=5 os_prio=0 tid=0x00007f990c023800 > nid=0x4947 runnable [0x00007f99465f7000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00007f9b540d6160> (a sun.nio.ch.Util$2) > - locked <0x00007f9b540d6178> (a java.util.Collections$UnmodifiableSet) > - locked <0x00007f9b540d60e8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "sparkExecutorActorSystem-akka.actor.default-dispatcher-3" #110 daemon prio=5 > os_prio=0 tid=0x00007faf60930000 nid=0x4942 waiting on condition > [0x00007f9946cfc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b5417bc68> (a > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > "sparkExecutorActorSystem-scheduler-1" #108 daemon prio=5 os_prio=0 > tid=0x00007faf608b1000 nid=0x4940 sleeping[0x00007f9b1dc00000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226) > at > akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405) > at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-31" #99 daemon prio=5 os_prio=0 tid=0x00007faf60783000 > nid=0x493f waiting on condition [0x00007f9a2ccc7000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-30" #98 daemon prio=5 os_prio=0 tid=0x00007faf60782000 > nid=0x493e waiting on condition [0x00007f9b1d2d1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-29" #97 daemon prio=5 os_prio=0 tid=0x00007faf60781000 > nid=0x493d waiting on condition [0x00007f9b1d1d0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-28" #96 daemon prio=5 os_prio=0 tid=0x00007faf60780000 > nid=0x493c waiting on condition [0x00007f9b1d0cf000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-27" #95 daemon prio=5 os_prio=0 tid=0x00007faf6077e800 > nid=0x493b waiting on condition [0x00007f9b1cfce000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-26" #94 daemon prio=5 os_prio=0 tid=0x00007faf6077d800 > nid=0x493a waiting on condition [0x00007f9b1cecd000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-25" #93 daemon prio=5 os_prio=0 tid=0x00007faf6077d000 > nid=0x4939 waiting on condition [0x00007f9b1cdcc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-24" #92 daemon prio=5 os_prio=0 tid=0x00007faf60779800 > nid=0x4938 waiting on condition [0x00007f9b1cccb000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-23" #91 daemon prio=5 os_prio=0 tid=0x00007faf60779000 > nid=0x4937 waiting on condition [0x00007f9b1cbca000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-22" #90 daemon prio=5 os_prio=0 tid=0x00007faf60778000 > nid=0x4936 waiting on condition [0x00007f9b1cac9000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-21" #89 daemon prio=5 os_prio=0 tid=0x00007faf60777800 > nid=0x4935 waiting on condition [0x00007f9b1c9c8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-20" #88 daemon prio=5 os_prio=0 tid=0x00007faf605ed800 > nid=0x4934 waiting on condition [0x00007f9b1c8c7000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-19" #87 daemon prio=5 os_prio=0 tid=0x00007faf6074c000 > nid=0x4933 waiting on condition [0x00007f9b1c7c6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-18" #86 daemon prio=5 os_prio=0 tid=0x00007faf60608000 > nid=0x4932 waiting on condition [0x00007f9b1c6c5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-17" #85 daemon prio=5 os_prio=0 tid=0x00007faf60627000 > nid=0x4931 waiting on condition [0x00007f9b1c5c4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-16" #84 daemon prio=5 os_prio=0 tid=0x00007faf60628800 > nid=0x4930 waiting on condition [0x00007f9b1c4c3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-15" #83 daemon prio=5 os_prio=0 tid=0x00007faf6062a800 > nid=0x492f waiting on condition [0x00007f9b1c3c2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-14" #82 daemon prio=5 os_prio=0 tid=0x00007faf60603000 > nid=0x492e waiting on condition [0x00007f9b1c2c1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-13" #81 daemon prio=5 os_prio=0 tid=0x00007faf60616000 > nid=0x492d waiting on condition [0x00007f9a2dffe000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-12" #80 daemon prio=5 os_prio=0 tid=0x00007faf60614000 > nid=0x492c waiting on condition [0x00007f9b1c1c0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-11" #79 daemon prio=5 os_prio=0 tid=0x00007faf60617800 > nid=0x492b waiting on condition [0x00007f9a2defd000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-10" #78 daemon prio=5 os_prio=0 tid=0x00007faf60619800 > nid=0x492a waiting on condition [0x00007f9a2ddfc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-9" #77 daemon prio=5 os_prio=0 tid=0x00007faf6061b000 > nid=0x4929 waiting on condition [0x00007f9a2dcfb000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-8" #76 daemon prio=5 os_prio=0 tid=0x00007faf6061c800 > nid=0x4928 waiting on condition [0x00007f9a2dbfa000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-7" #75 daemon prio=5 os_prio=0 tid=0x00007faf6061e800 > nid=0x4927 waiting on condition [0x00007f9a2daf9000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-6" #74 daemon prio=5 os_prio=0 tid=0x00007faf60620000 > nid=0x4926 waiting on condition [0x00007f9a2d9f8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-5" #73 daemon prio=5 os_prio=0 tid=0x00007faf60622000 > nid=0x4925 waiting on condition [0x00007f9a2d8f7000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-4" #72 daemon prio=5 os_prio=0 tid=0x00007faf60623800 > nid=0x4924 waiting on condition [0x00007f9a2d7f6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-3" #71 daemon prio=5 os_prio=0 tid=0x00007faf60625800 > nid=0x4922 waiting on condition [0x00007f9a2d6f5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-2" #70 daemon prio=5 os_prio=0 tid=0x00007faf60612800 > nid=0x4921 waiting on condition [0x00007f9a2d5f4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-1" #69 daemon prio=5 os_prio=0 tid=0x00007faf6060f000 > nid=0x4920 waiting on condition [0x00007f9a2d4f3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-0" #68 daemon prio=5 os_prio=0 tid=0x00007faf6060d800 > nid=0x491f waiting on condition [0x00007f9a2d3f2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50849100> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > "threadDeathWatcher-2-1" #67 daemon prio=1 os_prio=0 tid=0x00007f994c00c800 > nid=0x4917 sleeping[0x00007f9a2c9c4000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007faf60101000 > nid=0x48ee runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007faf600fb800 > nid=0x48ed waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007faf600fa000 > nid=0x48ec waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007faf600f7800 > nid=0x48eb waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007faf600f5800 > nid=0x48ea waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007faf600f3800 > nid=0x48e9 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007faf600f1800 > nid=0x48e8 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007faf600ef800 > nid=0x48e7 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007faf600ed000 > nid=0x48e6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007faf600eb000 > nid=0x48e5 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007faf600e9000 > nid=0x48e4 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007faf600de800 > nid=0x48e3 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007faf600dc800 > nid=0x48e2 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007faf600d8000 > nid=0x48e1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007faf600d6800 > nid=0x48e0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007faf600d3800 > nid=0x48df waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007faf600d1800 > nid=0x48de waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007faf600aa000 nid=0x48dd in > Object.wait() [0x00007f9b2c978000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0x00007f9b50819a78> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007faf600a5800 > nid=0x48dc in Object.wait() [0x00007f9b2ca79000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0x00007f9b5081f150> (a java.lang.ref.Reference$Lock) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > "main" #1 prio=5 os_prio=0 tid=0x00007faf6000a000 nid=0x48c3 waiting on > condition [0x00007faf672a0000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007f9b50924398> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465) > at > org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:182) > at > org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:275) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:200) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:151) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:253) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) > "VM Thread" os_prio=0 tid=0x00007faf6009e000 nid=0x48db runnable > "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007faf6001f800 nid=0x48c4 > runnable > "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007faf60021000 nid=0x48c5 > runnable > "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007faf60023000 nid=0x48c6 > runnable > "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007faf60024800 nid=0x48c7 > runnable > "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007faf60026800 nid=0x48c8 > runnable > "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007faf60028000 nid=0x48c9 > runnable > "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007faf60029800 nid=0x48ca > runnable > "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007faf6002b800 nid=0x48cb > runnable > "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007faf6002d000 nid=0x48cc > runnable > "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007faf6002f000 nid=0x48cd > runnable > "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007faf60030800 nid=0x48ce > runnable > "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007faf60032800 nid=0x48cf > runnable > "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007faf60034000 nid=0x48d0 > runnable > "GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007faf60036000 nid=0x48d1 > runnable > "GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007faf60037800 nid=0x48d2 > runnable > "GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007faf60039800 nid=0x48d3 > runnable > "GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007faf6003b000 nid=0x48d4 > runnable > "GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007faf6003d000 nid=0x48d5 > runnable > "GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007faf6003e800 nid=0x48d6 > runnable > "GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007faf60040800 nid=0x48d7 > runnable > "GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007faf60042000 nid=0x48d8 > runnable > "GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007faf60044000 nid=0x48d9 > runnable > "GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007faf60045800 nid=0x48da > runnable > "VM Periodic Task Thread" os_prio=0 tid=0x00007faf60103800 nid=0x48ef waiting > on condition > JNI global references: 521 > Heap > PSYoungGen total 27089920K, used 13321206K [0x00007fa8a5d80000, > 0x00007faf50800000, 0x00007faf50800000) > eden space 26267648K, 49% used > [0x00007fa8a5d80000,0x00007fabba38d5f0,0x00007faee9180000) > from space 822272K, 49% used > [0x00007faee9180000,0x00007faf01c70318,0x00007faf1b480000) > to space 808448K, 0% used > [0x00007faf1f280000,0x00007faf1f280000,0x00007faf50800000) > ParOldGen total 55924224K, used 40961643K [0x00007f9b50800000, > 0x00007fa8a5d80000, 0x00007fa8a5d80000) > object space 55924224K, 73% used > [0x00007f9b50800000,0x00007fa51499aee8,0x00007fa8a5d80000) > Metaspace used 57281K, capacity 59341K, committed 59392K, reserved > 59392K > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org