[ https://issues.apache.org/jira/browse/SPARK-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889564#comment-15889564 ]
Ari Gesher edited comment on SPARK-19764 at 3/1/17 6:05 AM: ------------------------------------------------------------ Nothing like that. Full logs in the attached tarball. Here's the stack trace from master: {noformat} Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode): "shuffle-server-7" #36 daemon prio=5 os_prio=0 tid=0x00007f0764019800 nid=0xcd7 runnable [0x00007f0720684000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0014950> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c00169d0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c00148a8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-6" #35 daemon prio=5 os_prio=0 tid=0x00007f0764017800 nid=0xb6a runnable [0x00007f07218f7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0017198> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0019218> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0017100> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-5" #34 daemon prio=5 os_prio=0 tid=0x00007f0764015800 nid=0xb69 runnable [0x00007f07219f8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c00199e0> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01eee80> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0019948> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-4" #33 daemon prio=5 os_prio=0 tid=0x00007f0764014000 nid=0xb1f runnable [0x00007f0721af9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01ef648> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01f16c8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01ef5b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-3" #32 daemon prio=5 os_prio=0 tid=0x00007f0764012000 nid=0xb0d runnable [0x00007f0721786000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01f1e90> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01f3f10> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01f1df8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-2" #31 daemon prio=5 os_prio=0 tid=0x00007f0764010800 nid=0xb06 runnable [0x00007f0722bfb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01f46d8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c022e768> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01f4640> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "threadDeathWatcher-2-1" #58 daemon prio=1 os_prio=0 tid=0x00007f0718010000 nid=0xab4 waiting on condition [0x00007f0723efd000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) "shuffle-server-1" #30 daemon prio=5 os_prio=0 tid=0x00007f076400f000 nid=0xab2 runnable [0x00007f0723ffe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c022ef30> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0230fb0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c022ee98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-57" #57 daemon prio=5 os_prio=0 tid=0x00007f0788122000 nid=0xaae waiting on condition [0x00007f07b83d5000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-56" #56 daemon prio=5 os_prio=0 tid=0x00007f0788120000 nid=0xaad waiting on condition [0x00007f07b84d6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-55" #55 daemon prio=5 os_prio=0 tid=0x00007f078811e800 nid=0xaac waiting on condition [0x00007f07b85d7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-54-acceptor-0@1b0de413-ServerConnector@15ca0902{HTTP/1.1}{172.31.20.156:6066}" #54 daemon prio=5 os_prio=0 tid=0x00007f078811c800 nid=0xaab runnable [0x00007f07b86d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked <0x00000000c0023068> (a java.lang.Object) at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-53-selector-ServerConnectorManager@55b255e7/3" #53 daemon prio=5 os_prio=0 tid=0x00007f078811b000 nid=0xaaa runnable [0x00007f07b87d9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b020> (a sun.nio.ch.Util$3) - locked <0x00000000c002b030> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002afd8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-52-selector-ServerConnectorManager@55b255e7/2" #52 daemon prio=5 os_prio=0 tid=0x00007f0788119000 nid=0xaa9 runnable [0x00007f07b88da000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0033010> (a sun.nio.ch.Util$3) - locked <0x00000000c0033020> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0032fc8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-51-selector-ServerConnectorManager@55b255e7/1" #51 daemon prio=5 os_prio=0 tid=0x00007f0788117000 nid=0xaa8 runnable [0x00007f07b8adc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001b1e8> (a sun.nio.ch.Util$3) - locked <0x00000000c001b1f8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001b1a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-50-selector-ServerConnectorManager@55b255e7/0" #50 daemon prio=5 os_prio=0 tid=0x00007f0788117800 nid=0xaa7 runnable [0x00007f07b89db000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001adb0> (a sun.nio.ch.Util$3) - locked <0x00000000c001adc0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001ad68> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "master-forward-message-thread" #48 daemon prio=5 os_prio=0 tid=0x00007f07880a6000 nid=0xaa3 waiting on condition [0x00007f07b8bdd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "MasterUI-47" #47 daemon prio=5 os_prio=0 tid=0x00007f078809c000 nid=0xaa2 waiting on condition [0x00007f07b8cde000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-46" #46 daemon prio=5 os_prio=0 tid=0x00007f078809a800 nid=0xaa1 waiting on condition [0x00007f07b8ddf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-45" #45 daemon prio=5 os_prio=0 tid=0x00007f0788099000 nid=0xaa0 waiting on condition [0x00007f07b8ee0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-44-acceptor-0@21c02ae7-ServerConnector@2f07d0af{HTTP/1.1}{172.31.20.156:8080}" #44 daemon prio=5 os_prio=0 tid=0x00007f0788097000 nid=0xa9f runnable [0x00007f07b8fe1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked <0x00000000c0033ac0> (a java.lang.Object) at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-43-selector-ServerConnectorManager@2bbaf8f7/3" #43 daemon prio=5 os_prio=0 tid=0x00007f0788095800 nid=0xa9e runnable [0x00007f07b90e2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0033da8> (a sun.nio.ch.Util$3) - locked <0x00000000c0033db8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0033d60> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-42-selector-ServerConnectorManager@2bbaf8f7/2" #42 daemon prio=5 os_prio=0 tid=0x00007f078808f800 nid=0xa9d runnable [0x00007f07b91e3000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b2c0> (a sun.nio.ch.Util$3) - locked <0x00000000c002b2d0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002b278> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-41-selector-ServerConnectorManager@2bbaf8f7/1" #41 daemon prio=5 os_prio=0 tid=0x00007f078808a800 nid=0xa9c runnable [0x00007f07b92e4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b560> (a sun.nio.ch.Util$3) - locked <0x00000000c002b570> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002b518> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-40-selector-ServerConnectorManager@2bbaf8f7/0" #40 daemon prio=5 os_prio=0 tid=0x00007f078808a000 nid=0xa9b runnable [0x00007f07b93e5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001b488> (a sun.nio.ch.Util$3) - locked <0x00000000c001b498> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001b440> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "netty-rpc-env-timeout" #38 daemon prio=5 os_prio=0 tid=0x00007f07f123b800 nid=0xa90 waiting on condition [0x00007f07b94e6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c001b698> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "shuffle-server-0" #29 daemon prio=5 os_prio=0 tid=0x00007f07f11dc800 nid=0xa8e runnable [0x00007f07b97e7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c00141e8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0014208> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c00141a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-7" #20 daemon prio=5 os_prio=0 tid=0x00007f07f0ffc800 nid=0xa89 waiting on condition [0x00007f07b9cfb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-6" #19 daemon prio=5 os_prio=0 tid=0x00007f07f0ffb000 nid=0xa88 waiting on condition [0x00007f07b9dfc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-5" #18 daemon prio=5 os_prio=0 tid=0x00007f07f0ff9000 nid=0xa85 waiting on condition [0x00007f07b9efd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-4" #17 daemon prio=5 os_prio=0 tid=0x00007f07f0ff7800 nid=0xa83 waiting on condition [0x00007f07b9ffe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-3" #16 daemon prio=5 os_prio=0 tid=0x00007f07f0ff6000 nid=0xa82 waiting on condition [0x00007f07c01f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-2" #15 daemon prio=5 os_prio=0 tid=0x00007f07f0fcd800 nid=0xa80 waiting on condition [0x00007f07c02f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-1" #14 daemon prio=5 os_prio=0 tid=0x00007f07f0fcc000 nid=0xa7e waiting on condition [0x00007f07c03f8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-0" #13 daemon prio=5 os_prio=0 tid=0x00007f07f0fcb800 nid=0xa7c waiting on condition [0x00007f07c08fb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f07f00d2000 nid=0xa6f runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f07f00c5800 nid=0xa6c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f07f00c1000 nid=0xa6b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f07f00bf000 nid=0xa6a waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f07f00bc000 nid=0xa69 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f07f00ba000 nid=0xa68 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f07f0092800 nid=0xa62 in Object.wait() [0x00007f07c37f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f07f008e000 nid=0xa60 in Object.wait() [0x00007f07c38f7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "main" #1 prio=5 os_prio=0 tid=0x00007f07f0010800 nid=0xa4b waiting on condition [0x00007f07f98c2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e3780> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465) at org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180) at org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:273) at org.apache.spark.deploy.master.Master$.main(Master.scala:1012) at org.apache.spark.deploy.master.Master.main(Master.scala) "VM Thread" os_prio=0 tid=0x00007f07f0086000 nid=0xa5d runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f07f0025800 nid=0xa4e runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f07f0027800 nid=0xa50 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f07f0029000 nid=0xa52 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f07f002b000 nid=0xa54 runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f07f002c800 nid=0xa56 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f07f002e800 nid=0xa58 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f07f0030000 nid=0xa5a runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f07f0032000 nid=0xa5c runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f07f00d4800 nid=0xa70 waiting on condition JNI global references: 313 Heap PSYoungGen total 286720K, used 150258K [0x00000000eab00000, 0x00000000feb00000, 0x0000000100000000) eden space 245760K, 45% used [0x00000000eab00000,0x00000000f17d2a00,0x00000000f9b00000) from space 40960K, 94% used [0x00000000fc300000,0x00000000fe8e9f40,0x00000000feb00000) to space 40960K, 0% used [0x00000000f9b00000,0x00000000f9b00000,0x00000000fc300000) ParOldGen total 264192K, used 83384K [0x00000000c0000000, 0x00000000d0200000, 0x00000000eab00000) object space 264192K, 31% used [0x00000000c0000000,0x00000000c516e088,0x00000000d0200000) Metaspace used 25818K, capacity 26160K, committed 26368K, reserved 1073152K class space used 3429K, capacity 3512K, committed 3584K, reserved 1048576K {noformat} was (Author: agesher): Nothing like that. Full logs in the attached tarball. Here's the stack trace from a stuck executor: {noformat} Full thread dump OpenJDK 64-Bit Server VM (25.121-b13 mixed mode): "shuffle-server-7" #36 daemon prio=5 os_prio=0 tid=0x00007f0764019800 nid=0xcd7 runnable [0x00007f0720684000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0014950> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c00169d0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c00148a8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-6" #35 daemon prio=5 os_prio=0 tid=0x00007f0764017800 nid=0xb6a runnable [0x00007f07218f7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0017198> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0019218> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0017100> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-5" #34 daemon prio=5 os_prio=0 tid=0x00007f0764015800 nid=0xb69 runnable [0x00007f07219f8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c00199e0> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01eee80> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0019948> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-4" #33 daemon prio=5 os_prio=0 tid=0x00007f0764014000 nid=0xb1f runnable [0x00007f0721af9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01ef648> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01f16c8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01ef5b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-3" #32 daemon prio=5 os_prio=0 tid=0x00007f0764012000 nid=0xb0d runnable [0x00007f0721786000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01f1e90> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c01f3f10> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01f1df8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "shuffle-server-2" #31 daemon prio=5 os_prio=0 tid=0x00007f0764010800 nid=0xb06 runnable [0x00007f0722bfb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c01f46d8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c022e768> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c01f4640> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "threadDeathWatcher-2-1" #58 daemon prio=1 os_prio=0 tid=0x00007f0718010000 nid=0xab4 waiting on condition [0x00007f0723efd000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) "shuffle-server-1" #30 daemon prio=5 os_prio=0 tid=0x00007f076400f000 nid=0xab2 runnable [0x00007f0723ffe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c022ef30> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0230fb0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c022ee98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-57" #57 daemon prio=5 os_prio=0 tid=0x00007f0788122000 nid=0xaae waiting on condition [0x00007f07b83d5000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-56" #56 daemon prio=5 os_prio=0 tid=0x00007f0788120000 nid=0xaad waiting on condition [0x00007f07b84d6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-55" #55 daemon prio=5 os_prio=0 tid=0x00007f078811e800 nid=0xaac waiting on condition [0x00007f07b85d7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0013b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-54-acceptor-0@1b0de413-ServerConnector@15ca0902{HTTP/1.1}{172.31.20.156:6066}" #54 daemon prio=5 os_prio=0 tid=0x00007f078811c800 nid=0xaab runnable [0x00007f07b86d8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked <0x00000000c0023068> (a java.lang.Object) at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-53-selector-ServerConnectorManager@55b255e7/3" #53 daemon prio=5 os_prio=0 tid=0x00007f078811b000 nid=0xaaa runnable [0x00007f07b87d9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b020> (a sun.nio.ch.Util$3) - locked <0x00000000c002b030> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002afd8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-52-selector-ServerConnectorManager@55b255e7/2" #52 daemon prio=5 os_prio=0 tid=0x00007f0788119000 nid=0xaa9 runnable [0x00007f07b88da000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0033010> (a sun.nio.ch.Util$3) - locked <0x00000000c0033020> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0032fc8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-51-selector-ServerConnectorManager@55b255e7/1" #51 daemon prio=5 os_prio=0 tid=0x00007f0788117000 nid=0xaa8 runnable [0x00007f07b8adc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001b1e8> (a sun.nio.ch.Util$3) - locked <0x00000000c001b1f8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001b1a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "qtp1090158868-50-selector-ServerConnectorManager@55b255e7/0" #50 daemon prio=5 os_prio=0 tid=0x00007f0788117800 nid=0xaa7 runnable [0x00007f07b89db000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001adb0> (a sun.nio.ch.Util$3) - locked <0x00000000c001adc0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001ad68> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "master-forward-message-thread" #48 daemon prio=5 os_prio=0 tid=0x00007f07880a6000 nid=0xaa3 waiting on condition [0x00007f07b8bdd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "MasterUI-47" #47 daemon prio=5 os_prio=0 tid=0x00007f078809c000 nid=0xaa2 waiting on condition [0x00007f07b8cde000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-46" #46 daemon prio=5 os_prio=0 tid=0x00007f078809a800 nid=0xaa1 waiting on condition [0x00007f07b8ddf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-45" #45 daemon prio=5 os_prio=0 tid=0x00007f0788099000 nid=0xaa0 waiting on condition [0x00007f07b8ee0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0033480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "MasterUI-44-acceptor-0@21c02ae7-ServerConnector@2f07d0af{HTTP/1.1}{172.31.20.156:8080}" #44 daemon prio=5 os_prio=0 tid=0x00007f0788097000 nid=0xa9f runnable [0x00007f07b8fe1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked <0x00000000c0033ac0> (a java.lang.Object) at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-43-selector-ServerConnectorManager@2bbaf8f7/3" #43 daemon prio=5 os_prio=0 tid=0x00007f0788095800 nid=0xa9e runnable [0x00007f07b90e2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c0033da8> (a sun.nio.ch.Util$3) - locked <0x00000000c0033db8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c0033d60> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-42-selector-ServerConnectorManager@2bbaf8f7/2" #42 daemon prio=5 os_prio=0 tid=0x00007f078808f800 nid=0xa9d runnable [0x00007f07b91e3000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b2c0> (a sun.nio.ch.Util$3) - locked <0x00000000c002b2d0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002b278> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-41-selector-ServerConnectorManager@2bbaf8f7/1" #41 daemon prio=5 os_prio=0 tid=0x00007f078808a800 nid=0xa9c runnable [0x00007f07b92e4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c002b560> (a sun.nio.ch.Util$3) - locked <0x00000000c002b570> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c002b518> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "MasterUI-40-selector-ServerConnectorManager@2bbaf8f7/0" #40 daemon prio=5 os_prio=0 tid=0x00007f078808a000 nid=0xa9b runnable [0x00007f07b93e5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c001b488> (a sun.nio.ch.Util$3) - locked <0x00000000c001b498> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c001b440> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "netty-rpc-env-timeout" #38 daemon prio=5 os_prio=0 tid=0x00007f07f123b800 nid=0xa90 waiting on condition [0x00007f07b94e6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c001b698> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "shuffle-server-0" #29 daemon prio=5 os_prio=0 tid=0x00007f07f11dc800 nid=0xa8e runnable [0x00007f07b97e7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c00141e8> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x00000000c0014208> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c00141a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-7" #20 daemon prio=5 os_prio=0 tid=0x00007f07f0ffc800 nid=0xa89 waiting on condition [0x00007f07b9cfb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-6" #19 daemon prio=5 os_prio=0 tid=0x00007f07f0ffb000 nid=0xa88 waiting on condition [0x00007f07b9dfc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-5" #18 daemon prio=5 os_prio=0 tid=0x00007f07f0ff9000 nid=0xa85 waiting on condition [0x00007f07b9efd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-4" #17 daemon prio=5 os_prio=0 tid=0x00007f07f0ff7800 nid=0xa83 waiting on condition [0x00007f07b9ffe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-3" #16 daemon prio=5 os_prio=0 tid=0x00007f07f0ff6000 nid=0xa82 waiting on condition [0x00007f07c01f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-2" #15 daemon prio=5 os_prio=0 tid=0x00007f07f0fcd800 nid=0xa80 waiting on condition [0x00007f07c02f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-1" #14 daemon prio=5 os_prio=0 tid=0x00007f07f0fcc000 nid=0xa7e waiting on condition [0x00007f07c03f8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "dispatcher-event-loop-0" #13 daemon prio=5 os_prio=0 tid=0x00007f07f0fcb800 nid=0xa7c waiting on condition [0x00007f07c08fb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e1ed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f07f00d2000 nid=0xa6f runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f07f00c5800 nid=0xa6c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f07f00c1000 nid=0xa6b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f07f00bf000 nid=0xa6a waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f07f00bc000 nid=0xa69 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f07f00ba000 nid=0xa68 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f07f0092800 nid=0xa62 in Object.wait() [0x00007f07c37f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000000c01e8880> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f07f008e000 nid=0xa60 in Object.wait() [0x00007f07c38f7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000000c01e8ab0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "main" #1 prio=5 os_prio=0 tid=0x00007f07f0010800 nid=0xa4b waiting on condition [0x00007f07f98c2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00e3780> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465) at org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180) at org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:273) at org.apache.spark.deploy.master.Master$.main(Master.scala:1012) at org.apache.spark.deploy.master.Master.main(Master.scala) "VM Thread" os_prio=0 tid=0x00007f07f0086000 nid=0xa5d runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f07f0025800 nid=0xa4e runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f07f0027800 nid=0xa50 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f07f0029000 nid=0xa52 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f07f002b000 nid=0xa54 runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f07f002c800 nid=0xa56 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f07f002e800 nid=0xa58 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f07f0030000 nid=0xa5a runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f07f0032000 nid=0xa5c runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f07f00d4800 nid=0xa70 waiting on condition JNI global references: 313 Heap PSYoungGen total 286720K, used 150258K [0x00000000eab00000, 0x00000000feb00000, 0x0000000100000000) eden space 245760K, 45% used [0x00000000eab00000,0x00000000f17d2a00,0x00000000f9b00000) from space 40960K, 94% used [0x00000000fc300000,0x00000000fe8e9f40,0x00000000feb00000) to space 40960K, 0% used [0x00000000f9b00000,0x00000000f9b00000,0x00000000fc300000) ParOldGen total 264192K, used 83384K [0x00000000c0000000, 0x00000000d0200000, 0x00000000eab00000) object space 264192K, 31% used [0x00000000c0000000,0x00000000c516e088,0x00000000d0200000) Metaspace used 25818K, capacity 26160K, committed 26368K, reserved 1073152K class space used 3429K, capacity 3512K, committed 3584K, reserved 1048576K {noformat} > Executors hang with supposedly running task that are really finished. > --------------------------------------------------------------------- > > Key: SPARK-19764 > URL: https://issues.apache.org/jira/browse/SPARK-19764 > Project: Spark > Issue Type: Bug > Components: PySpark, Spark Core > Affects Versions: 2.0.2 > Environment: Ubuntu 16.04 LTS > OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > Spark 2.0.2 - Spark Cluster Manager > Reporter: Ari Gesher > Attachments: driver-log-stderr.log, executor-2.log, netty-6153.jpg, > SPARK-19764.tgz > > > We've come across a job that won't finish. Running on a six-node cluster, > each of the executors end up with 5-7 tasks that are never marked as > completed. > Here's an excerpt from the web UI: > ||Index ▴||ID||Attempt||Status||Locality Level||Executor ID / Host||Launch > Time||Duration||Scheduler Delay||Task Deserialization Time||GC Time||Result > Serialization Time||Getting Result Time||Peak Execution Memory||Shuffle Read > Size / Records||Errors|| > |105 | 1131 | 0 | SUCCESS |PROCESS_LOCAL |4 / 172.31.24.171 | > 2017/02/27 22:51:36 | 1.9 min | 9 ms | 4 ms | 0.7 s | 2 ms| 6 ms| > 384.1 MB| 90.3 MB / 572 | | > |106| 1168| 0| RUNNING |ANY| 2 / 172.31.16.112| 2017/02/27 > 22:53:25| 6.5 h |0 ms| 0 ms| 1 s |0 ms| 0 ms| |384.1 MB > |98.7 MB / 624 | | > However, the Executor reports the task as finished: > {noformat} > 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168) > 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). > 2633558 bytes result sent via BlockManager) > {noformat} > As does the driver log: > {noformat} > 17/02/27 22:53:25 INFO Executor: Running task 106.0 in stage 5.0 (TID 1168) > 17/02/27 22:55:29 INFO Executor: Finished task 106.0 in stage 5.0 (TID 1168). > 2633558 bytes result sent via BlockManager) > {noformat} > Full log from this executor and the {{stderr}} from > {{app-20170227223614-0001/2/stderr}} attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org