Thanks for your reply, Ted. Below is the stack dump for all threads:
Thread dump for executor driver Updated at 2016/01/14 20:35:41 Collapse All Thread 89: Executor task launch worker-0 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 90: Executor task launch worker-1 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 91: Executor task launch worker-2 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 92: Executor task launch worker-3 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 37: BLOCK_MANAGER cleanup timer (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.util.TimerThread.mainLoop(Timer.java:526) java.util.TimerThread.run(Timer.java:505) Thread 38: BROADCAST_VARS cleanup timer (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.util.TimerThread.mainLoop(Timer.java:526) java.util.TimerThread.run(Timer.java:505) Thread 61: dag-scheduler-event-loop (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489) java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678) org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46) Thread 62: driver-heartbeater (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 3: Finalizer (WAITING) java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 59: heartbeat-receiver-event-loop-thread (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 88: IPC Parameter Sending Thread #0 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 1: main (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73) org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:559) org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) org.apache.spark.SparkContext.runJob(SparkContext.scala:1850) org.apache.spark.SparkContext.runJob(SparkContext.scala:1921) org.apache.spark.rdd.RDD.count(RDD.scala:1125) $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20) $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25) $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27) $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29) $line15.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31) $line15.$read$$iwC$$iwC$$iwC.<init>(<console>:33) $line15.$read$$iwC$$iwC.<init>(<console>:35) $line15.$read$$iwC.<init>(<console>:37) $line15.$read.<init>(<console>:39) $line15.$read$.<init>(<console>:43) $line15.$read$.<clinit>(<console>) $line15.$eval$.<init>(<console>:7) $line15.$eval$.<clinit>(<console>) $line15.$eval.$print(<console>) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) org.apache.spark.repl.SparkILoop.org $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) org.apache.spark.repl.SparkILoop.org $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) org.apache.spark.repl.Main$.main(Main.scala:31) org.apache.spark.repl.Main.main(Main.scala) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Thread 36: MAP_OUTPUT_TRACKER cleanup timer (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.util.TimerThread.mainLoop(Timer.java:526) java.util.TimerThread.run(Timer.java:505) Thread 32: New I/O boss #3 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 35: New I/O server boss #6 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 29: New I/O worker #1 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 30: New I/O worker #2 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 33: New I/O worker #4 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 34: New I/O worker #5 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 97: org.apache.hadoop.hdfs.PeerCache@285b3467 (TIMED_WAITING) java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Thread 10: process reaper (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 51: qtp1194494596-51 Selector0 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.spark-project.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) org.spark-project.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 52: qtp1194494596-52 Acceptor0 SelectChannelConnector@0.0.0.0:4040 (RUNNABLE) sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) org.spark-project.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 53: qtp1194494596-53 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 54: qtp1194494596-54 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 55: qtp1194494596-55 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 56: qtp1194494596-56 (RUNNABLE) sun.management.ThreadImpl.dumpThreads0(Native Method) sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) org.apache.spark.util.Utils$.getThreadDump(Utils.scala:1860) org.apache.spark.SparkContext.getExecutorThreadDump(SparkContext.scala:611) org.apache.spark.ui.exec.ExecutorThreadDumpPage.render(ExecutorThreadDumpPage.scala:49) org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:79) org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:69) javax.servlet.http.HttpServlet.service(HttpServlet.java:735) javax.servlet.http.HttpServlet.service(HttpServlet.java:848) org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) org.spark-project.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264) org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) org.spark-project.jetty.server.Server.handle(Server.java:370) org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644) org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 57: qtp1194494596-57 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 58: qtp1194494596-58 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 13: qtp638016280-13 Acceptor0 SocketConnector@0.0.0.0:41668 (RUNNABLE) java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) java.net.ServerSocket.implAccept(ServerSocket.java:530) java.net.ServerSocket.accept(ServerSocket.java:498) org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117) org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 14: qtp638016280-14 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 15: qtp638016280-15 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 16: qtp638016280-16 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 17: qtp638016280-17 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 18: qtp638016280-18 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 19: qtp638016280-19 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 20: qtp638016280-20 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 40: qtp658016306-40 Acceptor0 SocketConnector@0.0.0.0:60101 (RUNNABLE) java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) java.net.ServerSocket.implAccept(ServerSocket.java:530) java.net.ServerSocket.accept(ServerSocket.java:498) org.spark-project.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117) org.spark-project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745) Thread 41: qtp658016306-41 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 42: qtp658016306-42 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 43: qtp658016306-43 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 44: qtp658016306-44 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 45: qtp658016306-45 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 46: qtp658016306-46 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 47: qtp658016306-47 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) java.lang.Thread.run(Thread.java:745) Thread 2: Reference Handler (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Thread 49: refresh progress (TIMED_WAITING) java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 67: shuffle-server-0 (RUNNABLE) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) Thread 4: Signal Dispatcher (RUNNABLE) Thread 72: Spark Context Cleaner (TIMED_WAITING) java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:157) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) org.apache.spark.ContextCleaner.org $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:154) org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:67) Thread 48: SPARK_CONTEXT cleanup timer (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.util.TimerThread.mainLoop(Timer.java:526) java.util.TimerThread.run(Timer.java:505) Thread 24: sparkDriver-akka.actor.default-dispatcher-2 (WAITING) sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread 25: sparkDriver-akka.actor.default-dispatcher-3 (WAITING) sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread 26: sparkDriver-akka.actor.default-dispatcher-4 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread 27: sparkDriver-akka.remote.default-remote-dispatcher-5 (TIMED_WAITING) sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread 28: sparkDriver-akka.remote.default-remote-dispatcher-6 (WAITING) sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread 23: sparkDriver-scheduler-1 (TIMED_WAITING) java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226) akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405) akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) java.lang.Thread.run(Thread.java:745) Thread 22: SparkListenerBus (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.Semaphore.acquire(Semaphore.java:317) org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:65) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136) org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Thread 94: task-result-getter-0 (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 95: task-result-getter-1 (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 96: task-result-getter-2 (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 98: task-result-getter-3 (WAITING) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Thread 60: Timer-0 (WAITING) java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) java.util.TimerThread.mainLoop(Timer.java:526) java.util.TimerThread.run(Timer.java:505) On Fri, Jan 15, 2016 at 2:09 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Can you capture one or two stack traces of the local master process and > pastebin them ? > > Thanks > > On Thu, Jan 14, 2016 at 6:01 AM, Kai Wei <w...@pythian.com> wrote: > >> Hi list, >> >> I ran into an issue which I think could be a bug. >> >> I have a Hive table stored as parquet files. Let's say it's called >> testtable. I found the code below stuck forever in spark-shell with a local >> master or driver/executor: >> sqlContext.sql("select * from testtable").rdd.cache.zipWithIndex().count >> >> But it works if I use a standalone master. >> >> I also tried several different variants: >> don't cache the rdd(works): >> sqlContext.sql("select * from testtable").rdd.zipWithIndex().count >> >> cache the rdd after zipWithIndex(works): >> sqlContext.sql("select * from testtable").rdd.zipWithIndex().cache.count >> >> use parquet file reader(doesn't work): >> >> sqlContext.read.parquet("hdfs://localhost:8020/user/hive/warehouse/testtable").rdd.cache.zipWithIndex().count >> >> use parquet files on local file system(works) >> sqlContext.read.parquet("/tmp/testtable").rdd.cache.zipWithIndex().count >> >> I read the code of zipWithIndex() and the DAG visualization. I think the >> function cause the Spark firstly retrieve n-1 partitions of target table >> and cache them, then the last partition. It must be something wrong when >> the driver/executor tries to read the last parition from HDFS . >> >> I am using spark-1.5.2-bin-hadoop-2.6 on cloudera quickstart vm 5.4.2. >> >> -- >> Kai Wei >> Big Data Developer >> >> Pythian - love your data >> >> w...@pythian.com >> Tel: +1 613 565 8696 x1579 >> Mobile: +61 403 572 456 >> >> -- >> >> >> >> > -- Kai Wei Big Data Developer Pythian - love your data w...@pythian.com Tel: +1 613 565 8696 x1579 Mobile: +61 403 572 456 -- --