[ https://issues.apache.org/jira/browse/SPARK-30488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013651#comment-17013651 ]
Ajith S commented on SPARK-30488: --------------------------------- As from log, i see `sbt` classes in the deadlock threads,this is related to internal classloaders in sbt which was fixed in sbt 1.3.3 by marking classloaders as parallel capable. [https://github.com/sbt/sbt/pull/5131] also a similar issue '[https://github.com/sbt/sbt/issues/5116]' [~rohit21agrawal] Thanks for reporting this. Some questions, can you also please mention how the sparkcontext was created.? > Deadlock between block-manager-slave-async-thread-pool and spark context > cleaner > -------------------------------------------------------------------------------- > > Key: SPARK-30488 > URL: https://issues.apache.org/jira/browse/SPARK-30488 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.3 > Reporter: Rohit Agrawal > Priority: Major > > Deadlock happens while cleaning up the spark context. Here is the full thread > dump: > > > 2020-01-10T20:13:16.2884057Z Full thread dump Java HotSpot(TM) 64-Bit > Server VM (25.221-b11 mixed mode): > 2020-01-10T20:13:16.2884392Z > 2020-01-10T20:13:16.2884660Z "SIGINT handler" #488 daemon prio=9 os_prio=2 > tid=0x00000000111fa000 nid=0x4794 waiting for monitor entry > [0x000000001c86e000] > 2020-01-10T20:13:16.2884807Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2884879Z at java.lang.Shutdown.exit(Shutdown.java:212) > 2020-01-10T20:13:16.2885693Z - waiting to lock <0x00000000c0155de0> (a > java.lang.Class for java.lang.Shutdown) > 2020-01-10T20:13:16.2885840Z at > java.lang.Terminator$1.handle(Terminator.java:52) > 2020-01-10T20:13:16.2885965Z at sun.misc.Signal$1.run(Signal.java:212) > 2020-01-10T20:13:16.2886329Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2886430Z > 2020-01-10T20:13:16.2886752Z "Thread-3" #108 prio=5 os_prio=0 > tid=0x00000000111f7800 nid=0x48cc waiting for monitor entry > [0x000000002c33f000] > 2020-01-10T20:13:16.2886881Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2886999Z at > org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:273) > 2020-01-10T20:13:16.2887107Z at > org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121) > 2020-01-10T20:13:16.2887212Z at > org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95) > 2020-01-10T20:13:16.2887421Z > 2020-01-10T20:13:16.2887798Z "block-manager-slave-async-thread-pool-81" #486 > daemon prio=5 os_prio=0 tid=0x00000000111fe800 nid=0x2e34 waiting for monitor > entry [0x000000002bf3d000] > 2020-01-10T20:13:16.2889192Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2889305Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:404) > 2020-01-10T20:13:16.2889405Z - waiting to lock <0x00000000c1f359f0> (a > sbt.internal.LayeredClassLoader) > 2020-01-10T20:13:16.2889482Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:411) > 2020-01-10T20:13:16.2889582Z - locked <0x00000000ca33e4c8> (a > sbt.internal.ManagedClassLoader$ZombieClassLoader) > 2020-01-10T20:13:16.2889659Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) > 2020-01-10T20:13:16.2890881Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply$mcZ$sp(BlockManagerSlaveEndpoint.scala:58) > 2020-01-10T20:13:16.2891006Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57) > 2020-01-10T20:13:16.2891142Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57) > 2020-01-10T20:13:16.2891260Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:86) > 2020-01-10T20:13:16.2891375Z at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > 2020-01-10T20:13:16.2891624Z at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > 2020-01-10T20:13:16.2891737Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.2891833Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2891925Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2891967Z > 2020-01-10T20:13:16.2892066Z "pool-31-thread-16" #335 prio=5 os_prio=0 > tid=0x00000000153b2000 nid=0x1aac waiting on condition [0x000000004b2ff000] > 2020-01-10T20:13:16.2892147Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2892241Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2892328Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2892437Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2892530Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2892649Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2892753Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2892865Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2892969Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2893060Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2893150Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2893192Z > 2020-01-10T20:13:16.2893271Z "pool-31-thread-15" #334 prio=5 os_prio=0 > tid=0x000000001a699000 nid=0x1110 waiting for monitor entry > [0x000000004aefe000] > 2020-01-10T20:13:16.2893375Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2893457Z at > com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:30) > 2020-01-10T20:13:16.2893574Z at > com.microsoft.azure.eventhubs.impl.MessageSender.onClose(MessageSender.java:424) > 2020-01-10T20:13:16.2893849Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:70) > 2020-01-10T20:13:16.2894172Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:106) > 2020-01-10T20:13:16.2894278Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) > 2020-01-10T20:13:16.2894469Z at > org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) > 2020-01-10T20:13:16.2894565Z at > org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) > 2020-01-10T20:13:16.2894645Z at > org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) > 2020-01-10T20:13:16.2894745Z at > org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) > 2020-01-10T20:13:16.2894827Z at > com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:512) > 2020-01-10T20:13:16.2894997Z at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > 2020-01-10T20:13:16.2895073Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2020-01-10T20:13:16.2895178Z at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > 2020-01-10T20:13:16.2895271Z at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > 2020-01-10T20:13:16.2895389Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.2895469Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2895555Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2895594Z > 2020-01-10T20:13:16.2895691Z "pool-31-thread-14" #333 prio=5 os_prio=0 > tid=0x000000001a697800 nid=0x31d4 waiting on condition [0x000000004aafe000] > 2020-01-10T20:13:16.2895764Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2895855Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2895936Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2896037Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2896122Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2896238Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2896329Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2896433Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2896532Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2896613Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2896708Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2896748Z > 2020-01-10T20:13:16.2896819Z "pool-31-thread-13" #332 prio=5 os_prio=0 > tid=0x000000001a69e800 nid=0x4fe8 waiting on condition [0x000000004a6ff000] > 2020-01-10T20:13:16.2896914Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2897080Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2897183Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2897265Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2897370Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2897461Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2897576Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2897659Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2897759Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2897836Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2898012Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2898050Z > 2020-01-10T20:13:16.2898140Z "pool-31-thread-12" #331 prio=5 os_prio=0 > tid=0x000000001a697000 nid=0x3a7c waiting on condition [0x000000004a2fe000] > 2020-01-10T20:13:16.2898212Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2898294Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2898376Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2898536Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2898795Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2898911Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2899003Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2899115Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2899370Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2899474Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2899719Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2899787Z > 2020-01-10T20:13:16.2899866Z "pool-31-thread-11" #330 prio=5 os_prio=0 > tid=0x000000001a69d800 nid=0x5e48 waiting on condition [0x0000000049eff000] > 2020-01-10T20:13:16.2899975Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2900043Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2900155Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2900253Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2900369Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2900469Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2900587Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2900679Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2900794Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2900900Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2900975Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2901016Z > 2020-01-10T20:13:16.2901118Z "pool-31-thread-10" #329 prio=5 os_prio=0 > tid=0x000000001a69d000 nid=0x1e48 waiting on condition [0x0000000049aff000] > 2020-01-10T20:13:16.2901208Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2901297Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2901386Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2901496Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2901589Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2901720Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2901840Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2901932Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2902117Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2902205Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2902300Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2902343Z > 2020-01-10T20:13:16.2902422Z "pool-31-thread-9" #328 prio=5 os_prio=0 > tid=0x000000001a69c000 nid=0x558c waiting on condition [0x00000000496fe000] > 2020-01-10T20:13:16.2902524Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2902614Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2902776Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2903046Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2903138Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2903265Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2903362Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2903473Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2903557Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2903664Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2903744Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2903804Z > 2020-01-10T20:13:16.2903881Z "pool-31-thread-8" #327 prio=5 os_prio=0 > tid=0x000000001a68f800 nid=0x12a8 waiting on condition [0x00000000492ff000] > 2020-01-10T20:13:16.2903980Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2904046Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2904157Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2904400Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2904510Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2904604Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2904715Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2904807Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2905125Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2905202Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2905296Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2905334Z > 2020-01-10T20:13:16.2905425Z "pool-31-thread-7" #326 prio=5 os_prio=0 > tid=0x000000001a696000 nid=0x2e08 waiting on condition [0x0000000048eff000] > 2020-01-10T20:13:16.2905498Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2905582Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2905662Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2905762Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2905853Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2905964Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2906052Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2906230Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2906333Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2906412Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2906498Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2906537Z > 2020-01-10T20:13:16.2906608Z "pool-31-thread-6" #325 prio=5 os_prio=0 > tid=0x000000001a694000 nid=0x3d24 waiting on condition [0x0000000048aff000] > 2020-01-10T20:13:16.2906759Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2906822Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2906922Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2907003Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2907115Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2907224Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2907313Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2907414Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2907493Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2907599Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2907667Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2907725Z > 2020-01-10T20:13:16.2907795Z "pool-31-thread-5" #324 prio=5 os_prio=0 > tid=0x000000001a691000 nid=0x3c58 waiting on condition [0x00000000486fe000] > 2020-01-10T20:13:16.2907892Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2907954Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2908054Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2908135Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2908237Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2908329Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2908443Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2908526Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2908628Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2908715Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2908800Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2908838Z > 2020-01-10T20:13:16.2908930Z "pool-31-thread-4" #323 prio=5 os_prio=0 > tid=0x000000001a692800 nid=0x360c waiting on condition [0x00000000482ff000] > 2020-01-10T20:13:16.2909003Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2909085Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2909166Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2909277Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2909362Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2909473Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2909634Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2909736Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2909990Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2910094Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2910162Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2910284Z > 2020-01-10T20:13:16.2910359Z "pool-31-thread-3" #322 prio=5 os_prio=0 > tid=0x000000001a693000 nid=0x1554 waiting on condition [0x0000000047efe000] > 2020-01-10T20:13:16.2910642Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2910882Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2910991Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2911088Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2911201Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2911302Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2911420Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2911536Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2911623Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2911730Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2911806Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2911855Z > 2020-01-10T20:13:16.2911955Z "pool-31-thread-2" #321 prio=5 os_prio=0 > tid=0x000000001a12f000 nid=0x4be4 waiting on condition [0x0000000047afe000] > 2020-01-10T20:13:16.2912036Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2912128Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2912217Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2912326Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2912439Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2912548Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2912666Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2912758Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2912871Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2912958Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2913053Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2913224Z > 2020-01-10T20:13:16.2913327Z "pool-31-thread-1" #320 prio=5 os_prio=0 > tid=0x000000001a130800 nid=0x567c waiting on condition [0x00000000476ff000] > 2020-01-10T20:13:16.2913408Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2913504Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2913593Z - parking to wait for <0x00000000c9cad078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2913702Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2913796Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2913994Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2914094Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2914206Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2914313Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2914465Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2914560Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2914602Z > 2020-01-10T20:13:16.2914840Z "task-result-getter-3" #170 daemon prio=5 > os_prio=0 tid=0x000000001a695800 nid=0x5ac8 waiting on condition > [0x000000003eb3e000] > 2020-01-10T20:13:16.2914945Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2915019Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2915287Z - parking to wait for <0x00000000c3ce7cb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2915390Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2915478Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2915585Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2915674Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2915773Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2916071Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2916159Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2916203Z > 2020-01-10T20:13:16.2916277Z "task-result-getter-2" #169 daemon prio=5 > os_prio=0 tid=0x000000001a694800 nid=0x62f0 waiting on condition > [0x000000003e73f000] > 2020-01-10T20:13:16.2916372Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2916455Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2916535Z - parking to wait for <0x00000000c3ce7cb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2916635Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2916720Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2916829Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2916907Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2917006Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2917090Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2917177Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2917214Z > 2020-01-10T20:13:16.2917305Z "task-result-getter-1" #168 daemon prio=5 > os_prio=0 tid=0x000000001a690000 nid=0x5c60 waiting on condition > [0x000000003e33f000] > 2020-01-10T20:13:16.2917381Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2917462Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2917540Z - parking to wait for <0x00000000c3ce7cb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2917650Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2917734Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2917835Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2917985Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2918082Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2918160Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2918247Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2918284Z > 2020-01-10T20:13:16.2918374Z "ABFS-prefetch-7" #162 daemon prio=5 os_prio=0 > tid=0x000000001f6ae000 nid=0x63e0 in Object.wait() [0x000000003b33e000] > 2020-01-10T20:13:16.2918509Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2941181Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2941508Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2941599Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2941688Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2941799Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2941870Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2941930Z > 2020-01-10T20:13:16.2942007Z "ABFS-prefetch-6" #161 daemon prio=5 os_prio=0 > tid=0x000000001f6b0800 nid=0x4f28 in Object.wait() [0x000000003af3e000] > 2020-01-10T20:13:16.2942093Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2942157Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2942236Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2942315Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2942406Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2942488Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2942571Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2942610Z > 2020-01-10T20:13:16.2942683Z "ABFS-prefetch-5" #160 daemon prio=5 os_prio=0 > tid=0x000000001f6b2800 nid=0x254c in Object.wait() [0x000000003ab3f000] > 2020-01-10T20:13:16.2942770Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2942839Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2942902Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2943155Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2943252Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2943344Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2943416Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2943455Z > 2020-01-10T20:13:16.2943542Z "ABFS-prefetch-4" #159 daemon prio=5 os_prio=0 > tid=0x000000001f6b2000 nid=0x5c44 in Object.wait() [0x000000003a73e000] > 2020-01-10T20:13:16.2943622Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2943695Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2943758Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2943843Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2943931Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2944030Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2944110Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2944148Z > 2020-01-10T20:13:16.2944222Z "ABFS-prefetch-3" #158 daemon prio=5 os_prio=0 > tid=0x000000001f6b1000 nid=0x17f4 in Object.wait() [0x000000003a33f000] > 2020-01-10T20:13:16.2944307Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2944719Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2944803Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2944890Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2945002Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2945093Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2945182Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2945313Z > 2020-01-10T20:13:16.2945408Z "ABFS-prefetch-2" #157 daemon prio=5 os_prio=0 > tid=0x000000001f6ad800 nid=0x21f4 in Object.wait() [0x0000000039f3e000] > 2020-01-10T20:13:16.2945494Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2945571Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2945640Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2945845Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2946012Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2946210Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2946379Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2946515Z > 2020-01-10T20:13:16.2946738Z "ABFS-prefetch-1" #156 daemon prio=5 os_prio=0 > tid=0x000000001f6af000 nid=0x3274 in Object.wait() [0x0000000039b3f000] > 2020-01-10T20:13:16.2947028Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2947182Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2947265Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2947352Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2947463Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2947553Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2947634Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2947676Z > 2020-01-10T20:13:16.2947760Z "ABFS-prefetch-0" #155 daemon prio=5 os_prio=0 > tid=0x000000001f6ac800 nid=0x5970 in Object.wait() [0x000000003973e000] > 2020-01-10T20:13:16.2947845Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2947921Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2947996Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2948087Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager.getNextBlockToRead(ReadBufferManager.java:338) > 2020-01-10T20:13:16.2948180Z - locked <0x00000000c583a5e8> (a > org.apache.hadoop.fs.azurebfs.services.ReadBufferManager) > 2020-01-10T20:13:16.2948449Z at > org.apache.hadoop.fs.azurebfs.services.ReadBufferWorker.run(ReadBufferWorker.java:56) > 2020-01-10T20:13:16.2948531Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2948577Z > 2020-01-10T20:13:16.2948662Z "abfs-timer-client-throttling-analyzer-write" > #154 daemon prio=5 os_prio=0 tid=0x000000001f6af800 nid=0x2cc0 in > Object.wait() [0x000000003933e000] > 2020-01-10T20:13:16.2948765Z java.lang.Thread.State: TIMED_WAITING (on object > monitor) > 2020-01-10T20:13:16.2948835Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2948914Z at java.util.TimerThread.mainLoop(Timer.java:552) > 2020-01-10T20:13:16.2948991Z - locked <0x00000000c57ba7f0> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2949070Z at java.util.TimerThread.run(Timer.java:505) > 2020-01-10T20:13:16.2949111Z > 2020-01-10T20:13:16.2949195Z "abfs-timer-client-throttling-analyzer-read" > #153 daemon prio=5 os_prio=0 tid=0x000000001f7e4000 nid=0x37d0 in > Object.wait() [0x0000000038f3e000] > 2020-01-10T20:13:16.2949301Z java.lang.Thread.State: TIMED_WAITING (on object > monitor) > 2020-01-10T20:13:16.2949695Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2949765Z at java.util.TimerThread.mainLoop(Timer.java:552) > 2020-01-10T20:13:16.2949847Z - locked <0x00000000c5c3e478> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2949918Z at java.util.TimerThread.run(Timer.java:505) > 2020-01-10T20:13:16.2949960Z > 2020-01-10T20:13:16.2950064Z "ForkJoinPool-1-worker-9" #151 daemon prio=5 > os_prio=0 tid=0x000000001f7e3800 nid=0x6338 waiting on condition > [0x000000002d33f000] > 2020-01-10T20:13:16.2950150Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2950312Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2950396Z - parking to wait for <0x00000000c5757d20> (a > scala.concurrent.forkjoin.ForkJoinPool) > 2020-01-10T20:13:16.2950493Z at > scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) > 2020-01-10T20:13:16.2950579Z at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > 2020-01-10T20:13:16.2950683Z at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > 2020-01-10T20:13:16.2950733Z > 2020-01-10T20:13:16.2950835Z > "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" > #150 daemon prio=5 os_prio=0 tid=0x000000001f7e2000 nid=0x5b48 in > Object.wait() [0x000000002953f000] > 2020-01-10T20:13:16.2950933Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2951010Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2951086Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > 2020-01-10T20:13:16.2951182Z - locked <0x00000000c58260c8> (a > java.lang.ref.ReferenceQueue$Lock) > 2020-01-10T20:13:16.2951263Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) > 2020-01-10T20:13:16.2951361Z at > org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3762) > 2020-01-10T20:13:16.2951444Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2951502Z > 2020-01-10T20:13:16.2951587Z "task-result-getter-0" #149 daemon prio=5 > os_prio=0 tid=0x000000001f7e1000 nid=0x58e8 waiting on condition > [0x000000002913f000] > 2020-01-10T20:13:16.2951686Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2951755Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2951856Z - parking to wait for <0x00000000c3ce7cb0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2951948Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2952059Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2952162Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2952259Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2952348Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2952452Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2952528Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2952577Z > 2020-01-10T20:13:16.2952657Z "pool-30-thread-16" #147 prio=5 os_prio=0 > tid=0x000000001f7e0800 nid=0x3c1c waiting on condition [0x000000002873e000] > 2020-01-10T20:13:16.2952907Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2952974Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2953069Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2953164Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2953262Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2953361Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2953538Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2953635Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2953721Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2953817Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2953889Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2953991Z > 2020-01-10T20:13:16.2954084Z "pool-30-thread-15" #146 prio=5 os_prio=0 > tid=0x000000001f7de000 nid=0x3578 waiting on condition [0x000000002833e000] > 2020-01-10T20:13:16.2954336Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2954419Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2954508Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2954614Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2954874Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2955166Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2955265Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2955351Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2955452Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2955704Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2955786Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2955836Z > 2020-01-10T20:13:16.2955915Z "pool-30-thread-14" #145 prio=5 os_prio=0 > tid=0x000000001f7df800 nid=0x6048 waiting on condition [0x0000000027f3e000] > 2020-01-10T20:13:16.2956007Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2956082Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2956172Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2956270Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2956365Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2956478Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2956577Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2956676Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2956770Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2956863Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2956937Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2956985Z > 2020-01-10T20:13:16.2957065Z "pool-30-thread-13" #144 prio=5 os_prio=0 > tid=0x000000001f7df000 nid=0x6a8 waiting on condition [0x0000000027b3e000] > 2020-01-10T20:13:16.2957154Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2957229Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2957323Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2957413Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2957515Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2957693Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2957803Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2957896Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2957993Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2958080Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2958232Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2958275Z > 2020-01-10T20:13:16.2958370Z "pool-30-thread-12" #143 prio=5 os_prio=0 > tid=0x000000001f7dd800 nid=0x1990 waiting on condition [0x000000002773e000] > 2020-01-10T20:13:16.2958452Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2958538Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2958629Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2958736Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2958989Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2959275Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2959375Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2959629Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2959714Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2959792Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2959872Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2959910Z > 2020-01-10T20:13:16.2959983Z "pool-30-thread-11" #142 prio=5 os_prio=0 > tid=0x000000001f7dc800 nid=0x3a8c waiting on condition [0x000000002733f000] > 2020-01-10T20:13:16.2960061Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2960122Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2960209Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2960296Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2960385Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2960481Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2960576Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2960663Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2960741Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2960829Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2960895Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2960938Z > 2020-01-10T20:13:16.2961010Z "pool-30-thread-10" #141 prio=5 os_prio=0 > tid=0x000000001f7d6000 nid=0x1334 waiting on condition [0x0000000026f3e000] > 2020-01-10T20:13:16.2961096Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2961158Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2961244Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2961403Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2961494Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2961585Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2961681Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2961764Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2961909Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2961989Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2962063Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2962101Z > 2020-01-10T20:13:16.2962179Z "pool-30-thread-9" #140 prio=5 os_prio=0 > tid=0x000000001f7d9000 nid=0x31cc waiting on condition [0x0000000026b3f000] > 2020-01-10T20:13:16.2962261Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2962331Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2962412Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2962499Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2962582Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2962688Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2962777Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2962868Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2962953Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2963039Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2963106Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2963151Z > 2020-01-10T20:13:16.2963223Z "pool-30-thread-8" #139 prio=5 os_prio=0 > tid=0x000000001f7d8000 nid=0x43f4 waiting for monitor entry > [0x000000002673e000] > 2020-01-10T20:13:16.2963310Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2963587Z at > com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:30) > 2020-01-10T20:13:16.2963687Z at > com.microsoft.azure.eventhubs.impl.MessageSender.onClose(MessageSender.java:424) > 2020-01-10T20:13:16.2963771Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:70) > 2020-01-10T20:13:16.2963864Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:106) > 2020-01-10T20:13:16.2963963Z at > com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) > 2020-01-10T20:13:16.2964045Z at > org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) > 2020-01-10T20:13:16.2964129Z at > org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) > 2020-01-10T20:13:16.2964254Z at > org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) > 2020-01-10T20:13:16.2964335Z at > org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) > 2020-01-10T20:13:16.2964431Z at > com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:512) > 2020-01-10T20:13:16.2964512Z at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > 2020-01-10T20:13:16.2964592Z at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2020-01-10T20:13:16.2964678Z at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > 2020-01-10T20:13:16.2965031Z at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > 2020-01-10T20:13:16.2965116Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.2965201Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2965483Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2965528Z > 2020-01-10T20:13:16.2965604Z "pool-30-thread-7" #138 prio=5 os_prio=0 > tid=0x000000001f7d5000 nid=0x32d8 waiting on condition [0x000000002633e000] > 2020-01-10T20:13:16.2965752Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2965818Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2965907Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2965992Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2966094Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2966189Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2966288Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2966379Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2966469Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2966556Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2966626Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2966837Z > 2020-01-10T20:13:16.2966923Z "pool-30-thread-6" #137 prio=5 os_prio=0 > tid=0x000000001f7dc000 nid=0x62a0 waiting on condition [0x0000000025f3e000] > 2020-01-10T20:13:16.2967012Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2967089Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2967178Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2967278Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2967378Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2967487Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2967598Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2967690Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2967783Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2967955Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2968076Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2968119Z > 2020-01-10T20:13:16.2968206Z "pool-30-thread-5" #136 prio=5 os_prio=0 > tid=0x000000001f7da800 nid=0x3f24 waiting on condition [0x0000000025b3e000] > 2020-01-10T20:13:16.2968289Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2968363Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2968461Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2968557Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2968651Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2968757Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2968947Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2969049Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2969138Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2969231Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2969373Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2969423Z > 2020-01-10T20:13:16.2969504Z "pool-30-thread-4" #135 prio=5 os_prio=0 > tid=0x000000001f7db000 nid=0x22ec waiting on condition [0x00000000252bf000] > 2020-01-10T20:13:16.2969592Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2969661Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2969756Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2969854Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2970118Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2970400Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2970496Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2970744Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2970830Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2970907Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2970986Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2971031Z > 2020-01-10T20:13:16.2971110Z "pool-30-thread-3" #134 prio=5 os_prio=0 > tid=0x000000001f7d7800 nid=0x56ec waiting on condition [0x0000000024ebe000] > 2020-01-10T20:13:16.2971183Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2971251Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2971331Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2971416Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2971506Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2971606Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2971694Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2971797Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2971889Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2971967Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2972049Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2972086Z > 2020-01-10T20:13:16.2972156Z "pool-30-thread-2" #133 prio=5 os_prio=0 > tid=0x000000001f7d6800 nid=0x4d18 waiting on condition [0x0000000024abf000] > 2020-01-10T20:13:16.2972248Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2972310Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2972405Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2972490Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2972645Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2972743Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2972834Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2972923Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2973002Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2973148Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2973216Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2973259Z > 2020-01-10T20:13:16.2973330Z "pool-30-thread-1" #132 prio=5 os_prio=0 > tid=0x0000000019ee7800 nid=0x6390 waiting on condition [0x000000003873f000] > 2020-01-10T20:13:16.2973417Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2973480Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2973574Z - parking to wait for <0x00000000c3d74118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2973655Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2973753Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2973843Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) > 2020-01-10T20:13:16.2973954Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2974036Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2974121Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2974207Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2974281Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2974318Z > 2020-01-10T20:13:16.2974401Z "spark-listener-group-streams" #130 daemon > prio=5 os_prio=0 tid=0x00000000134f5800 nid=0x2ba0 waiting on condition > [0x0000000037f3e000] > 2020-01-10T20:13:16.2974481Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2974549Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2974629Z - parking to wait for <0x00000000c3a32880> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2974722Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2974806Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2974896Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2974998Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) > 2020-01-10T20:13:16.2975112Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2975223Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2975310Z at > scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > 2020-01-10T20:13:16.2975416Z at > org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2975511Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) > 2020-01-10T20:13:16.2975611Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.2975767Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) > 2020-01-10T20:13:16.2975819Z > 2020-01-10T20:13:16.2975897Z "spark-listener-group-shared" #129 daemon prio=5 > os_prio=0 tid=0x00000000134f4000 nid=0x3010 waiting on condition > [0x0000000037b3e000] > 2020-01-10T20:13:16.2975980Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2976042Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2976130Z - parking to wait for <0x00000000c3d10230> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2976268Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2976359Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2976619Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2976725Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) > 2020-01-10T20:13:16.2976842Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2976952Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2977041Z at > scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > 2020-01-10T20:13:16.2977136Z at > org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2977246Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) > 2020-01-10T20:13:16.2977331Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.2977419Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) > 2020-01-10T20:13:16.2977473Z > 2020-01-10T20:13:16.2977552Z "element-tracking-store-worker" #128 daemon > prio=5 os_prio=0 tid=0x00000000134fa000 nid=0x434c waiting on condition > [0x000000003773f000] > 2020-01-10T20:13:16.2977640Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2977709Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2977965Z - parking to wait for <0x00000000c3b39828> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2978062Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2978167Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2978266Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2978404Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2978511Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2978598Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2978678Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2978720Z > 2020-01-10T20:13:16.2978808Z "spark-listener-group-executorManagement" #118 > daemon prio=5 os_prio=0 tid=0x00000000134fb000 nid=0xaf0 waiting on condition > [0x000000003733f000] > 2020-01-10T20:13:16.2978904Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2978977Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2979075Z - parking to wait for <0x00000000c3b3a818> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2979171Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2979265Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2979459Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2979564Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) > 2020-01-10T20:13:16.2979689Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2979803Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2979973Z at > scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > 2020-01-10T20:13:16.2980071Z at > org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2980180Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) > 2020-01-10T20:13:16.2980285Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.2980373Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) > 2020-01-10T20:13:16.2980423Z > 2020-01-10T20:13:16.2980516Z "spark-listener-group-appStatus" #90 daemon > prio=5 os_prio=0 tid=0x00000000134f2800 nid=0x1f30 waiting on condition > [0x0000000036f3e000] > 2020-01-10T20:13:16.2980603Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2980679Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2980776Z - parking to wait for <0x00000000c3a74eb8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2980873Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2980974Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2981228Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.2981345Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97) > 2020-01-10T20:13:16.2981456Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2981725Z at > org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2981820Z at > scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > 2020-01-10T20:13:16.2981923Z at > org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) > 2020-01-10T20:13:16.2982178Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) > 2020-01-10T20:13:16.2982272Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.2982351Z at > org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) > 2020-01-10T20:13:16.2982402Z > 2020-01-10T20:13:16.2982479Z "context-cleaner-periodic-gc" #127 daemon prio=5 > os_prio=0 tid=0x00000000134f9800 nid=0x51c8 waiting on condition > [0x0000000036b3f000] > 2020-01-10T20:13:16.2982562Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2982626Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2982711Z - parking to wait for <0x00000000c3d74480> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2982800Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2982893Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2982987Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > 2020-01-10T20:13:16.2983153Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2983244Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2983324Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2983407Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2983475Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2983573Z > 2020-01-10T20:13:16.2983657Z "Spark Context Cleaner" #126 daemon prio=5 > os_prio=0 tid=0x00000000134f8000 nid=0x620c waiting for monitor entry > [0x000000003673c000] > 2020-01-10T20:13:16.2983736Z java.lang.Thread.State: BLOCKED (on object > monitor) > 2020-01-10T20:13:16.2983814Z at > java.lang.ClassLoader.checkCerts(ClassLoader.java:887) > 2020-01-10T20:13:16.2983893Z - waiting to lock <0x00000000ca33e4c8> (a > sbt.internal.ManagedClassLoader$ZombieClassLoader) > 2020-01-10T20:13:16.2983986Z at > java.lang.ClassLoader.preDefineClass(ClassLoader.java:668) > 2020-01-10T20:13:16.2984063Z at > java.lang.ClassLoader.defineClass(ClassLoader.java:761) > 2020-01-10T20:13:16.2984301Z at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > 2020-01-10T20:13:16.2984612Z at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) > 2020-01-10T20:13:16.2984682Z at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) > 2020-01-10T20:13:16.2984755Z at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) > 2020-01-10T20:13:16.2984825Z at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) > 2020-01-10T20:13:16.2984897Z at > java.security.AccessController.doPrivileged(Native Method) > 2020-01-10T20:13:16.2984964Z at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) > 2020-01-10T20:13:16.2985042Z at > sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(LayeredClassLoaders.scala:336) > 2020-01-10T20:13:16.2985301Z at > sbt.internal.ManagedClassLoader.findClass(LayeredClassLoaders.scala:375) > 2020-01-10T20:13:16.2985617Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) > 2020-01-10T20:13:16.2985689Z - locked <0x00000000c1f359f0> (a > sbt.internal.LayeredClassLoader) > 2020-01-10T20:13:16.2985767Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) > 2020-01-10T20:13:16.2985845Z at > org.apache.spark.storage.BlockManagerMaster.removeShuffle(BlockManagerMaster.scala:138) > 2020-01-10T20:13:16.2985932Z at > org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:226) > 2020-01-10T20:13:16.2986033Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:192) > 2020-01-10T20:13:16.2986143Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185) > 2020-01-10T20:13:16.2986228Z at scala.Option.foreach(Option.scala:257) > 2020-01-10T20:13:16.2986320Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185) > 2020-01-10T20:13:16.2986404Z - locked <0x00000000c3d74cd0> (a > org.apache.spark.ContextCleaner) > 2020-01-10T20:13:16.2986486Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.2986573Z at > org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178) > 2020-01-10T20:13:16.2986655Z at > org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73) > 2020-01-10T20:13:16.2986704Z > 2020-01-10T20:13:16.2986783Z "shuffle-server-5-1" #125 daemon prio=5 > os_prio=0 tid=0x00000000134f8800 nid=0x1e2c runnable [0x000000003633e000] > 2020-01-10T20:13:16.2986857Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.2986931Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.2987007Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.2987162Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.2987241Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.2987322Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.2987397Z - locked <0x00000000c3c203a0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > 2020-01-10T20:13:16.2987483Z - locked <0x00000000c3c551c0> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.2987804Z - locked <0x00000000c3c202d0> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.2987877Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.2987967Z at > io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) > 2020-01-10T20:13:16.2988051Z at > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) > 2020-01-10T20:13:16.2988137Z at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) > 2020-01-10T20:13:16.2988220Z at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > 2020-01-10T20:13:16.2988317Z at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > 2020-01-10T20:13:16.2988393Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2988437Z > 2020-01-10T20:13:16.2988514Z "driver-heartbeater" #124 daemon prio=5 > os_prio=0 tid=0x00000000134f2000 nid=0x3940 waiting on condition > [0x0000000035f3f000] > 2020-01-10T20:13:16.2988606Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2988671Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2988760Z - parking to wait for <0x00000000c3d00fc8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2988847Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2989138Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2989242Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > 2020-01-10T20:13:16.2989351Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2989444Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2989540Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2989636Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2989717Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2989760Z > 2020-01-10T20:13:16.2989849Z "dag-scheduler-event-loop" #123 daemon prio=5 > os_prio=0 tid=0x00000000134f0800 nid=0x3a40 waiting on condition > [0x0000000035b3f000] > 2020-01-10T20:13:16.2989944Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.2990020Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2990108Z - parking to wait for <0x00000000c3ce7280> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2990208Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.2990303Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.2990405Z at > java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) > 2020-01-10T20:13:16.2990499Z at > java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) > 2020-01-10T20:13:16.2990590Z at > org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:47) > 2020-01-10T20:13:16.2990637Z > 2020-01-10T20:13:16.2990727Z "netty-rpc-env-timeout" #122 daemon prio=5 > os_prio=0 tid=0x00000000134f1000 nid=0x296c waiting on condition > [0x000000003573e000] > 2020-01-10T20:13:16.2990889Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2990966Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2991057Z - parking to wait for <0x00000000c3dbd078> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2991157Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2991255Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2991429Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > 2020-01-10T20:13:16.2991529Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2991629Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2991724Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2991819Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2991898Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2991942Z > 2020-01-10T20:13:16.2992021Z "Timer-3" #121 daemon prio=5 os_prio=0 > tid=0x00000000134f6800 nid=0xef8 in Object.wait() [0x000000003533e000] > 2020-01-10T20:13:16.2992111Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2992182Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2992429Z - waiting on <0x00000000c3c4b628> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2992667Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2992741Z at java.util.TimerThread.mainLoop(Timer.java:526) > 2020-01-10T20:13:16.2992809Z - locked <0x00000000c3c4b628> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2993043Z at java.util.TimerThread.run(Timer.java:505) > 2020-01-10T20:13:16.2993088Z > 2020-01-10T20:13:16.2993162Z "Timer-2" #120 daemon prio=5 os_prio=0 > tid=0x00000000134f3800 nid=0x163c in Object.wait() [0x0000000034f3f000] > 2020-01-10T20:13:16.2993238Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.2993304Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.2993370Z - waiting on <0x00000000c3c4aca0> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2993436Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.2993498Z at java.util.TimerThread.mainLoop(Timer.java:526) > 2020-01-10T20:13:16.2993569Z - locked <0x00000000c3c4aca0> (a > java.util.TaskQueue) > 2020-01-10T20:13:16.2993638Z at java.util.TimerThread.run(Timer.java:505) > 2020-01-10T20:13:16.2993676Z > 2020-01-10T20:13:16.2993759Z "heartbeat-receiver-event-loop-thread" #119 > daemon prio=5 os_prio=0 tid=0x00000000134f7000 nid=0x4984 waiting on > condition [0x0000000034b3f000] > 2020-01-10T20:13:16.2993844Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2993913Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2993999Z - parking to wait for <0x00000000c3de5808> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2994245Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2994335Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2994425Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > 2020-01-10T20:13:16.2994523Z at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > 2020-01-10T20:13:16.2994604Z at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > 2020-01-10T20:13:16.2994686Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > 2020-01-10T20:13:16.2994761Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.2994902Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2994940Z > 2020-01-10T20:13:16.2995017Z "SparkUI-117" #117 daemon prio=5 os_prio=0 > tid=0x0000000012be9800 nid=0x6208 waiting on condition [0x000000003473f000] > 2020-01-10T20:13:16.2995090Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2995155Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2995234Z - parking to wait for <0x00000000c3de5f90> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2995376Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2995461Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2995550Z at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) > 2020-01-10T20:13:16.2995638Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) > 2020-01-10T20:13:16.2995727Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) > 2020-01-10T20:13:16.2995808Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > 2020-01-10T20:13:16.2995882Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2995920Z > 2020-01-10T20:13:16.2995994Z "SparkUI-116" #116 daemon prio=5 os_prio=0 > tid=0x0000000012be2000 nid=0x3b34 waiting on condition [0x000000003433e000] > 2020-01-10T20:13:16.2996072Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2996139Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2996218Z - parking to wait for <0x00000000c3de5f90> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2996304Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2996394Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2996483Z at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) > 2020-01-10T20:13:16.2996564Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) > 2020-01-10T20:13:16.2996650Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) > 2020-01-10T20:13:16.2996730Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > 2020-01-10T20:13:16.2996810Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2996846Z > 2020-01-10T20:13:16.2996935Z > "SparkUI-115-acceptor-0@4be1e6e6-ServerConnector@452601ea\{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" > #115 daemon prio=3 os_prio=-1 tid=0x0000000012be9000 nid=0x3a54 runnable > [0x0000000033f3e000] > 2020-01-10T20:13:16.2997016Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.2997094Z at > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > 2020-01-10T20:13:16.2997168Z at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > 2020-01-10T20:13:16.2997250Z at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > 2020-01-10T20:13:16.2997317Z - locked <0x00000000c3e3e9a8> (a > java.lang.Object) > 2020-01-10T20:13:16.2997397Z at > org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397) > 2020-01-10T20:13:16.2997477Z at > org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) > 2020-01-10T20:13:16.2997571Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > 2020-01-10T20:13:16.2997656Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > 2020-01-10T20:13:16.2997724Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2997760Z > 2020-01-10T20:13:16.2997836Z "SparkUI-114" #114 daemon prio=5 os_prio=0 > tid=0x0000000012be6000 nid=0x5474 waiting on condition [0x0000000033b3f000] > 2020-01-10T20:13:16.2997974Z java.lang.Thread.State: TIMED_WAITING (parking) > 2020-01-10T20:13:16.2998040Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.2998118Z - parking to wait for <0x00000000c3de5f90> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.2998204Z at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > 2020-01-10T20:13:16.2998288Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > 2020-01-10T20:13:16.2998439Z at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) > 2020-01-10T20:13:16.2998527Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) > 2020-01-10T20:13:16.2998609Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) > 2020-01-10T20:13:16.2998702Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > 2020-01-10T20:13:16.2998768Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.2998806Z > 2020-01-10T20:13:16.2998878Z "SparkUI-113" #113 daemon prio=5 os_prio=0 > tid=0x0000000012be3800 nid=0x22b4 runnable [0x000000003373e000] > 2020-01-10T20:13:16.2998946Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.2999024Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.2999098Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.2999189Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.2999270Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.2999341Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.2999411Z - locked <0x00000000c3e3aac0> (a > sun.nio.ch.Util$3) > 2020-01-10T20:13:16.2999488Z - locked <0x00000000c3e3aab0> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.2999564Z - locked <0x00000000c3e3a960> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.2999631Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.2999702Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > 2020-01-10T20:13:16.2999779Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) > 2020-01-10T20:13:16.2999871Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) > 2020-01-10T20:13:16.2999966Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) > 2020-01-10T20:13:16.3000064Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > 2020-01-10T20:13:16.3000154Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) > 2020-01-10T20:13:16.3000245Z at > org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147) > 2020-01-10T20:13:16.3000326Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > 2020-01-10T20:13:16.3000410Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > 2020-01-10T20:13:16.3000477Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3000520Z > 2020-01-10T20:13:16.3000588Z "SparkUI-112" #112 daemon prio=5 os_prio=0 > tid=0x0000000012be6800 nid=0x21c4 runnable [0x000000003333e000] > 2020-01-10T20:13:16.3000667Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3000732Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.3000813Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.3000891Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.3001039Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.3001111Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.3001182Z - locked <0x00000000c3de66a8> (a > sun.nio.ch.Util$3) > 2020-01-10T20:13:16.3001252Z - locked <0x00000000c3de6698> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.3001328Z - locked <0x00000000c3de6548> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.3001396Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.3001467Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > 2020-01-10T20:13:16.3001604Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) > 2020-01-10T20:13:16.3001693Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) > 2020-01-10T20:13:16.3001940Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) > 2020-01-10T20:13:16.3002040Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > 2020-01-10T20:13:16.3002129Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) > 2020-01-10T20:13:16.3002206Z at > org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147) > 2020-01-10T20:13:16.3002289Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > 2020-01-10T20:13:16.3002367Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > 2020-01-10T20:13:16.3002442Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3002478Z > 2020-01-10T20:13:16.3002543Z "SparkUI-111" #111 daemon prio=5 os_prio=0 > tid=0x0000000012be3000 nid=0x1cbc runnable [0x0000000032f3e000] > 2020-01-10T20:13:16.3002615Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3002686Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.3002764Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.3002852Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.3002924Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.3002998Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.3003062Z - locked <0x00000000c3dea598> (a > sun.nio.ch.Util$3) > 2020-01-10T20:13:16.3003134Z - locked <0x00000000c3dea588> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.3003208Z - locked <0x00000000c3dea438> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.3003278Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.3003342Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > 2020-01-10T20:13:16.3003420Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) > 2020-01-10T20:13:16.3003506Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) > 2020-01-10T20:13:16.3003596Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) > 2020-01-10T20:13:16.3003683Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > 2020-01-10T20:13:16.3003774Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) > 2020-01-10T20:13:16.3003856Z at > org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147) > 2020-01-10T20:13:16.3003939Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > 2020-01-10T20:13:16.3004016Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > 2020-01-10T20:13:16.3004085Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3004182Z > 2020-01-10T20:13:16.3004254Z "SparkUI-110" #110 daemon prio=5 os_prio=0 > tid=0x0000000012be4800 nid=0x3920 runnable [0x0000000032b3e000] > 2020-01-10T20:13:16.3004321Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3004390Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.3004462Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.3004543Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.3004616Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.3004749Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.3004814Z - locked <0x00000000c3e3f0c8> (a > sun.nio.ch.Util$3) > 2020-01-10T20:13:16.3004887Z - locked <0x00000000c3e3f0b8> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.3004956Z - locked <0x00000000c3e3ef68> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.3005802Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.3005881Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > 2020-01-10T20:13:16.3005976Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) > 2020-01-10T20:13:16.3006084Z at > org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) > 2020-01-10T20:13:16.3006188Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) > 2020-01-10T20:13:16.3006307Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > 2020-01-10T20:13:16.3006409Z at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) > 2020-01-10T20:13:16.3006507Z at > org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147) > 2020-01-10T20:13:16.3006605Z at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > 2020-01-10T20:13:16.3006703Z at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > 2020-01-10T20:13:16.3006780Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3006829Z > 2020-01-10T20:13:16.3006915Z "RemoteBlock-temp-file-clean-thread" #109 daemon > prio=5 os_prio=0 tid=0x0000000012be5000 nid=0x3978 in Object.wait() > [0x000000003273f000] > 2020-01-10T20:13:16.3007014Z java.lang.Thread.State: TIMED_WAITING (on object > monitor) > 2020-01-10T20:13:16.3007093Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.3007175Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > 2020-01-10T20:13:16.3007257Z - locked <0x00000000c3cfec80> (a > java.lang.ref.ReferenceQueue$Lock) > 2020-01-10T20:13:16.3007382Z at > org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:1724) > 2020-01-10T20:13:16.3007504Z at > org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:1692) > 2020-01-10T20:13:16.3007564Z > 2020-01-10T20:13:16.3007649Z "map-output-dispatcher-7" #107 daemon prio=5 > os_prio=0 tid=0x0000000012be8000 nid=0x1dc0 waiting on condition > [0x000000003233e000] > 2020-01-10T20:13:16.3007740Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3007810Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3007905Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3008004Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3008105Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3008198Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3008415Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3008508Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3008602Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3008678Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3008867Z > 2020-01-10T20:13:16.3009124Z "map-output-dispatcher-6" #106 daemon prio=5 > os_prio=0 tid=0x0000000012be7800 nid=0x4930 waiting on condition > [0x0000000031f3e000] > 2020-01-10T20:13:16.3010302Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3010365Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3010448Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3010525Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3010620Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3010700Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3010783Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3010858Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3010938Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3011015Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3011052Z > 2020-01-10T20:13:16.3011122Z "map-output-dispatcher-5" #105 daemon prio=5 > os_prio=0 tid=0x0000000012bdb800 nid=0x320c waiting on condition > [0x0000000031b3f000] > 2020-01-10T20:13:16.3011200Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3011259Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3011347Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3011423Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3011683Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3011765Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3011848Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3011937Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3012013Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3012083Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3012120Z > 2020-01-10T20:13:16.3012192Z "map-output-dispatcher-4" #104 daemon prio=5 > os_prio=0 tid=0x0000000012bda800 nid=0x1180 waiting on condition > [0x000000003173f000] > 2020-01-10T20:13:16.3012282Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3012342Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3012427Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3012505Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3012592Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3012684Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3012761Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3012842Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3012919Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3013075Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3013111Z > 2020-01-10T20:13:16.3013183Z "map-output-dispatcher-3" #103 daemon prio=5 > os_prio=0 tid=0x0000000012bdc000 nid=0x27f4 waiting on condition > [0x000000003133f000] > 2020-01-10T20:13:16.3013265Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3013325Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3013409Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3014283Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3014372Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3014624Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3014699Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3014792Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3014866Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3014941Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3014976Z > 2020-01-10T20:13:16.3015046Z "map-output-dispatcher-2" #102 daemon prio=5 > os_prio=0 tid=0x0000000012bdd000 nid=0x2e04 waiting on condition > [0x0000000030f3e000] > 2020-01-10T20:13:16.3015127Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3015197Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3015273Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3015356Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3015435Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3015525Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3015599Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3015681Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3015754Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3015829Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3015870Z > 2020-01-10T20:13:16.3015951Z "map-output-dispatcher-1" #101 daemon prio=5 > os_prio=0 tid=0x0000000012be1800 nid=0x358c waiting on condition > [0x0000000030b3f000] > 2020-01-10T20:13:16.3016022Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3016091Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3016166Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3016263Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3016340Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3016427Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3016500Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3016589Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3016667Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3016741Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3016775Z > 2020-01-10T20:13:16.3016851Z "map-output-dispatcher-0" #100 daemon prio=5 > os_prio=0 tid=0x0000000012bde800 nid=0xfe8 waiting on condition > [0x000000003073f000] > 2020-01-10T20:13:16.3016994Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3017061Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3017136Z - parking to wait for <0x00000000c3b84b08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3017220Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3017297Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3017381Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3017884Z at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384) > 2020-01-10T20:13:16.3017970Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3018043Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3018120Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3018156Z > 2020-01-10T20:13:16.3018229Z "rpc-server-3-1" #99 daemon prio=5 os_prio=0 > tid=0x0000000012be0800 nid=0x63d0 runnable [0x000000003033e000] > 2020-01-10T20:13:16.3018295Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3018365Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > 2020-01-10T20:13:16.3018437Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > 2020-01-10T20:13:16.3018517Z at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > 2020-01-10T20:13:16.3018598Z at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > 2020-01-10T20:13:16.3018674Z at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > 2020-01-10T20:13:16.3018744Z - locked <0x00000000c3e64038> (a > io.netty.channel.nio.SelectedSelectionKeySet) > 2020-01-10T20:13:16.3018821Z - locked <0x00000000c3ec1920> (a > java.util.Collections$UnmodifiableSet) > 2020-01-10T20:13:16.3018897Z - locked <0x00000000c3e63f68> (a > sun.nio.ch.WindowsSelectorImpl) > 2020-01-10T20:13:16.3018968Z at > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > 2020-01-10T20:13:16.3019043Z at > io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) > 2020-01-10T20:13:16.3019126Z at > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) > 2020-01-10T20:13:16.3019194Z at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) > 2020-01-10T20:13:16.3019278Z at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > 2020-01-10T20:13:16.3019367Z at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > 2020-01-10T20:13:16.3019442Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3019477Z > 2020-01-10T20:13:16.3019553Z "dispatcher-event-loop-7" #98 daemon prio=5 > os_prio=0 tid=0x0000000012bdf000 nid=0x58f4 waiting on condition > [0x000000002ff3e000] > 2020-01-10T20:13:16.3019630Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3019700Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3019776Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3019857Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3019937Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3020019Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3020100Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3020182Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3020262Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3020397Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3020433Z > 2020-01-10T20:13:16.3020510Z "dispatcher-event-loop-6" #97 daemon prio=5 > os_prio=0 tid=0x0000000012be0000 nid=0x1b6c waiting on condition > [0x000000002fb3f000] > 2020-01-10T20:13:16.3020580Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3020644Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3020719Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3020800Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3021399Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3021493Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3021573Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3021656Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3021736Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3021799Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3021834Z > 2020-01-10T20:13:16.3021913Z "dispatcher-event-loop-5" #96 daemon prio=5 > os_prio=0 tid=0x0000000012bdd800 nid=0x4888 waiting on condition > [0x000000002f73e000] > 2020-01-10T20:13:16.3021984Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3022050Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3022307Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3022393Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3022479Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3022566Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3022873Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3022954Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3023036Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3023104Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3023141Z > 2020-01-10T20:13:16.3023229Z "dispatcher-event-loop-4" #95 daemon prio=5 > os_prio=0 tid=0x0000000019eea800 nid=0x2b3c waiting on condition > [0x000000002f33f000] > 2020-01-10T20:13:16.3023311Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3023470Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3023550Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3023637Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3023888Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3023970Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3024050Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3024125Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3024207Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3024278Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3024319Z > 2020-01-10T20:13:16.3024390Z "dispatcher-event-loop-3" #94 daemon prio=5 > os_prio=0 tid=0x00000000151b8000 nid=0x6254 waiting on condition > [0x000000002ef3f000] > 2020-01-10T20:13:16.3024467Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3025337Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3025430Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3025678Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3025768Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3025847Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3025928Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3026072Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3026154Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3026216Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3026257Z > 2020-01-10T20:13:16.3026333Z "dispatcher-event-loop-2" #93 daemon prio=5 > os_prio=0 tid=0x00000000151b7000 nid=0x10c4 waiting on condition > [0x000000002eb3e000] > 2020-01-10T20:13:16.3026411Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3026468Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3026549Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3026625Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3026708Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3026793Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3026905Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3026979Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3027063Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3027126Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3027168Z > 2020-01-10T20:13:16.3027237Z "dispatcher-event-loop-1" #92 daemon prio=5 > os_prio=0 tid=0x00000000151b6800 nid=0x5cf4 waiting on condition > [0x000000002e73f000] > 2020-01-10T20:13:16.3027312Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3027370Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3027458Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3027540Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3027625Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3027704Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3027789Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3027862Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3027942Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3028005Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3028046Z > 2020-01-10T20:13:16.3028115Z "dispatcher-event-loop-0" #91 daemon prio=5 > os_prio=0 tid=0x00000000151b5000 nid=0x960 waiting on condition > [0x000000002e33e000] > 2020-01-10T20:13:16.3028197Z java.lang.Thread.State: WAITING (parking) > 2020-01-10T20:13:16.3028255Z at sun.misc.Unsafe.park(Native Method) > 2020-01-10T20:13:16.3028336Z - parking to wait for <0x00000000c38337c0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > 2020-01-10T20:13:16.3028412Z at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > 2020-01-10T20:13:16.3028498Z at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > 2020-01-10T20:13:16.3029049Z at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > 2020-01-10T20:13:16.3029130Z at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215) > 2020-01-10T20:13:16.3029203Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3029282Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3029345Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3029452Z > 2020-01-10T20:13:16.3029524Z "classloader-cache-cleanup-0" #14 daemon prio=5 > os_prio=0 tid=0x0000000013e0c000 nid=0x4e24 in Object.wait() > [0x00000000196ae000] > 2020-01-10T20:13:16.3029609Z java.lang.Thread.State: TIMED_WAITING (on object > monitor) > 2020-01-10T20:13:16.3029670Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.3029740Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > 2020-01-10T20:13:16.3029816Z - locked <0x00000000c13d6698> (a > java.lang.ref.ReferenceQueue$Lock) > 2020-01-10T20:13:16.3029898Z at > sbt.internal.classpath.ClassLoaderCache$CleanupThread.run(ClassLoaderCache.scala:102) > 2020-01-10T20:13:16.3029942Z > 2020-01-10T20:13:16.3030012Z "Service Thread" #10 daemon prio=9 os_prio=0 > tid=0x000000000f400000 nid=0x346c runnable [0x0000000000000000] > 2020-01-10T20:13:16.3030080Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3030114Z > 2020-01-10T20:13:16.3030189Z "C1 CompilerThread3" #9 daemon prio=9 os_prio=2 > tid=0x000000000f36e000 nid=0x37b8 waiting on condition [0x0000000000000000] > 2020-01-10T20:13:16.3030264Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3030306Z > 2020-01-10T20:13:16.3030374Z "C2 CompilerThread2" #8 daemon prio=9 os_prio=2 > tid=0x000000000f36a000 nid=0x5898 waiting on condition [0x0000000000000000] > 2020-01-10T20:13:16.3030455Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3030495Z > 2020-01-10T20:13:16.3030568Z "C2 CompilerThread1" #7 daemon prio=9 os_prio=2 > tid=0x000000000f368000 nid=0x62dc waiting on condition [0x0000000000000000] > 2020-01-10T20:13:16.3030636Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3030669Z > 2020-01-10T20:13:16.3030743Z "C2 CompilerThread0" #6 daemon prio=9 os_prio=2 > tid=0x000000000f366800 nid=0x28fc waiting on condition [0x0000000000000000] > 2020-01-10T20:13:16.3030810Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3030851Z > 2020-01-10T20:13:16.3030917Z "Attach Listener" #5 daemon prio=5 os_prio=2 > tid=0x000000000f361000 nid=0x4c30 runnable [0x0000000000000000] > 2020-01-10T20:13:16.3030996Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3031030Z > 2020-01-10T20:13:16.3031098Z "Signal Dispatcher" #4 daemon prio=9 os_prio=2 > tid=0x000000000f360000 nid=0x3408 waiting on condition [0x0000000000000000] > 2020-01-10T20:13:16.3031171Z java.lang.Thread.State: RUNNABLE > 2020-01-10T20:13:16.3031204Z > 2020-01-10T20:13:16.3031281Z "Finalizer" #3 daemon prio=8 os_prio=1 > tid=0x000000000d543800 nid=0x34b4 in Object.wait() [0x000000000ff2f000] > 2020-01-10T20:13:16.3031352Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.3031417Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.3031483Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > 2020-01-10T20:13:16.3031558Z - locked <0x00000000c000bd38> (a > java.lang.ref.ReferenceQueue$Lock) > 2020-01-10T20:13:16.3031627Z at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) > 2020-01-10T20:13:16.3031700Z at > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) > 2020-01-10T20:13:16.3031745Z > 2020-01-10T20:13:16.3031819Z "Reference Handler" #2 daemon prio=10 os_prio=2 > tid=0x000000000d53c800 nid=0x4f44 in Object.wait() [0x000000000fb2f000] > 2020-01-10T20:13:16.3031891Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.3031957Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.3032447Z at java.lang.Object.wait(Object.java:502) > 2020-01-10T20:13:16.3032522Z at > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > 2020-01-10T20:13:16.3032589Z - locked <0x00000000c0012c40> (a > java.lang.ref.Reference$Lock) > 2020-01-10T20:13:16.3032665Z at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > 2020-01-10T20:13:16.3032705Z > 2020-01-10T20:13:16.3032776Z "main" #1 prio=5 os_prio=0 > tid=0x00000000015e4800 nid=0x4938 in Object.wait() [0x0000000001bde000] > 2020-01-10T20:13:16.3032846Z java.lang.Thread.State: WAITING (on object > monitor) > 2020-01-10T20:13:16.3032918Z at java.lang.Object.wait(Native Method) > 2020-01-10T20:13:16.3033046Z at java.lang.Thread.join(Thread.java:1252) > 2020-01-10T20:13:16.3033293Z - locked <0x00000000c370a108> (a > org.apache.hadoop.util.ShutdownHookManager$1) > 2020-01-10T20:13:16.3033360Z at java.lang.Thread.join(Thread.java:1326) > 2020-01-10T20:13:16.3033436Z at > java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107) > 2020-01-10T20:13:16.3033519Z at > java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) > 2020-01-10T20:13:16.3033592Z at java.lang.Shutdown.runHooks(Shutdown.java:123) > 2020-01-10T20:13:16.3033653Z at java.lang.Shutdown.sequence(Shutdown.java:167) > 2020-01-10T20:13:16.3033719Z at java.lang.Shutdown.exit(Shutdown.java:212) > 2020-01-10T20:13:16.3033787Z - locked <0x00000000c0155de0> (a java.lang.Class > for java.lang.Shutdown) > 2020-01-10T20:13:16.3033860Z at java.lang.Runtime.exit(Runtime.java:109) > 2020-01-10T20:13:16.3033919Z at java.lang.System.exit(System.java:971) > 2020-01-10T20:13:16.3033999Z at > xsbt.boot.Boot$.xsbt$boot$Boot$$exit(Boot.scala:74) > 2020-01-10T20:13:16.3034066Z at > xsbt.boot.Boot$$anonfun$runImpl$1.apply(Boot.scala:56) > 2020-01-10T20:13:16.3034139Z at scala.Option.map(Option.scala:145) > 2020-01-10T20:13:16.3034197Z at xsbt.boot.Boot$.runImpl(Boot.scala:56) > 2020-01-10T20:13:16.3034267Z at xsbt.boot.Boot$.main(Boot.scala:18) > 2020-01-10T20:13:16.3034325Z at xsbt.boot.Boot.main(Boot.scala) > 2020-01-10T20:13:16.3034366Z > 2020-01-10T20:13:16.3034439Z "VM Thread" os_prio=2 tid=0x000000000d539000 > nid=0xd8c runnable > 2020-01-10T20:13:16.3034478Z > 2020-01-10T20:13:16.3034552Z "GC task thread#0 (ParallelGC)" os_prio=0 > tid=0x00000000015fa000 nid=0xf68 runnable > 2020-01-10T20:13:16.3034597Z > 2020-01-10T20:13:16.3034675Z "GC task thread#1 (ParallelGC)" os_prio=0 > tid=0x00000000015fc000 nid=0x4914 runnable > 2020-01-10T20:13:16.3034719Z > 2020-01-10T20:13:16.3034780Z "GC task thread#2 (ParallelGC)" os_prio=0 > tid=0x00000000015fd800 nid=0x4e3c runnable > 2020-01-10T20:13:16.3034843Z > 2020-01-10T20:13:16.3034905Z "GC task thread#3 (ParallelGC)" os_prio=0 > tid=0x0000000001600000 nid=0x2f54 runnable > 2020-01-10T20:13:16.3034946Z > 2020-01-10T20:13:16.3035020Z "GC task thread#4 (ParallelGC)" os_prio=0 > tid=0x0000000001601000 nid=0x30b4 runnable > 2020-01-10T20:13:16.3035062Z > 2020-01-10T20:13:16.3035135Z "GC task thread#5 (ParallelGC)" os_prio=0 > tid=0x0000000001602000 nid=0x4d78 runnable > 2020-01-10T20:13:16.3035183Z > 2020-01-10T20:13:16.3035252Z "GC task thread#6 (ParallelGC)" os_prio=0 > tid=0x0000000001605800 nid=0x4330 runnable > 2020-01-10T20:13:16.3035294Z > 2020-01-10T20:13:16.3035356Z "GC task thread#7 (ParallelGC)" os_prio=0 > tid=0x0000000001606800 nid=0x4c94 runnable > 2020-01-10T20:13:16.3035404Z > 2020-01-10T20:13:16.3035468Z "VM Periodic Task Thread" os_prio=2 > tid=0x000000000f426000 nid=0x3034 waiting on condition > 2020-01-10T20:13:16.3035513Z > 2020-01-10T20:13:16.3035567Z JNI global references: 5610 > 2020-01-10T20:13:16.3035601Z > 2020-01-10T20:13:16.3035628Z > 2020-01-10T20:13:16.3035689Z Found one Java-level deadlock: > 2020-01-10T20:13:16.3035745Z ============================= > 2020-01-10T20:13:16.3035806Z "Thread-3": > 2020-01-10T20:13:16.3035876Z waiting to lock monitor 0x000000001fb7e568 > (object 0x00000000c1f359f0, a sbt.internal.LayeredClassLoader), > 2020-01-10T20:13:16.3035952Z which is held by "Spark Context Cleaner" > 2020-01-10T20:13:16.3036009Z "Spark Context Cleaner": > 2020-01-10T20:13:16.3036609Z waiting to lock monitor 0x000000000d540db8 > (object 0x00000000ca33e4c8, a > sbt.internal.ManagedClassLoader$ZombieClassLoader), > 2020-01-10T20:13:16.3036689Z which is held by > "block-manager-slave-async-thread-pool-81" > 2020-01-10T20:13:16.3036760Z "block-manager-slave-async-thread-pool-81": > 2020-01-10T20:13:16.3036837Z waiting to lock monitor 0x000000001fb7e568 > (object 0x00000000c1f359f0, a sbt.internal.LayeredClassLoader), > 2020-01-10T20:13:16.3036912Z which is held by "Spark Context Cleaner" > 2020-01-10T20:13:16.3036949Z > 2020-01-10T20:13:16.3037009Z Java stack information for the threads listed > above: > 2020-01-10T20:13:16.3037144Z > =================================================== > 2020-01-10T20:13:16.3037215Z "Thread-3": > 2020-01-10T20:13:16.3037283Z at > org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:273) > 2020-01-10T20:13:16.3037371Z at > org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:121) > 2020-01-10T20:13:16.3037458Z at > org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95) > 2020-01-10T20:13:16.3037528Z "Spark Context Cleaner": > 2020-01-10T20:13:16.3037590Z at > java.lang.ClassLoader.checkCerts(ClassLoader.java:887) > 2020-01-10T20:13:16.3037672Z - waiting to lock <0x00000000ca33e4c8> (a > sbt.internal.ManagedClassLoader$ZombieClassLoader) > 2020-01-10T20:13:16.3037747Z at > java.lang.ClassLoader.preDefineClass(ClassLoader.java:668) > 2020-01-10T20:13:16.3037821Z at > java.lang.ClassLoader.defineClass(ClassLoader.java:761) > 2020-01-10T20:13:16.3037895Z at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > 2020-01-10T20:13:16.3037982Z at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) > 2020-01-10T20:13:16.3038052Z at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) > 2020-01-10T20:13:16.3038132Z at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) > 2020-01-10T20:13:16.3038199Z at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) > 2020-01-10T20:13:16.3038280Z at > java.security.AccessController.doPrivileged(Native Method) > 2020-01-10T20:13:16.3038356Z at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) > 2020-01-10T20:13:16.3038435Z at > sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(LayeredClassLoaders.scala:336) > 2020-01-10T20:13:16.3038519Z at > sbt.internal.ManagedClassLoader.findClass(LayeredClassLoaders.scala:375) > 2020-01-10T20:13:16.3038590Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:424) > 2020-01-10T20:13:16.3038664Z - locked <0x00000000c1f359f0> (a > sbt.internal.LayeredClassLoader) > 2020-01-10T20:13:16.3038739Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) > 2020-01-10T20:13:16.3038819Z at > org.apache.spark.storage.BlockManagerMaster.removeShuffle(BlockManagerMaster.scala:138) > 2020-01-10T20:13:16.3038899Z at > org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:226) > 2020-01-10T20:13:16.3038996Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:192) > 2020-01-10T20:13:16.3039104Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185) > 2020-01-10T20:13:16.3039187Z at scala.Option.foreach(Option.scala:257) > 2020-01-10T20:13:16.3039268Z at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185) > 2020-01-10T20:13:16.3039357Z - locked <0x00000000c3d74cd0> (a > org.apache.spark.ContextCleaner) > 2020-01-10T20:13:16.3039438Z at > org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302) > 2020-01-10T20:13:16.3039529Z at > org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178) > 2020-01-10T20:13:16.3039607Z at > org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73) > 2020-01-10T20:13:16.3039679Z "block-manager-slave-async-thread-pool-81": > 2020-01-10T20:13:16.3039814Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:404) > 2020-01-10T20:13:16.3039893Z - waiting to lock <0x00000000c1f359f0> (a > sbt.internal.LayeredClassLoader) > 2020-01-10T20:13:16.3039990Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:411) > 2020-01-10T20:13:16.3040063Z - locked <0x00000000ca33e4c8> (a > sbt.internal.ManagedClassLoader$ZombieClassLoader) > 2020-01-10T20:13:16.3040140Z at > java.lang.ClassLoader.loadClass(ClassLoader.java:357) > 2020-01-10T20:13:16.3040228Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply$mcZ$sp(BlockManagerSlaveEndpoint.scala:58) > 2020-01-10T20:13:16.3040393Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57) > 2020-01-10T20:13:16.3040496Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$3.apply(BlockManagerSlaveEndpoint.scala:57) > 2020-01-10T20:13:16.3040600Z at > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:86) > 2020-01-10T20:13:16.3040683Z at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > 2020-01-10T20:13:16.3040767Z at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > 2020-01-10T20:13:16.3040844Z at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-01-10T20:13:16.3040928Z at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-01-10T20:13:16.3040999Z at java.lang.Thread.run(Thread.java:748) > 2020-01-10T20:13:16.3041053Z > 2020-01-10T20:13:16.3041101Z Found 1 deadlock. > 2020-01-10T20:13:16.3041130Z -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org