[ https://issues.apache.org/jira/browse/SPARK-26609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aman Mundra updated SPARK-26609: -------------------------------- Attachment: 2.PNG 1.PNG > Kinesis-Spark Stream unable to process records > ---------------------------------------------- > > Key: SPARK-26609 > URL: https://issues.apache.org/jira/browse/SPARK-26609 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 2.2.0 > Environment: > {code:java} > <spark.version>2.2.0</spark.version> > <!-- Spark Core --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core_${scala.binary.version}</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- Spark SQL --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-sql_${scala.binary.version}</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- Spark Hive --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-hive_${scala.binary.version}</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- Spark MlLib --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-mllib_${scala.binary.version}</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- Spark Streaming --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-streaming_${scala.binary.version}</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- > https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl > --> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-streaming-kinesis-asl_2.11</artifactId> > <version>${spark.version}</version> > </dependency> > <!-- https://mvnrepository.com/artifact/com.databricks/spark-redshift --> > <dependency> > <groupId>com.databricks</groupId> > <artifactId>spark-redshift_2.11</artifactId> > <version>3.0.0-preview1</version> > </dependency> > <!-- https://mvnrepository.com/artifact/com.amazon.redshift/redshift-jdbc42 > --> > <dependency> > <groupId>com.amazon.redshift</groupId> > <artifactId>redshift-jdbc42</artifactId> > <version>1.2.18.1036</version> > </dependency> > {code} > > > spark.driver.cores=6 > spark.driver.memory=12g > spark.yarn.driver.memoryOverhead=1g > spark.driver.maxResultSize=4g > spark.executor.memory=8g > spark.executor.cores=4 > spark.yarn.executor.memoryOverhead=1g > spark.executor.instances=4 > spark.shuffle.service.enabled=true > spark.shuffle.registration.timeout=600 > spark.sql.shuffle.partitions=8 > spark.scheduler.mode=FIFO > maximizeResourceAllocation=true > spark.dynamicAllocation.enabled=true > spark.dynamicAllocation.executorIdleTimeout=60s > > Reporter: Aman Mundra > Priority: Major > Attachments: 1.PNG, 2.PNG > > > I'm trying to consume kinesis stream via spark streaming and amazon KCL lib. > Streaming job gets stuck at processing as so > on as it gets the first batch of non zero records. > I'm getting json data in my kinesis stream and here's what I'm trying to > achieve: > Get Dstream[ArrayByte] > convert to Dstream[String] > RDD > load as json to > create dataframe and perform transformations. > > Similar error links: > [https://stackoverflow.com/questions/40225135/spark-streaming-kafka-job-stuck-in-processing-stage] > I'm running the job in emr-5.8.0 with enough number of cores and executors > but still the job gets stuck in processing stage and build a huge pile of > queued batches over time. > Not able to process even a single record. > > Here's the code I'm using: > > > {code:java} > val numStreams=2 > val sparkStreamingBatchInterval=10 > val kinesisCheckpointInterval=5 > > val kinesisStreams = (0 until kinesisConfig("numStreams").toInt).map { i => > KinesisInputDStream.builder > .streamingContext(ssc) > .endpointUrl(kinesisConfig("endpointUrl")) > .regionName(kinesisConfig("regionName")) > .streamName(kinesisConfig("streamName")) > .initialPositionInStream(InitialPositionInStream.LATEST) > .checkpointAppName(kinesisConfig("appName")) > > .checkpointInterval(Seconds(kinesisConfig("kinesisCheckpointInterval").toInt)) > .storageLevel(StorageLevel.MEMORY_AND_DISK_2) > .kinesisCredentials(awsCredentials.build()) > .build() > } > val unionStreams = ssc.union(kinesisStreams) > val lines = unionStreams.flatMap(byteArray => new String(byteArray).split(" > ")) > lines.print(2) > lines.foreachRDD(rdd => { > if > (!rdd.partitions.isEmpty){ > println("New records found\nmetrics count in the batch: > %s".format(rdd.count())) //works > println("performing transformations") > rdd.saveAsTextFile("path")//works > import sparkSession.implicits._ > println(rdd.toString()) //not working > val records = rdd.toDF("records") //not working > println(records.take(2)) //not working > println(records.count()) //not working > } > else > println("No new record found") > }) > > {code} > > Attaching Thread dump: > h3. Thread dump for executor 2 > Updated at 2019/01/12 10:22:52 > > Collapse All > Search: > > ||Thread ID||Thread Name||Thread State||Thread Locks|| > |65|Executor task launch worker for task > 70|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1560902703})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > org.apache.spark.streaming.receiver.ReceiverSupervisor.awaitTermination(ReceiverSupervisor.scala:219) > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:608) > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597) > org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173) > org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173) > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > org.apache.spark.scheduler.Task.run(Task.scala:108) > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |123|Attach Listener|RUNNABLE| | > | | > |75|cw-metrics-publisher|TIMED_WAITING| | > |java.lang.Object.wait(Native Method) > com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:136) > > com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90) > java.lang.Thread.run(Thread.java:748)| > |31|dispatcher-event-loop-0|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1406253491})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |32|dispatcher-event-loop-1|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@749604930})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |33|dispatcher-event-loop-2|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@373097727})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |34|dispatcher-event-loop-3|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@2082509879})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |35|dispatcher-event-loop-4|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@576894497})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |36|dispatcher-event-loop-5|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@2123533871})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |37|dispatcher-event-loop-6|RUNNABLE|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@16148478})| > |sun.management.ThreadImpl.dumpThreads0(Native Method) > sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) > org.apache.spark.util.Utils$.getThreadDump(Utils.scala:2136) > org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerSlaveEndpoint.scala:76) > > org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105) > org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) > org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |38|dispatcher-event-loop-7|WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@2022689531})| > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |62|driver-heartbeater|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |3|Finalizer|WAITING| | > |java.lang.Object.wait(Native Method) > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)| > |74|java-sdk-http-connection-reaper|TIMED_WAITING| | > |java.lang.Thread.sleep(Native Method) > com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181)| > |76|Kinesis Receiver 0|TIMED_WAITING| | > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.runProcessLoop(Worker.java:394) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:364) > > org.apache.spark.streaming.kinesis.KinesisReceiver$$anon$1.run(KinesisReceiver.scala:178)| > |77|LeaseCoordinator-1|WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |78|LeaseCoordinator-2|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |79|LeaseRenewer-1|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |109|LeaseRenewer-10|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |110|LeaseRenewer-11|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |111|LeaseRenewer-12|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |112|LeaseRenewer-13|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |113|LeaseRenewer-14|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |114|LeaseRenewer-15|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |115|LeaseRenewer-16|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |116|LeaseRenewer-17|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |117|LeaseRenewer-18|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |118|LeaseRenewer-19|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |80|LeaseRenewer-2|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |119|LeaseRenewer-20|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |81|LeaseRenewer-3|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |83|LeaseRenewer-4|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |90|LeaseRenewer-5|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |91|LeaseRenewer-6|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |106|LeaseRenewer-7|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |107|LeaseRenewer-8|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |108|LeaseRenewer-9|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |1|main|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475) > org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180) > org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:281) > > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:231) > > org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:67) > org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66) > java.security.AccessController.doPrivileged(Native Method) > javax.security.auth.Subject.doAs(Subject.java:422) > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) > > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188) > > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:284) > > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)| > |48|netty-rpc-env-timeout|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |82|RecordProcessor-0000|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1420638034})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |92|RecordProcessor-0001|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |93|RecordProcessor-0002|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |94|RecordProcessor-0003|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@232363476})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |95|RecordProcessor-0004|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@6750217})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |96|RecordProcessor-0005|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |97|RecordProcessor-0006|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@925698142})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |98|RecordProcessor-0007|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |99|RecordProcessor-0008|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1321496024})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |100|RecordProcessor-0009|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@551084388})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |101|RecordProcessor-0010|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@2145940185})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |102|RecordProcessor-0011|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1396545762})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |103|RecordProcessor-0012|TIMED_WAITING|Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1048217072})| > |java.lang.Thread.sleep(Native Method) > com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:142) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49) > > com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24) > java.util.concurrent.FutureTask.run(FutureTask.java:266) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |104|RecordProcessor-0013|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |105|RecordProcessor-0014|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748)| > |69|RecurringTimer - BlockGenerator|TIMED_WAITING| | > |java.lang.Thread.sleep(Native Method) > org.apache.spark.util.SystemClock.waitTillTime(Clock.scala:63) > org.apache.spark.streaming.util.RecurringTimer.triggerActionForNextInterval(RecurringTimer.scala:93) > > org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:106) > > org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:29)| > |71|RecurringTimer - BlockGenerator|TIMED_WAITING| | > |java.lang.Thread.sleep(Native Method) > org.apache.spark.util.SystemClock.waitTillTime(Clock.scala:63) > org.apache.spark.streaming.util.RecurringTimer.triggerActionForNextInterval(RecurringTimer.scala:93) > > org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:106) > > org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:29)| > |73|RecurringTimer - Kinesis Checkpointer - Worker > ip-10-225-172-133.ec2.internal:b0416d26-5a93-474a-962c-f83bd458cf45|TIMED_WAITING| > | > |java.lang.Thread.sleep(Native Method) > org.apache.spark.util.SystemClock.waitTillTime(Clock.scala:63) > org.apache.spark.streaming.util.RecurringTimer.triggerActionForNextInterval(RecurringTimer.scala:93) > > org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:106) > > org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:29)| > |2|Reference Handler|WAITING| | > |java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) > java.lang.ref.Reference.tryHandlePending(Reference.java:191) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)| > |39|rpc-client-3-1|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1293902103}), > Monitor(java.util.Collections$UnmodifiableSet@467054947}), > Monitor(sun.nio.ch.EPollSelectorImpl@1112349689})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@1112349689}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |50|shuffle-client-4-1|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1266242029}), > Monitor(java.util.Collections$UnmodifiableSet@1515042964}), > Monitor(sun.nio.ch.EPollSelectorImpl@1722122330})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@1722122330}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |51|shuffle-client-4-2|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1189671289}), > Monitor(java.util.Collections$UnmodifiableSet@1426119010}), > Monitor(sun.nio.ch.EPollSelectorImpl@686366461})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@686366461}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |52|shuffle-client-4-3|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@786936733}), > Monitor(java.util.Collections$UnmodifiableSet@651917814}), > Monitor(sun.nio.ch.EPollSelectorImpl@439429438})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@439429438}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |53|shuffle-client-4-4|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1558875100}), > Monitor(java.util.Collections$UnmodifiableSet@134646289}), > Monitor(sun.nio.ch.EPollSelectorImpl@1102296783})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@1102296783}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |58|shuffle-client-6-1|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@2046802270}), > Monitor(java.util.Collections$UnmodifiableSet@865227115}), > Monitor(sun.nio.ch.EPollSelectorImpl@1273360262})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@1273360262}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |59|shuffle-client-6-2|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@339187701}), > Monitor(java.util.Collections$UnmodifiableSet@1383232807}), > Monitor(sun.nio.ch.EPollSelectorImpl@701268413})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@701268413}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |60|shuffle-client-6-3|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@886987703}), > Monitor(java.util.Collections$UnmodifiableSet@224314223}), > Monitor(sun.nio.ch.EPollSelectorImpl@2087146587})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@2087146587}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |54|shuffle-server-5-1|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@208363137}), > Monitor(java.util.Collections$UnmodifiableSet@140124955}), > Monitor(sun.nio.ch.EPollSelectorImpl@702689906})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@702689906}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |55|shuffle-server-5-2|RUNNABLE|Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1867504979}), > Monitor(java.util.Collections$UnmodifiableSet@1198487408}), > Monitor(sun.nio.ch.EPollSelectorImpl@108390903})| > |sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) => holding > Monitor(sun.nio.ch.EPollSelectorImpl@108390903}) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:747) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| > |4|Signal Dispatcher|RUNNABLE| | > | | > |70|Thread-3|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:268) > > org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:110)| > |72|Thread-4|TIMED_WAITING| | > |sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:268) > > org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:110)| > |68|threadDeathWatcher-8-1|TIMED_WAITING| | > |java.lang.Thread.sleep(Native Method) > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org