[ https://issues.apache.org/jira/browse/SPARK-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu resolved SPARK-26042. ---------------------------------- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > KafkaContinuousSourceTopicDeletionSuite may hang forever > -------------------------------------------------------- > > Key: SPARK-26042 > URL: https://issues.apache.org/jira/browse/SPARK-26042 > Project: Spark > Issue Type: Test > Components: Structured Streaming, Tests > Affects Versions: 2.4.0 > Reporter: Shixiong Zhu > Assignee: Shixiong Zhu > Priority: Major > Fix For: 2.4.1, 3.0.0 > > > Saw the following thread dump in some build: > {code} > "stream execution thread for [id = 1c13482e-1edf-4b5c-b63a-d652738c8a48, > runId = 10667ce9-7eac-4cef-a525-f1bd08eb50f1]" #4406 daemon prio=5 os_prio=0 > tid=0x00007fab1d3c5000 nid=0x7f4b waiting on condition [0x00007fa96efcb000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000000070a904cf8> (a > scala.concurrent.impl.Promise$CompletionLatch) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > ... > at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:180) > at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:131) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:109) > - locked <0x000000070a913ee8> (a > org.apache.spark.sql.execution.streaming.IncrementalExecution) > at > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:109) > at > org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution$$anonfun$runContinuous$3$$anonfun$apply$1.apply(ContinuousExecution.scala:270) > at > org.apache.spark.sql.execution.streaming.continuous.ContinuousExecution$$anonfun$runContinuous$3$$anonfun$apply$1.apply(ContinuousExecution.scala:270) > ,,, > "pool-1-thread-1-ScalaTest-running-KafkaContinuousSourceTopicDeletionSuite" > #20 prio=5 os_prio=0 tid=0x00007fabc4e78800 nid=0x23be waiting for monitor > entry [0x00007fab3dbff000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:100) > - waiting to lock <0x000000070a913ee8> (a > org.apache.spark.sql.execution.streaming.IncrementalExecution) > at > org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:100) > at > org.apache.spark.sql.kafka010.KafkaContinuousSourceTopicDeletionSuite$$anonfun$3$$anonfun$apply$mcV$sp$12$$anonfun$apply$15.apply(KafkaContinuousSourceSuite.scala:210) > at > org.apache.spark.sql.kafka010.KafkaContinuousSourceTopicDeletionSuite$$anonfun$3$$anonfun$apply$mcV$sp$12$$anonfun$apply$15.apply(KafkaContinuousSourceSuite.scala:209) > ... > {code} > It hung forever because the test main thread was trying to access > `executedPlan` but the lock was held by the streaming thread. > This is a pretty common issue when using lazy vals as all lazy vals share the > same lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org