[ https://issues.apache.org/jira/browse/SPARK-33121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350629#comment-17350629 ]
L. C. Hsieh commented on SPARK-33121: ------------------------------------- Hmm, I cannot reproduce in branch-3.1/2.4 or master branch. If you don't have Thread.sleep(5000), does it stop gracefully? > Spark Streaming 3.1.1 hangs on shutdown > --------------------------------------- > > Key: SPARK-33121 > URL: https://issues.apache.org/jira/browse/SPARK-33121 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 3.1.1 > Reporter: Dmitry Tverdokhleb > Priority: Major > Labels: Streaming, hang, shutdown > > Hi. I am trying to migrate from spark 2.4.5 to 3.1.1 and there is a problem > in graceful shutdown. > Config parameter "spark.streaming.stopGracefullyOnShutdown" is set as "true". > Here is the code: > {code:java} > inputStream.foreachRDD { > rdd => > rdd.foreachPartition { > Thread.sleep(5000) > } > } > {code} > I send a SIGTERM signal to stop the spark streaming and after sleeping an > exception arises: > {noformat} > streaming-agg-tds-data_1 | java.util.concurrent.RejectedExecutionException: > Task org.apache.spark.executor.Executor$TaskRunner@7ca7f0b8 rejected from > java.util.concurrent.ThreadPoolExecutor@2474219c[Terminated, pool size = 0, > active threads = 0, queued tasks = 0, completed tasks = 1] > streaming-agg-tds-data_1 | at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) > streaming-agg-tds-data_1 | at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) > streaming-agg-tds-data_1 | at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) > streaming-agg-tds-data_1 | at > org.apache.spark.executor.Executor.launchTask(Executor.scala:270) > streaming-agg-tds-data_1 | at > org.apache.spark.scheduler.local.LocalEndpoint.$anonfun$reviveOffers$1(LocalSchedulerBackend.scala:93) > streaming-agg-tds-data_1 | at > org.apache.spark.scheduler.local.LocalEndpoint.$anonfun$reviveOffers$1$adapted(LocalSchedulerBackend.scala:91) > streaming-agg-tds-data_1 | at > scala.collection.Iterator.foreach(Iterator.scala:941) > streaming-agg-tds-data_1 | at > scala.collection.Iterator.foreach$(Iterator.scala:941) > streaming-agg-tds-data_1 | at > scala.collection.AbstractIterator.foreach(Iterator.scala:1429) > streaming-agg-tds-data_1 | at > scala.collection.IterableLike.foreach(IterableLike.scala:74) > streaming-agg-tds-data_1 | at > scala.collection.IterableLike.foreach$(IterableLike.scala:73) > streaming-agg-tds-data_1 | at > scala.collection.AbstractIterable.foreach(Iterable.scala:56) > streaming-agg-tds-data_1 | at > org.apache.spark.scheduler.local.LocalEndpoint.reviveOffers(LocalSchedulerBackend.scala:91) > streaming-agg-tds-data_1 | at > org.apache.spark.scheduler.local.LocalEndpoint$$anonfun$receive$1.applyOrElse(LocalSchedulerBackend.scala:68) > streaming-agg-tds-data_1 | at > org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115) > streaming-agg-tds-data_1 | at > org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213) > streaming-agg-tds-data_1 | at > org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100) > streaming-agg-tds-data_1 | at > org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75) > streaming-agg-tds-data_1 | at > org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41) > streaming-agg-tds-data_1 | at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > streaming-agg-tds-data_1 | at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > streaming-agg-tds-data_1 | at java.lang.Thread.run(Thread.java:748) > streaming-agg-tds-data_1 | 2021-04-22 13:33:41 WARN JobGenerator - Timed > out while stopping the job generator (timeout = 10000) > streaming-agg-tds-data_1 | 2021-04-22 13:33:41 INFO JobGenerator - Waited > for jobs to be processed and checkpoints to be written > streaming-agg-tds-data_1 | 2021-04-22 13:33:41 INFO JobGenerator - Stopped > JobGenerator{noformat} > After this exception and "JobGenerator - Stopped JobGenerator" log, streaming > freezes, and halts by timeout (Config parameter > "hadoop.service.shutdown.timeout"). > Besides, there is no problem with the graceful shutdown in spark 2.4.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org