[ https://issues.apache.org/jira/browse/SPARK-48717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-48717. ---------------------------------- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47093 [https://github.com/apache/spark/pull/47093] > Python foreachBatch streaming query cannot be stopped gracefully after pin > thread mode is enabled and is running spark queries > ------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-48717 > URL: https://issues.apache.org/jira/browse/SPARK-48717 > Project: Spark > Issue Type: New Feature > Components: PySpark, SS > Affects Versions: 4.0.0 > Reporter: Wei Liu > Assignee: Wei Liu > Priority: Major > Fix For: 4.0.0 > > > Followup of https://issues.apache.org/jira/browse/SPARK-39218 > > It only considered the InterruptedException is thrown when time.sleep(10) is > intercepted. But when a spark query is executing: > {code:java} > def func(batch_df, batch_id): > batch_df.sparkSession.range(10000000).write.saveAsTable("oops") > print(batch_df.count()) {code} > the actual error would be: > > {code:java} > py4j.protocol.Py4JJavaError: An error occurred while calling > o2141502.saveAsTable. > : org.apache.spark.SparkException: Job aborted. > <REDACTED> > <REDACTED> > <REDACTED> > ... > at > org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$2(StreamExecution.scala:262) > > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at > org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:262) > > *Caused by: java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1000)* > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308) > {code} > We should also add consideration to this scenario -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org