Hyukjin Kwon created SPARK-61388:
------------------------------------
Summary: Interrupt only the execution that an interrupted call
waits for, in the Python client as PySpark does
Key: SPARK-61388
URL: https://issues.apache.org/jira/browse/SPARK-61388
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61259 and SPARK-61353, found in a post-merge review. The
in-flight registry that a SIGINT interrupts is keyed by thread only, and its
innermost entry wins, so SIGINT during next() of one toLocalIterator interrupts
another half-consumed iterator of the thread instead. With such an iterator or
a streaming listener alive on the thread, SIGINT during SparkSession.stop()
also interrupts that execution and then waits for the release. The watch also
latches the first signal it sees when no execution is in flight (for example
during the plan compression Config RPC), so an execution that the same call
starts afterwards runs to its end before KeyboardInterrupt is raised.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]