[
https://issues.apache.org/jira/browse/SPARK-61481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61481:
------------------------------------
Assignee: Hyukjin Kwon
> Let a second stop() release the session after an interrupted one, and let a
> signal end the waits of the streaming listener bus, in the Python client as
> PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61481
> URL: https://issues.apache.org/jira/browse/SPARK-61481
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Two differences of the Python client's stop() and streaming listener bus from
> PySpark 4.3.0, which the review of SPARK-61353 and SPARK-61315 found:
> 1. Concurrent stop(): the core SparkSession::stop marked the session stopped
> before the release, so a second thread's stop() returned at once. If a signal
> then interrupted the first stop()'s release (which keeps the session open, as
> PySpark's does), the second caller had been told the session stopped while it
> stayed open. PySpark runs stop() under SparkSession._lock (an RLock), so a
> second stop() waits, and after the first one's KeyboardInterrupt it releases
> and closes the session itself; is_stopped is client.is_closed.
> 2. The removal of the last streaming listener (and close()) waited on a
> Condvar for another thread's started-event delivery, which no signal could
> interrupt. PySpark's remove waits on the bus's threading.Lock, whose acquire
> a signal interrupts.
> Fix: serialize stop() per session with a re-entrant lock whose wait a signal
> ends in the Python client, and mark the session stopped after its release;
> wait for the listener bus through the same abortable wait, leaving the bus as
> it was when the wait is interrupted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]