[ 
https://issues.apache.org/jira/browse/SPARK-61481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61481:
-----------------------------------
    Labels: pull-request-available  (was: )

> Let a second stop() release the session after an interrupted one, and let a 
> signal end the waits of the streaming listener bus, in the Python client as 
> PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61481
>                 URL: https://issues.apache.org/jira/browse/SPARK-61481
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Two differences of the Python client's stop() and streaming listener bus from 
> PySpark 4.3.0, which the review of SPARK-61353 and SPARK-61315 found:
> 1. Concurrent stop(): the core SparkSession::stop marked the session stopped 
> before the release, so a second thread's stop() returned at once. If a signal 
> then interrupted the first stop()'s release (which keeps the session open, as 
> PySpark's does), the second caller had been told the session stopped while it 
> stayed open. PySpark runs stop() under SparkSession._lock (an RLock), so a 
> second stop() waits, and after the first one's KeyboardInterrupt it releases 
> and closes the session itself; is_stopped is client.is_closed.
> 2. The removal of the last streaming listener (and close()) waited on a 
> Condvar for another thread's started-event delivery, which no signal could 
> interrupt. PySpark's remove waits on the bus's threading.Lock, whose acquire 
> a signal interrupts.
> Fix: serialize stop() per session with a re-entrant lock whose wait a signal 
> ends in the Python client, and mark the session stopped after its release; 
> wait for the listener bus through the same abortable wait, leaving the bus as 
> it was when the wait is interrupted.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to