[ 
https://issues.apache.org/jira/browse/SPARK-61532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61532:
------------------------------------

    Assignee: Hyukjin Kwon

> Keep the listener of an add that a signal ends before the server answers, and 
> warn of a failed interrupt of the listener bus, as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61532
>                 URL: https://issues.apache.org/jira/browse/SPARK-61532
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> From a review of SPARK-61470 (Ctrl-C ends the waits of the Python client) and 
> its follow-ups. The listener bus of the Python client still differed from 
> PySpark's in two cases of a signal:
> 1. A signal during addListener before the server's response headers came: an 
> unreachable or slow server, or the plan compression lookup. That signal 
> dropped the listener from the bus, while PySpark's append, which puts the 
> listener in the bus first, keeps it. The listener now stays, as for a signal 
> during the wait for the acknowledgement, and the server's listener, which the 
> ADD may have created, is removed by the next removal or subscription.
> 2. A failed Interrupt (the server fails the interrupt of the wait) during 
> addListener, removeListener or close was raised as the interrupt's error. 
> PySpark's append and remove catch it with `except Exception`, warn, and 
> return; append also removes the listener from its bus. These calls now warn 
> and return, and addListener drops the registration that the interrupted add 
> kept.
> Not covered: PySpark's remove blocks on the lock that post_to_all holds while 
> a callback runs, so a Ctrl-C then sends no REMOVE. The drop-in sends the 
> REMOVE while a callback runs, so that a callback can make lifecycle calls 
> during an external close, which 
> test_streaming_listener_callback_lifecycle_calls_during_external_close pins. 
> In PySpark those calls deadlock on its non-reentrant lock. Waiting for the 
> delivery first needs a separate design decision.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to