[
https://issues.apache.org/jira/browse/SPARK-61532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61532.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 564
[https://github.com/apache/spark-connect-rust/pull/564]
> Keep the listener of an add that a signal ends before the server answers, and
> warn of a failed interrupt of the listener bus, as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61532
> URL: https://issues.apache.org/jira/browse/SPARK-61532
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> From a review of SPARK-61470 (Ctrl-C ends the waits of the Python client) and
> its follow-ups. The listener bus of the Python client still differed from
> PySpark's in two cases of a signal:
> 1. A signal during addListener before the server's response headers came: an
> unreachable or slow server, or the plan compression lookup. That signal
> dropped the listener from the bus, while PySpark's append, which puts the
> listener in the bus first, keeps it. The listener now stays, as for a signal
> during the wait for the acknowledgement, and the server's listener, which the
> ADD may have created, is removed by the next removal or subscription.
> 2. A failed Interrupt (the server fails the interrupt of the wait) during
> addListener, removeListener or close was raised as the interrupt's error.
> PySpark's append and remove catch it with `except Exception`, warn, and
> return; append also removes the listener from its bus. These calls now warn
> and return, and addListener drops the registration that the interrupted add
> kept.
> Not covered: PySpark's remove blocks on the lock that post_to_all holds while
> a callback runs, so a Ctrl-C then sends no REMOVE. The drop-in sends the
> REMOVE while a callback runs, so that a callback can make lifecycle calls
> during an external close, which
> test_streaming_listener_callback_lifecycle_calls_during_external_close pins.
> In PySpark those calls deadlock on its non-reentrant lock. Waiting for the
> delivery first needs a separate design decision.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]