[
https://issues.apache.org/jira/browse/SPARK-61532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61532:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Let a signal during a callback end the removal of the last listener before
> its REMOVE, keep the listener of an add that a signal ends before the server
> answers, and warn of a failed interrupt of the listener bus, as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61532
> URL: https://issues.apache.org/jira/browse/SPARK-61532
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> From a review of SPARK-61470 (Ctrl-C ends the waits of the Python client) and
> its follow-ups. The listener bus of the Python client still differed from
> PySpark's in three cases of a signal:
> 1. removeListener of the last listener (and streams.close()) while a listener
> callback runs on the worker. PySpark's remove acquires the lock of the bus,
> which post_to_all holds while it calls the listeners, and a signal interrupts
> that acquire. So PySpark raises KeyboardInterrupt at once, sends no REMOVE,
> and keeps the listener. Ours sent the REMOVE at once, and then waited for the
> callback in the join of the worker. The last-listener removal now waits for
> the delivery of the worker, as the removal of another listener already does,
> through the bus's wait, which a signal ends.
> 2. A signal during addListener before the server's response headers came: an
> unreachable or slow server, or the plan compression lookup. That signal
> dropped the listener from the bus, while PySpark's append, which puts the
> listener in the bus first, keeps it. The listener now stays, as for a signal
> during the wait for the acknowledgement, and the server's listener, which the
> ADD may have created, is removed by the next removal or subscription.
> 3. A failed Interrupt (the server fails the interrupt of the wait) during
> addListener, removeListener or close was raised as the interrupt's error.
> PySpark's append and remove catch it with `except Exception`, warn, and
> return; append also removes the listener from its bus. These calls now warn
> and return, and addListener drops the registration that the interrupted add
> kept.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]