Hyukjin Kwon created SPARK-61532:
------------------------------------
Summary: Let a signal during a callback end the removal of the
last listener before its REMOVE, keep the listener of an add that a signal ends
before the server answers, and warn of a failed interrupt of the listener bus,
as PySpark does
Key: SPARK-61532
URL: https://issues.apache.org/jira/browse/SPARK-61532
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
>From a review of SPARK-61470 (Ctrl-C ends the waits of the Python client) and
>its follow-ups. The listener bus of the Python client still differed from
>PySpark's in three cases of a signal:
1. removeListener of the last listener (and streams.close()) while a listener
callback runs on the worker. PySpark's remove acquires the lock of the bus,
which post_to_all holds while it calls the listeners, and a signal interrupts
that acquire. So PySpark raises KeyboardInterrupt at once, sends no REMOVE, and
keeps the listener. Ours sent the REMOVE at once, and then waited for the
callback in the join of the worker. The last-listener removal now waits for the
delivery of the worker, as the removal of another listener already does,
through the bus's wait, which a signal ends.
2. A signal during addListener before the server's response headers came: an
unreachable or slow server, or the plan compression lookup. That signal dropped
the listener from the bus, while PySpark's append, which puts the listener in
the bus first, keeps it. The listener now stays, as for a signal during the
wait for the acknowledgement, and the server's listener, which the ADD may have
created, is removed by the next removal or subscription.
3. A failed Interrupt (the server fails the interrupt of the wait) during
addListener, removeListener or close was raised as the interrupt's error.
PySpark's append and remove catch it with `except Exception`, warn, and return;
append also removes the listener from its bus. These calls now warn and return,
and addListener drops the registration that the interrupted add kept.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]