Hyukjin Kwon created SPARK-61540:
------------------------------------
Summary: Let the removal of the last streaming query listener wait
for the delivery in flight before it removes the server's listener, as PySpark
does
Key: SPARK-61540
URL: https://issues.apache.org/jira/browse/SPARK-61540
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's StreamingQueryListenerBus.remove acquires the lock of the bus, which
post_to_all holds while it calls the listeners, before it sends the REMOVE of
the last listener; a signal interrupts that acquire. So, while a listener
callback runs, PySpark's removeListener (and streams.close()) of the last
listener waits for the callback, and a Ctrl-C raises KeyboardInterrupt at once
with no REMOVE sent and the listener still registered and receiving events.
The Python client sent the REMOVE at once while the callback ran, and then
waited for the worker in the join (abortable since SPARK-61481's follow-up). So
after a Ctrl-C, the server's listener was removed, and the listener stopped
getting events.
The removal of the last listener from another thread than the worker now waits
for the delivery of the worker before it removes the server's listener, as the
removal of another listener already did. The bus is Stopping meanwhile, as for
a removal in flight. So a callback of that delivery that removes a listener or
closes the bus coalesces with the removal, which owns the result, and one that
adds a listener is refused, as before; one that stops the session leaves
nothing to remove. A signal ends the wait in the Python client before anything
is removed or sent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]