[
https://issues.apache.org/jira/browse/SPARK-61540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61540:
-----------------------------------
Labels: pull-request-available (was: )
> Let the removal of the last streaming query listener wait for the delivery in
> flight before it removes the server's listener, as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61540
> URL: https://issues.apache.org/jira/browse/SPARK-61540
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark's StreamingQueryListenerBus.remove acquires the lock of the bus,
> which post_to_all holds while it calls the listeners, before it sends the
> REMOVE of the last listener; a signal interrupts that acquire. So, while a
> listener callback runs, PySpark's removeListener (and streams.close()) of the
> last listener waits for the callback, and a Ctrl-C raises KeyboardInterrupt
> at once with no REMOVE sent and the listener still registered and receiving
> events.
> The Python client sent the REMOVE at once while the callback ran, and then
> waited for the worker in the join (abortable since SPARK-61481's follow-up).
> So after a Ctrl-C, the server's listener was removed, and the listener
> stopped getting events.
> The removal of the last listener from another thread than the worker now
> waits for the delivery of the worker before it removes the server's listener,
> as the removal of another listener already did. The bus is Stopping
> meanwhile, as for a removal in flight. So a callback of that delivery that
> removes a listener or closes the bus coalesces with the removal, which owns
> the result, and one that adds a listener is refused, as before; one that
> stops the session leaves nothing to remove. A signal ends the wait in the
> Python client before anything is removed or sent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]