Hyukjin Kwon created SPARK-61540:
------------------------------------

             Summary: Let the removal of the last streaming query listener wait 
for the delivery in flight before it removes the server's listener, as PySpark 
does
                 Key: SPARK-61540
                 URL: https://issues.apache.org/jira/browse/SPARK-61540
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


PySpark's StreamingQueryListenerBus.remove acquires the lock of the bus, which 
post_to_all holds while it calls the listeners, before it sends the REMOVE of 
the last listener; a signal interrupts that acquire. So, while a listener 
callback runs, PySpark's removeListener (and streams.close()) of the last 
listener waits for the callback, and a Ctrl-C raises KeyboardInterrupt at once 
with no REMOVE sent and the listener still registered and receiving events.

The Python client sent the REMOVE at once while the callback ran, and then 
waited for the worker in the join (abortable since SPARK-61481's follow-up). So 
after a Ctrl-C, the server's listener was removed, and the listener stopped 
getting events.

The removal of the last listener from another thread than the worker now waits 
for the delivery of the worker before it removes the server's listener, as the 
removal of another listener already did. The bus is Stopping meanwhile, as for 
a removal in flight. So a callback of that delivery that removes a listener or 
closes the bus coalesces with the removal, which owns the result, and one that 
adds a listener is refused, as before; one that stops the session leaves 
nothing to remove. A signal ends the wait in the Python client before anything 
is removed or sent.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to