L. C. Hsieh created SPARK-61441:
-----------------------------------

             Summary: Make the state lock of the streaming listener bus 
poison-tolerant and drop listeners outside it in Spark Connect Rust
                 Key: SPARK-61441
                 URL: https://issues.apache.org/jira/browse/SPARK-61441
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


The client-side streaming listener bus of spark-connect-rust 
(crates/spark-connect/src/streaming.rs) keeps its state behind one mutex that 
is taken with lock().unwrap() in about 55 places, three of them in Drop impls 
(StartedDispatch, Dispatch, FinishGeneration).

If any thread panics while it holds the lock, the mutex is poisoned: every 
later call on the bus of the session (addListener, removeListener, listeners, 
the delivery of the started event) panics, and a guard that is dropped by an 
unwinding panic panics again, which aborts the process.

Listener callbacks, RPCs and thread joins already run without the lock. Two 
things still run under it:

* The Drop of a listener, when the bus removes the last reference to it (the 
removal of a listener that is not the last one, the end of the generation of a 
worker, the end of a removal of the last listener, and the teardown of the 
session). It is code of the user: a panic in it poisons the lock, and a Drop 
that calls back into the bus deadlocks on the lock that its own thread holds.
* The spawn of the worker thread, which panics when the thread cannot be 
created, and then drops the event stream, whose progress handlers are of the 
user, under the lock.

Proposed change:

* Take the lock only through one accessor that recovers a poisoned guard, so 
that no Drop of the bus can panic.
* Drop the listeners that leave the bus, and the snapshot of a delivery, after 
the lock is released, isolating a panic of their Drop as that of a callback.
* Hand the event stream to the worker after it is spawned; a thread that cannot 
be spawned is an error of add_listener instead of a panic.
* Document the locking rule of the bus.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to