L. C. Hsieh created SPARK-61441:
-----------------------------------
Summary: Make the state lock of the streaming listener bus
poison-tolerant and drop listeners outside it in Spark Connect Rust
Key: SPARK-61441
URL: https://issues.apache.org/jira/browse/SPARK-61441
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
The client-side streaming listener bus of spark-connect-rust
(crates/spark-connect/src/streaming.rs) keeps its state behind one mutex that
is taken with lock().unwrap() in about 55 places, three of them in Drop impls
(StartedDispatch, Dispatch, FinishGeneration).
If any thread panics while it holds the lock, the mutex is poisoned: every
later call on the bus of the session (addListener, removeListener, listeners,
the delivery of the started event) panics, and a guard that is dropped by an
unwinding panic panics again, which aborts the process.
Listener callbacks, RPCs and thread joins already run without the lock. Two
things still run under it:
* The Drop of a listener, when the bus removes the last reference to it (the
removal of a listener that is not the last one, the end of the generation of a
worker, the end of a removal of the last listener, and the teardown of the
session). It is code of the user: a panic in it poisons the lock, and a Drop
that calls back into the bus deadlocks on the lock that its own thread holds.
* The spawn of the worker thread, which panics when the thread cannot be
created, and then drops the event stream, whose progress handlers are of the
user, under the lock.
Proposed change:
* Take the lock only through one accessor that recovers a poisoned guard, so
that no Drop of the bus can panic.
* Drop the listeners that leave the bus, and the snapshot of a delivery, after
the lock is released, isolating a panic of their Drop as that of a callback.
* Hand the event stream to the worker after it is spawned; a thread that cannot
be spawned is an error of add_listener instead of a panic.
* Document the locking rule of the bus.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]