zifeif2 opened a new pull request, #58897: URL: https://github.com/apache/spark/pull/58897
### What changes were proposed in this pull request? This PR fixes a deadlock when removing the last Python Spark Connect streaming query listener. It adds a listener lifecycle lock to serialize `append` and last-listener `remove` operations, releases the listener-bus data lock before waiting for the event thread to terminate, and reacquires the data lock for final cleanup. It also adds a deterministic regression test covering a pending event and a concurrent listener append during shutdown. ### Why are the changes needed? Previously, `StreamingQueryListenerBus.remove` called `self._execution_thread.join()` while holding `self._lock`. If the event thread had already received an event and was entering `post_to_all`, it needed the same lock to dispatch that event before it could terminate. The remover waited for the event thread, while the event thread waited for the remover's lock, causing a deadlock. The lifecycle lock preserves atomic listener startup/shutdown transitions without preventing the event thread from acquiring the data lock and draining pending events. ### Does this PR introduce _any_ user-facing change? Yes. Removing the final Python Spark Connect streaming query listener no longer hangs when an event is pending. There is no API change. ### How was this patch tested? Added `StreamingQueryListenerBusTests.test_remove_last_listener_with_pending_event`, which deterministically verifies that: - a pending event can be dispatched while last-listener removal waits for the event thread; - a concurrent listener append waits until shutdown completes; - the new listener starts a fresh event thread after shutdown. The targeted unit test passed. Ruff lint, Ruff formatting validation, and `git diff --check` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
