ericm-db opened a new pull request, #57686: URL: https://github.com/apache/spark/pull/57686
### What changes were proposed in this pull request? This is layer 3 of the local Connect pool stack: #57684 -> #57685 -> this PR. Until the lower layers merge, GitHub shows their cumulative diff; the new review unit here is commit `76def5665b6`. This layer adds process lifecycle management on top of the storage model: - janitor rules for dead or timed-out launches; - retirement of dead, unreachable, idle, and orphaned claimed servers; - graceful termination followed by SIGKILL escalation; - cleanup of stale member directories and half-started daemons; - idempotent release of a claimed member; and - a forceful pool purge escape hatch. It does not add background acquisition or SparkSession dispatch. ### Why are the changes needed? Pool members outlive individual Python call frames and can be orphaned by crashes or SIGKILL. Their cleanup rules need to be explicit, idempotent, and testable before attendants and acquisition loops begin invoking them concurrently. ### Does this PR introduce _any_ user-facing change? No. The pool is not wired into SparkSession in this layer. ### How was this patch tested? Expanded `LocalConnectServerPoolUnitTests` with fake-process coverage for dead and live pending launches, unreachable and idle members, orphaned claims, shutdown escalation, release, and purge. ```bash python -m unittest -v pyspark.sql.tests.connect.test_connect_local_server_pool.LocalConnectServerPoolUnitTests ``` All 14 tests passed. Ruff check, Ruff format check, and `git diff --check` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
