ericm-db opened a new pull request, #58367:
URL: https://github.com/apache/spark/pull/58367
### What changes were proposed in this pull request?
This is layer 6 of the ten-PR local Connect pool stack:
#57684 -> #57685 -> #57907 -> #57686 -> #58247 -> this PR ->
#57687 -> #58248 -> #57102 -> #57688
This layer adds recovery before a pool member has finished publishing:
- validated pending-launch records;
- cleanup of dead, timed-out, malformed, and partially published launches;
- attendant command and UID checks before acting on a reused PID;
- complete cleanup of an attendant's launch process group;
- recovery of server PIDs from malformed records and daemon PID files; and
- stale conf-file and unreferenced member-directory garbage collection.
Acquisition, forceful purge, SparkSession integration, and warmup remain in
later PRs.
### Why are the changes needed?
An attendant or its spawning client can die at any point between writing the
launch seed, starting
the JVM, and publishing the ready server. Those partial states must stop
counting toward refills,
but cleanup must not signal an unrelated process after PID reuse.
This layer builds on #58247's server-retirement machinery while keeping the
separate attendant and
process-group ownership contract independently reviewable.
### Does this PR introduce _any_ user-facing change?
No. The pool is not wired into SparkSession in this layer.
### How was this patch tested?
All 59 focused pool tests at this stack layer passed:
```bash
PYTHONPATH=python:python/lib/pyspark.zip:python/lib/py4j-0.10.9.9-src.zip \
SPARK_TESTING=1 \
.venv/bin/python -m unittest -v \
pyspark.sql.tests.connect.test_connect_local_server_pool
```
The changed files also passed Python AST parsing, Ruff checking and
formatting,
`git diff --check`, and changed-file ASCII and 100-column checks.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Fable 5) and OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
