LuciferYang opened a new pull request, #56668: URL: https://github.com/apache/spark/pull/56668
### What changes were proposed in this pull request? The Declarative Pipelines CLI `run()` (in `python/pyspark/pipelines/cli.py`) creates a Spark Connect session and previously stopped it only in a `finally` that wrapped `handle_pipeline_events`. Graph creation, element registration, and starting the run all happened before that `try`, so if any of them failed the session was never stopped and leaked. This moves those steps inside the `try` so the `finally: spark.stop()` runs once the session exists, regardless of where a later step fails. ### Why are the changes needed? A failure in `create_dataflow_graph`, `register_definitions`, or `start_run` left the Spark Connect session running. For a short-lived CLI invocation that means a leaked session (and its server-side resources) on every failed run. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added `test_run_stops_session_when_a_step_fails` in `python/pyspark/pipelines/tests/test_cli.py`. It mocks `create_dataflow_graph` to raise after the session is created and asserts the session is still stopped. The test fails without this change (`stop` is never called) and passes with it. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
