ivoson opened a new pull request, #58518: URL: https://github.com/apache/spark/pull/58518
### What changes were proposed in this pull request? Guards the `sc.dagScheduler` dereferences in `SQLExecution.withNewExecutionId0`'s `finally` block so a query unwinding while the `SparkContext` is being torn down no longer throws a `NullPointerException` that destroys the query's real failure. `SparkContext.stop()` nulls `_dagScheduler` before it stops the listener bus, so a query that ends during teardown can reach `sc.dagScheduler.cleanupQueryJobs(executionId)` with a null receiver. Because that call runs in a `finally`, the NPE *replaces* the query's real exception rather than being suppressed, and it skips the rest of the block: the `SparkListenerSQLExecutionEnd` event is never posted, and `observationManager.tryComplete` is never called (leaving any `Observation.get` waiter blocked). The `activeQueryToJobs` dereference under `Utils.isTesting` sits earlier in the same `finally` and hits the same race, so it is guarded as well; without that a test cannot reach the `cleanupQueryJobs` call site. ### Why are the changes needed? A query failing (or completing) exactly as its `SparkContext` is being stopped surfaces as an unclassified internal `NullPointerException` from `SQLExecution`, hiding the real error and leaving observation waiters hung, instead of propagating the true failure. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? UTs added. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
