shujingyang-db opened a new pull request, #59801: URL: https://github.com/apache/spark/pull/59801
### What changes were proposed in this pull request? This PR adds `AccumulatorContext.getOrNull`. It does the same lookup as `AccumulatorContext.get`, but returns `null` instead of `None` when the accumulator is not registered or has been garbage collected. `get` now delegates to `getOrNull` and wraps the result in `Option`. `DAGScheduler.updateAccumulators` now calls `getOrNull` and null-checks the result. ### Why are the changes needed? `DAGScheduler.updateAccumulators` runs on the driver for every finished task, and it looks up every accumulator update the task carries. A task can carry hundreds of updates when SQL metrics are involved. `AccumulatorContext.get` returns an `Option`, so each lookup allocates a `Some` that is dropped right away. `getOrNull` removes that allocation from the per-task path. ### Does this PR introduce _any_ user-facing change? No. `get` returns the same results as before, including `None` for a garbage-collected accumulator. The garbage-collected warning is still logged inside the lookup. A missing accumulator in `updateAccumulators` still throws `accessNonExistentAccumulatorError`. `AccumulatorContext` is `private[spark]`, so no public API changes. ### How was this patch tested? No new test, because the changed methods return, log, and throw exactly as before. Existing suites cover the changed paths and pass locally: ``` build/sbt "core/testOnly org.apache.spark.AccumulatorSuite org.apache.spark.util.AccumulatorV2Suite org.apache.spark.scheduler.DAGSchedulerSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
