liuzqt opened a new pull request, #58960: URL: https://github.com/apache/spark/pull/58960
### What changes were proposed in this pull request? Make the backing map a lazily created `var` reached only through `getOrCreate`, which is the fix SPARK-20977 applied to `CollectionAccumulator` for the same hazard. ### Why are the changes needed? Pretty much the same issue as SPARK-20977. the racing condition: - `PartitionKeyedAccumulator[T] extends AccumulatorV2` deserialized in executor - `AccumulatorV2.readObject` registers the accumulator with the `TaskContext`--> as this point, this accumulator is considered "published", i.e., accessible through other threads, however, the class is **not fully initialized**: only `AccumulatorV2` fields are properly init, but subclass's fields are not - Java deserialization then proceeds to read `PartitionKeyedAccumulator`'s own fields, **now it's fully initialized** So during the window between step 2 and step3, executor heartbeat polling comes in and access the not-initialized field, it would run into NPE ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Added UT case. ### Was this patch authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Generated by Claude with Opus5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
