Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/22371 @squito , thanks for the review. I intend to using `ConcurrentHashMap[Int, AtomicReferenceArray]` previously. After re-think the code, I can know the lock here is used to prevent the same task with different attempt to commit the shuffle writer result at the same time. The task has a different attempt can be caused by follows: 1. Failed task or stage. In this case, the previous task attempt should already finish(failed or killed) or the result is not used anymore. 2. `Speculative task`. In this case, the speculative task can't be scheduled to the same executor as other attempts. So, what's real value for the lock. Maybe I'm wrong, hopeful some answers.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org