Ryan Williams created SPARK-21425: ------------------------------------- Summary: LongAccumulator, DoubleAccumulator not threadsafe Key: SPARK-21425 URL: https://issues.apache.org/jira/browse/SPARK-21425 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.2.0 Reporter: Ryan Williams
[AccumulatorV2 docs|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L42-L43] acknowledge that accumulators must be concurrent-read-safe, but afaict they must also be concurrent-write-safe. The same docs imply that {{Int}} and {{Long}} meet either/both of these criteria, when afaict they do not. Relatedly, the provided [LongAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L291] and [DoubleAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L370] are not thread-safe, and should be expected to behave undefinedly when multiple concurrent tasks on the same executor write to them. [Here is a repro repo|https://github.com/ryan-williams/spark-bugs/tree/accum] with some simple applications that demonstrate incorrect results from {{LongAccumulator}}'s. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org