GitHub user artemrd opened a pull request:
https://github.com/apache/spark/pull/21114
[SPARK-22371][CORE] Return None instead of throwing an exception when an
accumulator is garbage collected.
## What changes were proposed in this pull request?
There's a period of time when an accumulator has been garbage collected,
but hasn't been removed from AccumulatorContext.originals by ContextCleaner.
When an update is received for such accumulator it will throw an exception and
kill the whole job. This can happen when a stage completes, but there're still
running tasks from other attempts, speculation etc. Since
AccumulatorContext.get() returns an option we can just return None in such case.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/artemrd/spark SPARK-22371
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21114.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21114
----
commit 65949e4d7438f13edd44633dbb66f296620ce0df
Author: Artem Rudoy <artem.rudoy@...>
Date: 2018-04-20T17:40:54Z
[SPARK-22371][CORE] Return None instead of throwing an exception when an
accumulator is garbage collected.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
