[ https://issues.apache.org/jira/browse/SPARK-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250460#comment-14250460 ]
Josh Rosen commented on SPARK-4772: ----------------------------------- I've merged this into {{branch-1.2}}, so this will be included in 1.2.1. Since that was the last remaining backport, I'm marking this as Fixed. > Accumulators leak memory, both temporarily and permanently > ---------------------------------------------------------- > > Key: SPARK-4772 > URL: https://issues.apache.org/jira/browse/SPARK-4772 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.0.0 > Reporter: Nathan Kronenfeld > Assignee: Nathan Kronenfeld > Labels: accumulators > Fix For: 1.0.3, 1.3.0, 1.1.2, 1.2.1 > > Original Estimate: 24h > Remaining Estimate: 24h > > Accumulators.localAccums is cleared at the beginning of a task, and not at > the end. > This means that any locally accumulated values hang around until another task > is run on that thread. > If for some reason, the thread dies, said values hang around indefinitely. > This is really only a big issue with very large accumulators. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org