[ https://issues.apache.org/jira/browse/SPARK-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218653#comment-14218653 ]
Apache Spark commented on SPARK-4495: ------------------------------------- User 'JoshRosen' has created a pull request for this issue: https://github.com/apache/spark/pull/3372 > Memory leak in JobProgressListener due to `spark.ui.retainedJobs` not being > used > -------------------------------------------------------------------------------- > > Key: SPARK-4495 > URL: https://issues.apache.org/jira/browse/SPARK-4495 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Josh Rosen > Assignee: Josh Rosen > Priority: Blocker > > In the JobProgressListener, we never actually remove jobs from the > jobIdToData map. In fact, there's a somewhat embarrassing typo: _retailed_ > jobs: > {code} > val retailedJobs = conf.getInt("spark.ui.retainedJobs", > DEFAULT_RETAINED_JOBS) > {code} > This was introduced by my patch for [SPARK-2321]: I should have added a test > that set {{spark.ui.retainedJobs}} to some low threshold and checked that we > only reported information for that many jobs; there was already a > corresponding "test LRU eviction of stages" test that I should have just > copied and modified. > This only affects the unreleased 1.2.0 version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org