[ https://issues.apache.org/jira/browse/SPARK-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-12189: ------------------------------------ Assignee: Apache Spark (was: Josh Rosen) > UnifiedMemoryManager double counts storage memory freed > ------------------------------------------------------- > > Key: SPARK-12189 > URL: https://issues.apache.org/jira/browse/SPARK-12189 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.0 > Reporter: Andrew Or > Assignee: Apache Spark > Priority: Blocker > > When execution evicts storage, we decrement the storage memory in two places: > (1) > https://github.com/apache/spark/blob/3e7e05f5ee763925ed60410d7de04cf36b723de1/core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala#L131 > (2) > https://github.com/apache/spark/blob/3e7e05f5ee763925ed60410d7de04cf36b723de1/core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala#L133 > (1) calls MemoryStore#ensureFreeSpace, which internally calls > MemoryManager#releaseStorageMemory for each block it drops. This call lowers > the storage memory used by the block size. > A seemingly simple fix is just to remove the line in (2). However, this bug > is actually masked by SPARK-12165, so this one must be fixed after that one. > Josh actually has an outstanding patch to fix both: > https://github.com/apache/spark/pull/10170 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org