[
https://issues.apache.org/jira/browse/SPARK-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-12817:
-------------------------------
Description:
CacheManager directly calls MemoryStore.unrollSafely() and has its own logic
for handling graceful fallback to disk when cached data does not fit in memory.
However, this logic also exists inside of the MemoryStore itself, so this
appears to be unnecessary duplication.
Thanks to the addition of block-level read/write locks, we can refactor the
code to remove the CacheManager and replace it with an atomic getOrElseUpdate
BlockManager method.
was:
CacheManager directly calls MemoryStore.unrollSafely() and has its own logic
for handling graceful fallback to disk when cached data does not fit in memory.
However, this logic also exists inside of the MemoryStore itself, so this
appears to be unnecessary duplication.
We can remove this duplication and delete a significant amount of BlockManager
code which existed only to support this CacheManager code.
> Remove CacheManager and replace it with new BlockManager.getOrElseUpdate
> method
> -------------------------------------------------------------------------------
>
> Key: SPARK-12817
> URL: https://issues.apache.org/jira/browse/SPARK-12817
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> CacheManager directly calls MemoryStore.unrollSafely() and has its own logic
> for handling graceful fallback to disk when cached data does not fit in
> memory. However, this logic also exists inside of the MemoryStore itself, so
> this appears to be unnecessary duplication.
> Thanks to the addition of block-level read/write locks, we can refactor the
> code to remove the CacheManager and replace it with an atomic getOrElseUpdate
> BlockManager method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
