Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1165#discussion_r14723210
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
* to the BlockManager as an iterator and expect to read it back
later. This is because
* we may end up dropping a partition from memory store before
getting it back, e.g.
* when the entirety of the RDD does not fit in memory. */
- val elements = new ArrayBuffer[Any]
- elements ++= values
- updatedBlocks ++= blockManager.put(key, elements, storageLevel,
tellMaster = true)
- elements.iterator.asInstanceOf[Iterator[T]]
+
--- End diff --
The logic here actually hasn't changed (the code was just moved elsewhere).
For the `MEMORY_ONLY` case, if we can't be sure that there is enough memory to
unroll the block, we simply drop it on the spot. I don't believe this is a
violation of the StorageLevel contract insofar as old blocks cached as
`MEMORY_ONLY` can also be dropped from the cache. The only difference here is
that we drop it earlier (before it has the potential to cause an OOM).
Among the options you listed, the last thing we want is (3), which
unfortunately is what happens in the existing code. It's true that if we our
estimate is too high, we may drop a block that can otherwise be cached.
However, the whole point of this is to guarantee memory safety (i.e. to avoid
OOMs), and to achieve that we need to err on the safe side.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
