Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r14738375 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -142,10 +151,76 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging { * to the BlockManager as an iterator and expect to read it back later. This is because * we may end up dropping a partition from memory store before getting it back, e.g. * when the entirety of the RDD does not fit in memory. */ - val elements = new ArrayBuffer[Any] - elements ++= values - updatedBlocks ++= blockManager.put(key, elements, storageLevel, tellMaster = true) - elements.iterator.asInstanceOf[Iterator[T]] + --- End diff -- To add, this specific issue I am referring to does not take away from rest of the patch. My concern is : if iterator cant be dropped to DISK/etc, then value in trying to determine if it can be cached in memory or not is not very high in my opinion. If it fails, it fails - a recomputation might succeed; and if no - reporting to user the failure and getting him to change to MEMORY_OR_DISK is better than suboptimal computational overheads.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---