Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/126#discussion_r10553044
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala ---
@@ -17,28 +17,24 @@
package org.apache.spark.scheduler
+import scala.collection.mutable.HashMap
+
import java.io._
import java.util.zip.{GZIPInputStream, GZIPOutputStream}
-import scala.collection.mutable.HashMap
-
import org.apache.spark._
import org.apache.spark.executor.ShuffleWriteMetrics
-import org.apache.spark.rdd.RDD
-import org.apache.spark.rdd.RDDCheckpointData
+import org.apache.spark.rdd.{RDD, RDDCheckpointData}
import org.apache.spark.storage._
-import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType,
TimeStampedHashMap}
+import org.apache.spark.util.BoundedHashMap
private[spark] object ShuffleMapTask {
// A simple map between the stage id to the serialized byte array of a
task.
// Served as a cache for task serialization because serialization can be
// expensive on the master node if it needs to launch thousands of tasks.
- val serializedInfoCache = new TimeStampedHashMap[Int, Array[Byte]]
-
- // TODO: This object shouldn't have global variables
- val metadataCleaner = new MetadataCleaner(
- MetadataCleanerType.SHUFFLE_MAP_TASK,
serializedInfoCache.clearOldValues, new SparkConf)
+ val MAX_CACHE_SIZE = 100
+ val serializedInfoCache = new BoundedHashMap[Int,
Array[Byte]](MAX_CACHE_SIZE, true)
--- End diff --
Sorry I meant this, when a shuffle dependency goes out of scope, we can
find the associated stages and then we can remove the data for each stage that
is held in this cache.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
