debugging a Spark error

[Chris Grier]

Hi,

I'm trying to figure out what the problem is with a job that we are running
on Spark 0.7.3. When we write out via saveAsTextFile we get an exception
that doesn't reveal much:

13/11/18 15:06:19 INFO cluster.TaskSetManager: Loss was due to
java.io.IOException
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
        at spark.storage.DiskStore.getBytes(DiskStore.scala:86)
        at spark.storage.DiskStore.getValues(DiskStore.scala:92)
        at spark.storage.BlockManager.getLocal(BlockManager.scala:284)
        at spark.storage.BlockFetcherIterator$$anonfun$
13.apply(BlockManager.scala:1027)
        at spark.storage.BlockFetcherIterator$$anonfun$
13.apply(BlockManager.scala:1026)
        at scala.collection.mutable.ResizableArray$class.foreach(
ResizableArray.scala:60)
        at scala.collection.mutable.ArrayBuffer.foreach(
ArrayBuffer.scala:47)
        at spark.storage.BlockFetcherIterator.<init>(
BlockManager.scala:1026)
        at spark.storage.BlockManager.getMultiple(BlockManager.scala:478)
        at spark.BlockStoreShuffleFetcher.fetch(BlockStoreShuffleFetcher.
scala:51)
        at spark.BlockStoreShuffleFetcher.fetch(BlockStoreShuffleFetcher.
scala:10)
        at spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(
CoGroupedRDD.scala:127)
        at spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(
CoGroupedRDD.scala:115)
        at scala.collection.IndexedSeqOptimized$class.
foreach(IndexedSeqOptimized.scala:34)
        at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
        at spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:115)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:704)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.FlatMappedValuesRDD.compute(PairRDDFunctions.scala:714)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.scheduler.ResultTask.run(ResultTask.scala:77)
        at spark.executor.Executor$TaskRunner.run(Executor.scala:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

Any ideas?

-Chris