Liang Lee created SPARK-14695: --------------------------------- Summary: Error occurs when using OFF_HEAP persistent level Key: SPARK-14695 URL: https://issues.apache.org/jira/browse/SPARK-14695 Project: Spark Issue Type: Bug Components: Block Manager, Spark Core Affects Versions: 1.6.0 Environment: Spark 1.6.0 Tachyon 0.8.2 Hadoop 2.6.0 Reporter: Liang Lee
When running a PageRank job through the default examples, e.g., the class 'org.apache.spark.examples.graphx.Analytics' in spark-examples-1.6.0-hadoop2.6.0.jar package, we got the following erors: 16/04/18 02:30:01 WARN scheduler.TaskSetManager: Lost task 9.0 in stage 6.0 (TID 53, R1S1): java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -1 at scala.Predef$.require(Predef.scala:233) at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:822) at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala: 645) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:15 3) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala: 38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scal a:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scal a:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) at java.lang.Thread.run(Thread.java:745) We use the following script to submit the job: /Hadoop/spark-1.6.0-bin-hadoop2.6/bin/spark-submit --class org.apache.spark.examples.graphx.Analytics /Hadoop/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar pagerank /data/soc-LiveJournal1.txt --output=/output/live-off.res --numEPart=10 --numIter=1 --edgeStorageLevel=OFF_HEAP --vertexStorageLevel=OFF_HEAP When we set the storage level to MEMORY_ONLY or DISK_ONLY, there is no error and the job can finished correctly. But when we set the storage level to OFF_HEAP, which means using Tachyon for the storage process, the error occurs. The executors stack is like this, seems the write block to Tahcyon failed. 16/04/18 02:25:54 ERROR ExternalBlockStore: Error in putValues(rdd_20_1) java.io.IOException: Fail to cache: null at tachyon.client.file.FileOutStream.handleCacheWriteException(FileOutStream.java:276) at tachyon.client.file.FileOutStream.close(FileOutStream.java:165) at org.apache.spark.storage.TachyonBlockManager.putValues(TachyonBlockManager.scala:126) at org.apache.spark.storage.ExternalBlockStore.putIntoExternalBlockStore(ExternalBlockStore.scala:79) at org.apache.spark.storage.ExternalBlockStore.putIterator(ExternalBlockStore.scala:67) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:798) at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:153) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:820) at tachyon.client.block.LocalBlockOutStream.flush(LocalBlockOutStream.java:108) at tachyon.client.block.LocalBlockOutStream.close(LocalBlockOutStream.java:92) at tachyon.client.file.FileOutStream.close(FileOutStream.java:160) ... 31 more 16/04/18 02:25:54 ERROR Executor: Exception in task 1.0 in stage 10.0 (TID 142) java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -1 at scala.Predef$.require(Predef.scala:233) at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:822) at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:153) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/04/18 02:25:54 INFO CoarseGrainedExecutorBackend: Got assigned task 160 16/04/18 02:25:54 INFO Executor: Running task 1.1 in stage 10.0 (TID 160) 16/04/18 02:25:54 INFO CacheManager: Partition rdd_30_1 not found, computing it 16/04/18 02:25:54 ERROR ExternalBlockStore: Error in getValues(rdd_20_1) java.io.IOException: tachyon.exception.TachyonException: FileId 1660944383 BlockIndex 0 is not a valid block. at tachyon.client.TachyonFS.getClientBlockInfo(TachyonFS.java:447) at tachyon.client.TachyonFile.getClientBlockInfo(TachyonFile.java:126) at tachyon.client.TachyonFile.getLocationHosts(TachyonFile.java:212) at org.apache.spark.storage.TachyonBlockManager.getValues(TachyonBlockManager.scala:152) at org.apache.spark.storage.ExternalBlockStore$$anonfun$getValues$1.apply(ExternalBlockStore.scala:147) at org.apache.spark.storage.ExternalBlockStore$$anonfun$getValues$1.apply(ExternalBlockStore.scala:147) at scala.Option.flatMap(Option.scala:170) at org.apache.spark.storage.ExternalBlockStore.getValues(ExternalBlockStore.scala:147) at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:486) at org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:420) at org.apache.spark.storage.BlockManager.get(BlockManager.scala:625) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:44) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: tachyon.exception.TachyonException: FileId 1660944383 BlockIndex 0 is not a valid block. at tachyon.client.FileSystemMasterClient.getFileBlockInfo(FileSystemMasterClient.java:151) at tachyon.client.TachyonFS.getClientBlockInfo(TachyonFS.java:444) ... 35 more 16/04/18 02:25:54 INFO CacheManager: Partition rdd_20_1 not found, computing it 16/04/18 02:25:54 INFO BlockManager: Found block rdd_3_1 locally 16/04/18 02:25:54 WARN BlockManager: Block rdd_20_1 already exists on this machine; not re-adding it And when we check the Storeag page of job UI, we find some blocks are not cached: RDD Name Storage Level Cached Partitions Fraction Cached Size in Memory Size in ExternalBlockStore Size on Disk EdgeRDD ExternalBlockStore Serialized 1x Replicated 18 90% 0.0 B 753.2 MB 0.0 B VertexRDD ExternalBlockStore Serialized 1x Replicated 20 100% 0.0 B 191.1 MB 0.0 B VertexRDD, VertexRDD ExternalBlockStore Serialized 1x Replicated 20 100% 0.0 B 121.1 MB 0.0 B VertexRDD ExternalBlockStore Serialized 1x Replicated 20 100% 0.0 B 121.3 MB 0.0 B GraphLoader.edgeListFile - edges (/data/soc-LiveJournal1.txt) ExternalBlockStore Serialized 1x Replicated 20 100% 0.0 B 871.8 MB 0.0 B EdgeRDD ExternalBlockStore Serialized 1x Replicated 18 90% 0.0 B 1307.6 MB 0.0 B I have check the onfiguration many times. Can anyone give me some ideas about this issue. I has puzzled me for 2 weeks. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org