[
https://issues.apache.org/jira/browse/SPARK-14695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245083#comment-15245083
]
Liang Lee commented on SPARK-14695:
-----------------------------------
The cluster enviroment is like this:
Totally 3 servers. One acts as NameNode, SparkMaster and TachyonMaster; the
other two act as DataNode, Spark Worker and Tachyon Worker.
We set the Tachyon Worker Memory to 64GB per node and Total 128GB, only memory
level is enabled in Tachyon.
We submit the job on Master node.
The most strange question is:
In the same worker, most executors can finish the task correctly but only 1 or
2 executors failed to cache the block and cause the above errors.
> Error occurs when using OFF_HEAP persistent level
> --------------------------------------------------
>
> Key: SPARK-14695
> URL: https://issues.apache.org/jira/browse/SPARK-14695
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
> Affects Versions: 1.6.0
> Environment: Spark 1.6.0
> Tachyon 0.8.2
> Hadoop 2.6.0
> Reporter: Liang Lee
>
> When running a PageRank job through the default examples, e.g., the class
> 'org.apache.spark.examples.graphx.Analytics' in
> spark-examples-1.6.0-hadoop2.6.0.jar package, we got the following erors:
> 16/04/18 02:30:01 WARN scheduler.TaskSetManager: Lost task 9.0 in stage 6.0
> (TID
> 53, R1S1): java.lang.IllegalArgumentException: requirement failed:
> sizeInBytes
> was negative: -1
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:822)
> at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:
> 645)
> at
> org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:15
> 3)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:
> 38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scal
> a:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scal
> a:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> .java:615)
> at java.lang.Thread.run(Thread.java:745)
> We use the following script to submit the job:
> /Hadoop/spark-1.6.0-bin-hadoop2.6/bin/spark-submit --class
> org.apache.spark.examples.graphx.Analytics
> /Hadoop/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar
> pagerank /data/soc-LiveJournal1.txt --output=/output/live-off.res
> --numEPart=10 --numIter=1 --edgeStorageLevel=OFF_HEAP
> --vertexStorageLevel=OFF_HEAP
> When we set the storage level to MEMORY_ONLY or DISK_ONLY, there is no error
> and the job can finished correctly.
> But when we set the storage level to OFF_HEAP, which means using Tachyon for
> the storage process, the error occurs.
> The executors stack is like this, seems the write block to Tahcyon failed.
> 16/04/18 02:25:54 ERROR ExternalBlockStore: Error in putValues(rdd_20_1)
> java.io.IOException: Fail to cache: null
> at
> tachyon.client.file.FileOutStream.handleCacheWriteException(FileOutStream.java:276)
> at tachyon.client.file.FileOutStream.close(FileOutStream.java:165)
> at
> org.apache.spark.storage.TachyonBlockManager.putValues(TachyonBlockManager.scala:126)
> at
> org.apache.spark.storage.ExternalBlockStore.putIntoExternalBlockStore(ExternalBlockStore.scala:79)
> at
> org.apache.spark.storage.ExternalBlockStore.putIterator(ExternalBlockStore.scala:67)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:798)
> at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645)
> at
> org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:153)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:820)
> at
> tachyon.client.block.LocalBlockOutStream.flush(LocalBlockOutStream.java:108)
> at
> tachyon.client.block.LocalBlockOutStream.close(LocalBlockOutStream.java:92)
> at tachyon.client.file.FileOutStream.close(FileOutStream.java:160)
> ... 31 more
> 16/04/18 02:25:54 ERROR Executor: Exception in task 1.0 in stage 10.0 (TID
> 142)
> java.lang.IllegalArgumentException: requirement failed: sizeInBytes was
> negative: -1
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:822)
> at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645)
> at
> org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:153)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 16/04/18 02:25:54 INFO CoarseGrainedExecutorBackend: Got assigned task 160
> 16/04/18 02:25:54 INFO Executor: Running task 1.1 in stage 10.0 (TID 160)
> 16/04/18 02:25:54 INFO CacheManager: Partition rdd_30_1 not found, computing
> it
> 16/04/18 02:25:54 ERROR ExternalBlockStore: Error in getValues(rdd_20_1)
> java.io.IOException: tachyon.exception.TachyonException: FileId 1660944383
> BlockIndex 0 is not a valid block.
> at tachyon.client.TachyonFS.getClientBlockInfo(TachyonFS.java:447)
> at tachyon.client.TachyonFile.getClientBlockInfo(TachyonFile.java:126)
> at tachyon.client.TachyonFile.getLocationHosts(TachyonFile.java:212)
> at
> org.apache.spark.storage.TachyonBlockManager.getValues(TachyonBlockManager.scala:152)
> at
> org.apache.spark.storage.ExternalBlockStore$$anonfun$getValues$1.apply(ExternalBlockStore.scala:147)
> at
> org.apache.spark.storage.ExternalBlockStore$$anonfun$getValues$1.apply(ExternalBlockStore.scala:147)
> at scala.Option.flatMap(Option.scala:170)
> at
> org.apache.spark.storage.ExternalBlockStore.getValues(ExternalBlockStore.scala:147)
> at
> org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:486)
> at
> org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:420)
> at org.apache.spark.storage.BlockManager.get(BlockManager.scala:625)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:44)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
> at
> org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: tachyon.exception.TachyonException: FileId 1660944383 BlockIndex 0
> is not a valid block.
> at
> tachyon.client.FileSystemMasterClient.getFileBlockInfo(FileSystemMasterClient.java:151)
> at tachyon.client.TachyonFS.getClientBlockInfo(TachyonFS.java:444)
> ... 35 more
> 16/04/18 02:25:54 INFO CacheManager: Partition rdd_20_1 not found, computing
> it
> 16/04/18 02:25:54 INFO BlockManager: Found block rdd_3_1 locally
> 16/04/18 02:25:54 WARN BlockManager: Block rdd_20_1 already exists on this
> machine; not re-adding it
> And when we check the Storeag page of job UI, we find some blocks are not
> cached:
> RDD Name Storage Level Cached Partitions Fraction Cached Size in
> Memory Size in ExternalBlockStore Size on Disk
> EdgeRDD ExternalBlockStore Serialized 1x Replicated 18 90%
> 0.0 B 753.2 MB 0.0 B
> VertexRDD ExternalBlockStore Serialized 1x Replicated 20 100%
> 0.0 B 191.1 MB 0.0 B
> VertexRDD, VertexRDD ExternalBlockStore Serialized 1x Replicated 20
> 100% 0.0 B 121.1 MB 0.0 B
> VertexRDD ExternalBlockStore Serialized 1x Replicated 20 100%
> 0.0 B 121.3 MB 0.0 B
> GraphLoader.edgeListFile - edges (/data/soc-LiveJournal1.txt)
> ExternalBlockStore Serialized 1x Replicated 20 100% 0.0 B 871.8
> MB 0.0 B
> EdgeRDD ExternalBlockStore Serialized 1x Replicated 18 90%
> 0.0 B 1307.6 MB 0.0 B
> I have check the onfiguration many times. Can anyone give me some ideas about
> this issue. I has puzzled me for 2 weeks.
> Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
