[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535044#comment-17535044 ]
Attila Zsolt Piros commented on SPARK-39152: -------------------------------------------- I am working on this > StreamCorruptedException cause job failure for disk persisted RDD > ----------------------------------------------------------------- > > Key: SPARK-39152 > URL: https://issues.apache.org/jira/browse/SPARK-39152 > Project: Spark > Issue Type: Improvement > Components: Block Manager > Affects Versions: 3.4.0 > Reporter: Attila Zsolt Piros > Assignee: Attila Zsolt Piros > Priority: Major > > In case of a disk corruption a disk persisted RDD block will lead to job > failure as the block registration is always leads to the same file. So even > when the task is rescheduled on a different executor the job will fail. > *Example* > First failure (the block is locally available): > {noformat} > 22/04/25 07:15:28 ERROR executor.Executor: Exception in task 17024.0 in stage > 12.0 (TID 51853) > java.io.StreamCorruptedException: invalid stream header: 00000000 > at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:401) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:122) > at > org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209) > at > org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:617) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:897) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:286) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) > {noformat} > Then the task might be rescheduled on a different executor but as the block > is registered to the first blockmanager the error will be the same: > {noformat} > java.io.StreamCorruptedException: invalid stream header: 00000000 > at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:943) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:401) > at > org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63) > at > org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63) > at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:122) > at > org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209) > at > org.apache.spark.storage.BlockManager$$anonfun$getRemoteValues$1.apply(BlockManager.scala:698) > at > org.apache.spark.storage.BlockManager$$anonfun$getRemoteValues$1.apply(BlockManager.scala:696) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:696) > at org.apache.spark.storage.BlockManager.get(BlockManager.scala:831) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886) > at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) > {noformat} > My idea is to retry the IO operations a few times and when all of them failed > deregistering the block and let the following task to recompute it. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org