[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005838#comment-17005838 ]
Marcelo Masiero Vanzin commented on SPARK-30225: ------------------------------------------------ The changes in SPARK-23366 may not necessarily have caused this. But that change actually flipped the configuration's default value from false to true; so 2.3 has the feature disabled by default, and 2.4 has it enabled by default. So the bug may have existed in the 2.3 version of the code, making it a bit harder to track. Still taking a look at the code but nothing popped up yet... > "Stream is corrupted at" exception on reading disk-spilled data of a shuffle > operation > -------------------------------------------------------------------------------------- > > Key: SPARK-30225 > URL: https://issues.apache.org/jira/browse/SPARK-30225 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 2.4.0 > Reporter: Mala Chikka Kempanna > Priority: Major > > There is issues with spark.unsafe.sorter.spill.read.ahead.enabled in spark > 2.4.0, which is introduced by > https://issues.apache.org/jira/browse/SPARK-23366 > > Workaround for this problem is to disable readahead of unsafe spill with > following. > --conf spark.unsafe.sorter.spill.read.ahead.enabled=false > > This issue can be reproduced on Spark 2.4.0 by following the steps in this > comment of Jira SPARK-18105. > https://issues.apache.org/jira/browse/SPARK-18105?focusedCommentId=16981461&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16981461 > > Exception looks like below: > {code:java} > 19/12/10 01:51:31 INFO sort.ShuffleExternalSorter: Thread 142 spilling sort > data of 5.1 GB to disk (1 time so far)19/12/10 01:51:31 INFO > sort.ShuffleExternalSorter: Thread 142 spilling sort data of 5.1 GB to disk > (1 time so far)19/12/10 01:52:48 INFO sort.ShuffleExternalSorter: Thread 142 > spilling sort data of 5.1 GB to disk (2 times so far)19/12/10 01:53:53 ERROR > executor.Executor: Exception in task 6.0 in stage 0.0 (TID > 6)java.io.IOException: Stream is corrupted at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:202) at > net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:228) at > net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157) at > org.apache.spark.io.ReadAheadInputStream$1.run(ReadAheadInputStream.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)19/12/10 01:53:53 INFO > executor.CoarseGrainedExecutorBackend: Got assigned task 3319/12/10 01:53:53 > INFO executor.Executor: Running task 8.1 in stage 0.0 (TID 33)19/12/10 > 01:54:00 INFO sort.UnsafeExternalSorter: Thread 142 spilling sort data of 3.3 > GB to disk (0 time so far)19/12/10 01:54:30 INFO executor.Executor: Executor > is trying to kill task 8.1 in stage 0.0 (TID 33), reason: Stage > cancelled19/12/10 01:54:30 INFO executor.Executor: Executor killed task 8.1 > in stage 0.0 (TID 33), reason: Stage cancelled19/12/10 01:54:52 INFO > executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown{code} > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org