[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391654#comment-17391654 ]
wuyi commented on SPARK-35275: ------------------------------ [~mridulm80] Yes, we shall have the doc task. Let me create it. > Add checksum for shuffle blocks > ------------------------------- > > Key: SPARK-35275 > URL: https://issues.apache.org/jira/browse/SPARK-35275 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 3.2.0 > Reporter: wuyi > Priority: Major > > Shuffle data corruption is a long-standing issue in Spark. For example, in > SPARK-18105, people continually reports corruption issue. However, data > corruption is difficult to reproduce in most cases and even harder to tell > the root cause. We don't know if it's a Spark issue or not. With the checksum > support for the shuffle, Spark itself can at least distinguish the cause > between disk and network, which is very important for users. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org