Visorgood commented on issue #6414: URL: https://github.com/apache/datafusion-comet/issues/6414#issuecomment-6914479362
Following up from a Discord thread with two updates and a proposal. **Spark 4.2 turns this on by default.** In 4.2.0, `spark.sql.shuffle.orderIndependentChecksum.enabled` and `spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch` both default to `true` (both are `false` in 4.1.3), and 4.2 adds `enableQueryLevelRollbackOnMismatch` (default `false`). So from 4.2 this affects every Comet shuffle, not only users who opt in. **The flags don't reach Spark either.** `CometShuffleDependency` passes only the first seven constructor arguments to `ShuffleDependency`, so for both native and JVM shuffle `rowBasedChecksums` is empty and `checksumMismatchFullRetryEnabled` is `false`. As a result, `DAGScheduler` keeps the legacy rollback for statically indeterminate stages on Comet shuffles, so today this is a missing protection rather than a broken one. It also means the two parts must change together: passing the full-retry flag while the writers still report 0 would turn off the legacy rollback and detect nothing. (This is separate from `spark.shuffle.checksum.enabled`, the order-dependent corruption checksum that Comet's JVM writers already support.) **Proposal for the JVM side:** 1. A 4.1+ shim that builds `CometShuffleDependency` with `rowBasedChecksums` and the flags, as `ShuffleExchangeExec` does, only for shuffle paths that compute the checksum. 2. Compute the checksum in `CometBypassMergeSortShuffleWriter` and `CometUnsafeShuffleWriter`, which already see each `UnsafeRow`, and pass it to `MapStatus` through `MapStatusHelper`. 3. Tests: a Comet test that checks non-zero checksums that don't depend on row order, and, where it applies, re-enabling `MapStatusEndToEndSuite`'s "Propagate checksum from executor to driver", which `dev/diffs/4.1.3.diff` and `4.2.0.diff` mark `IgnoreComet` with a link to this issue. The native writer needs an equivalent order-independent row checksum in Rust, so I'd leave that part to someone who writes Rust; until then native shuffle keeps today's behavior. As noted on Discord, this could also help round-robin partitioning (cc @comphead). This is from reading the code (Comet `main` at 983173e, Spark v4.1.3 and v4.2.0); I haven't run it yet. I can take the JVM part. Would you prefer it under this issue or as a separate sub-issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
