Visorgood commented on issue #6414:
URL: 
https://github.com/apache/datafusion-comet/issues/6414#issuecomment-6914479362

   Following up from a Discord thread with two updates and a proposal.
   
   **Spark 4.2 turns this on by default.** In 4.2.0, 
`spark.sql.shuffle.orderIndependentChecksum.enabled` and 
`spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch` both 
default to `true` (both are `false` in 4.1.3), and 4.2 adds 
`enableQueryLevelRollbackOnMismatch` (default `false`). So from 4.2 this 
affects every Comet shuffle, not only users who opt in.
   
   **The flags don't reach Spark either.** `CometShuffleDependency` passes only 
the first seven constructor arguments to `ShuffleDependency`, so for both 
native and JVM shuffle `rowBasedChecksums` is empty and 
`checksumMismatchFullRetryEnabled` is `false`. As a result, `DAGScheduler` 
keeps the legacy rollback for statically indeterminate stages on Comet 
shuffles, so today this is a missing protection rather than a broken one. It 
also means the two parts must change together: passing the full-retry flag 
while the writers still report 0 would turn off the legacy rollback and detect 
nothing.
   
   (This is separate from `spark.shuffle.checksum.enabled`, the order-dependent 
corruption checksum that Comet's JVM writers already support.)
   
   **Proposal for the JVM side:**
   1. A 4.1+ shim that builds `CometShuffleDependency` with `rowBasedChecksums` 
and the flags, as `ShuffleExchangeExec` does, only for shuffle paths that 
compute the checksum.
   2. Compute the checksum in `CometBypassMergeSortShuffleWriter` and 
`CometUnsafeShuffleWriter`, which already see each `UnsafeRow`, and pass it to 
`MapStatus` through `MapStatusHelper`.
   3. Tests: a Comet test that checks non-zero checksums that don't depend on 
row order, and, where it applies, re-enabling `MapStatusEndToEndSuite`'s 
"Propagate checksum from executor to driver", which `dev/diffs/4.1.3.diff` and 
`4.2.0.diff` mark `IgnoreComet` with a link to this issue.
   
   The native writer needs an equivalent order-independent row checksum in 
Rust, so I'd leave that part to someone who writes Rust; until then native 
shuffle keeps today's behavior.
   
   As noted on Discord, this could also help round-robin partitioning (cc 
@comphead).
   
   This is from reading the code (Comet `main` at 983173e, Spark v4.1.3 and 
v4.2.0); I haven't run it yet. I can take the JVM part. Would you prefer it 
under this issue or as a separate sub-issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to