LuciferYang opened a new pull request, #13553: URL: https://github.com/apache/gluten/pull/13553
## What changes are proposed in this pull request? `VeloxHashShuffleWriter::computeRowBasedChecksums` dropped the first column before hashing rows for hash and range partitioning, assuming it was the partition id. Every caller of `doSplit` already removes the partition id column, so the first data column was left out of the checksum. It now hashes the rows as they are. This affects Spark 4.1 and later with `spark.sql.shuffle.orderIndependentChecksum.enabled` or `spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch`. Fixes #13547. ## How was this patch tested? Added `Gluten row-based checksum covers the first data column` to `GlutenRowBasedChecksumSuite` for Spark 4.1 and 4.2. It writes two tables that differ only in the first column, partitioned by the second column, and checks that the map output checksums differ. Before this change they were identical. ## Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Opus. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
