pingzh opened a new pull request, #5650: URL: https://github.com/apache/datafusion-comet/pull/5650
## Which issue does this PR close? Closes #5536. Follow-up to #5352, discovered during post-merge review of #5531. ## Rationale for this change The remote shuffle validator rejects dictionary encodings that `ShuffleBlockWriter` can emit and the local reader can consume, including numeric dictionaries, alternate key widths, and dictionaries inside containers. The current Spark pipeline normally strips these encodings before writing, but a future change could expose this mismatch and cause repeated fetch failures. Relaxing validation alone is insufficient because the JVM Arrow importer supports fewer dictionary layouts. ## What changes are included in this PR? - Compare dictionary value types recursively while retaining strict checks for logical types, struct field identity, and container shape. - Share remote batch decoding between native shuffle scans and JNI, validating buffers and logical types before normalizing dictionary encodings. - Preserve null values, nested nullability reconciliation, and zero-column batch row counts. - Add writer-to-reader regression coverage for supported dictionary layouts. ## How are these changes tested? - All 99 shuffle crate tests pass. - All 7 native shuffle-scan tests pass. - Writer round trips cover all eight integer dictionary key types, numeric/string/binary values, nested containers, both writer modes, and all four compression codecs. - Negative tests verify that logical-type and container-shape mismatches remain rejected. - Clippy passes for both affected crates' libraries and tests with warnings denied. - `cargo fmt --all -- --check` and `git diff --check` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
