peterxcli opened a new issue, #5980: URL: https://github.com/apache/datafusion-comet/issues/5980
### What is the problem the feature request solves? For scalar and array shredding states with both `typed_value` and residual `value` present, Spark uses `typed_value`. Arrow rejects that combination. #5868 therefore [masks the ignored residual before Arrow validation](https://github.com/apache/datafusion-comet/blob/9a4cd83b59807522f4310778be78f9190a5dde59/native/core/src/parquet/cast_column/variant.rs#L244-L256), including residual bytes that Spark would never inspect. The [Parquet specification](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) permits both values only for partially shredded objects. This is Spark compatibility for noncanonical input, rather than a request to weaken canonical validation globally. ### Describe the potential solution Move the precedence rule into a shared native reconstruction or compatibility entry point so Comet can remove the preparatory residual-masking pass. Completion requires: - Prefer a present scalar/array `typed_value` over the redundant residual, without decoding or validating the ignored bytes. - Preserve object merging: a partially shredded object's residual still contains fields that must be read and validated. - Preserve residual-only values, nested states, list offsets/slices, parent SQL NULL, and Variant null. - Compare the output and failure behavior with supported Spark readers and assert native scan use. - Delete the masking branch only after the replacement provides equivalent behavior. ### Additional context Parent: #5477; implementation and regression coverage: #5868. This issue is separate from missing metadata keys and missing required-value validation. It does not authorize dropping residual fields from partially shredded objects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
