dwsmith1983 commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5528110202
Fixed in 4a57686cc. The per file policies are still computed against the physical struct so the footer's INT96 ordinal attribution stays valid, but the wrapper now derives a requested mask by pairing the physical type with the requested logical type through exactly the pairings the struct conversion narrows (struct children by name under the adapter's case rules, list elements, map entries with matching key ordering); every physical leaf outside that mask is marked corrected and never checked. Anything the conversion does not narrow (large lists, fixed size lists, dictionaries, mismatched map ordering) keeps every leaf, so the mask is a superset of what the reader materializes. Your probe is the Spark regression: a converted metadata-free file with s.d modern and s.ts ancient under EXCEPTION read modes, select s.d reads natively and matches Spark, while selecting s.ts still fails loudly. Rust unit tests cover the same at the wrapper level plus a list of struct variant and the non narrowed shapes. Batteries at this head: 236/236 on both spark-3.5 and spark-4.0, 343 native tests, clippy and fmt clean. Independent review of the change caught that the first cut recursed through list and dictionary pairings the conversion never narrows and a first-match tie-break mismatch on folded name collisions; both fixed before this push. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
