ErikBPF commented on PR #5786: URL: https://github.com/apache/datafusion-comet/pull/5786#issuecomment-5647662940
Thanks for the detailed review and reproductions. Addressed the requests in b696cc319 and rebased onto main, preserving the updated reader metrics and Variant handling. - Restrict duplicate validation to required top-level columns, recursively checking each selected subtree. Field-ID reads retain conservative whole-schema validation; empty projections skip all roots. Metadata-cache hits remain validated. - Added raw root-duplicate coverage plus unrelated-column, case-insensitive, repeated-read, pruning, count-only, and renamed field-ID cases. Added Rust LIST/MAP/separate-group coverage and removed the redundant batch-size dimension. - Updated the compatibility documentation and opened #5884 for Spark-compatible duplicate resolution. Independent Spark 4.1.3 vectorized testing found reader-dependent nested behavior; the follow-up includes that reproduction rather than assuming universal last-wins semantics. Validation: reproduced both valid-projection failures before the fix. Afterward, the full Spark 4.1 native-reader suite passed 70 tests (one existing NullType cancellation), and all 8 focused cases passed on Spark 3.5. Rust Parquet tests: 188 passed, one existing ignored benchmark. Native build, whole-reactor packaging, all-target workspace Clippy with warnings denied, semantic/syntactic Scalafix, Spotless, formatting, and whitespace checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
