ErikBPF opened a new issue, #5884: URL: https://github.com/apache/datafusion-comet/issues/5884
## Description Implement Spark-compatible duplicate Parquet sibling-name resolution in the native reader, replacing the explicit rejection proposed in PR #5786 for issue #5783. As reported in [the PR review](https://github.com/apache/datafusion-comet/pull/5786#pullrequestreview-5187242277), Spark's behavior depends on where the duplicate occurs: - Root group: with `message spark_schema { optional int64 a; optional int64 a; optional int64 b; }` and one raw row `a=1, a=2, b=3`, an explicit `a bigint` read schema returns `[1]` (first child wins). - Nested structs: issue #5783 demonstrates last-wins resolution through `caseSensitiveParquetFieldMap`. The native Arrow decoder can combine distinct leaves into one column before schema adaptation, multiplying rows or failing with a decoder synchronization error. Resolution must select the correct physical leaf before decoding, rather than dropping duplicate logical fields afterward. ## Acceptance criteria Match Spark results, row counts, and error behavior for root, nested struct, array-element, and map-value duplicates using explicit read schemas. Cover case sensitivity and field-ID reads; preserve unrelated-column projection, filter pruning, and metadata-cache behavior. Replace the scan compatibility limitation once supported. Related: #5783, #5786. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
