sunchao commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5654386714
Reviewed head `1ea1be161` against base `db790673d`. **I found one P2 correctness regression to fix before merging.** ### P2: Duplicate root names now select the wrong column The new [root-field shadowing](https://github.com/apache/datafusion-comet/blob/1ea1be161ec664b8705b109be5004593a5d7bb4f/native/core/src/parquet/schema_adapter.rs#L348) hides earlier duplicate names and selects the last column. I tested one Parquet file containing two separate `d` columns, reading it with explicit schema `d bigint` and case sensitivity enabled: | Reader | Result | |---|---| | Spark 4.1.3, vectorized | `[1, 2, 3]` | | Base native scan harness | `[1, 2, 3]` | | PR native scan harness | `[10, 20, 30]` | This changes previously matching results. Spark’s schema-clipping `.toMap` behavior does not establish what its reader actually returns. Preserve the working root-column selection and add a comparison against Spark using the same file. This supplies concrete differential evidence for the concern already discussed in [#5884](https://github.com/apache/datafusion-comet/issues/5884). ### Validation and remaining assessment - **CI:** 56 successful, 10 skipped; `Required Checks` passed. - **Local:** 22 component tests passed. Additional scan probes verified the earlier default-value and duplicate-ID fixes. - **Build limit:** The registry lacks pinned DataFusion 55.1. Native probes used current changed modules with DataFusion 55.0 / Arrow 59.3, rather than a full current-head Comet build. Relevant upstream root-read paths were checked against 55.1. - No other actionable findings emerged from mapping, projection, ownership, footer-cache, or JNI-error review. Evidence: Spark result, base result, PR result (local reproduction logs). Nothing was published to GitHub. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
