dwsmith1983 commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5666177302
> Preserve the working root-column selection and add a comparison against Spark using the same file. The comparison is in: `test-data/duplicate-root-names.parquet` carries two root `d` columns, and a new `ParquetReadSuite` test reads it with `d bigint` under case sensitivity and asserts Comet's answer and plan against Spark. It passes on 3.5 and 4.0, both returning `[1, 2, 3]`, because the JVM-side projection binds the first `d` before the native adapter sees the file, so the adapter's last-wins rule only decides between columns that are both projected. Your harness drove the adapter alone, which is where `[10, 20, 30]` comes from. The end-to-end selection is unchanged from base and now pinned. > Could the new one be named so the difference is visible at the call site, or could the old one gain a line saying it is deliberately root only? Both: the recursive predicate is now `any_nested_field_has_id`, and `schema_has_field_ids` says it is root only on purpose and names the other. > Could you file a tracking issue and link it here? Will do once we settle the nested-duplicate behaviour with @ErikBPF on #5786, since the same issue should record both the field-id gating gap and the nested resolution we want; I would rather not open two overlapping ones. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
