peterxcli commented on PR #5407: URL: https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5450797456
Both confirmed @sunchao — addressed in 53c58be40 by restructuring rather than patching. The guard moved out of `remap_physical_schema` into the per-referenced-column walk in `SparkPhysicalExprAdapter::rewrite`, alongside the existing case-insensitive duplicate detection: - **Referenced-only:** projection and predicate expressions both flow through `rewrite`, so the guard now sees exactly the columns the scan evaluates. Your `SELECT v` over `(v VARIANT, k INT)` with physical `K` succeeds natively (covered by a new `CometNativeReaderSuite` regression that also asserts `SELECT v, k` still fails closed). - **Per-field ID exemption:** instead of disabling on `should_match_by_id` globally, a referenced column is exempt only when field-ID matching is enabled and that logical field itself carries a `PARQUET:field_id` — the same eligibility rule the name matcher applies. An ID-less `k VARIANT` beside an ID-matched sibling now fails closed; the same field carrying an ID resolves by ID and stays exempt. Both directions are covered by adapter-level Rust tests driving `create()`/`rewrite()` with mixed-ID schemas. The fold detection itself is unchanged (`check_column_unicode_fold_mismatch`, mirroring Spark's `toLowerCase(Locale.ROOT)` grouping). Validated on Spark 4.0 and 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
