peterxcli commented on PR #5407: URL: https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5449196674
Good catch @sunchao — confirmed, and addressed in e78da57aa. The planning gate checks required names, and a physical name like `K` (U+212A) folds to a requested ASCII `k` only in Spark's Unicode resolver, so the guard admits the scan and the ASCII matcher null-fills. Notably the existing regression fixture already contained that exact column; it stayed safe only because the test also requests `é`/`σ`. Since Spark-parity Unicode matching (including the duplicate-grouping semantics) is what #5495 is scoped to do across all the matcher's call sites, this PR now fails closed instead of resolving: in case-insensitive mode, when the required schema carries a marked Variant field and a requested name has no ASCII match, `remap_physical_schema` checks the file for a non-ASCII physical name whose Unicode lowercase equals the requested name's — the same fold Spark groups by — and errors with a pointer to #5495 rather than null-filling. Because the check mirrors Spark's grouping rule, it fires exactly where Spark would resolve and Comet would not; a name like `ſ`, which does not lowercase to `s`, stays on the ordinary missing-column path in both engines. The check is gated on Variant-bearing required schemas, so pre-existing non-Variant behavior is unchanged and remains with #5495. Covered by a Rust unit test (Kelvin errors, ASCII case matches, `ſ` agrees-missing, non-Variant unchanged) and a `CometNativeReaderSuite` regression that writes physical `K`, asserts the native scan is admitted, `SELECT k` fails with the folding error, and the `ſ`/`s` control agrees with Spark natively — passing on Spark 4.0 and 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
