sunchao commented on code in PR #5407:
URL: https://github.com/apache/datafusion-comet/pull/5407#discussion_r3879267550
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -101,6 +102,43 @@ fn remap_physical_schema(
)));
}
+ // Spark resolves case-insensitive names by Unicode lowercasing
(`toLowerCase(Locale.ROOT)`
+ // in `ParquetReadSupport`), while the matcher below folds ASCII only. A
physical name such
+ // as `K` (U+212A KELVIN SIGN) resolves to a requested ASCII `k` in Spark
but stays
+ // unmatched here, and the scan-planning gate cannot see physical file
names, so the column
+ // would be silently null-filled as missing. Until #5495 brings
Spark-parity Unicode
+ // matching, fail Variant-bearing scans whose resolution would differ
under Unicode folding
+ // rather than return values that disagree with Spark.
+ if !case_sensitive
+ && !should_match_by_id
+ && logical_schema.fields().iter().any(|f| is_variant_field(f))
+ {
+ for logical_field in logical_schema.fields() {
+ let name = logical_field.name();
Review Comment:
[P2] Restrict the Unicode guard to fields required by the scan
This loop sees the full logical file schema, including ordinary columns that
were pruned from the scan. For example, read a valid Spark-written Parquet file
containing `v VARIANT` and `K INT` with explicit schema `(v VARIANT, k INT)`,
then run `SELECT v` with case-insensitive resolution,
`spark.sql.variant.allowReadingShredded=true`, and
`spark.sql.variant.pushVariantIntoScan=false` (no field IDs or filters). Spark
requests only `v`, but `CometNativeScan` keeps the unused ordinary `k` in
`nativeDataSchema`, and DataFusion passes that full schema to this adapter
before applying projection. The loop therefore raises the Unicode-matching
execution error for `k`, aborting a query that never reads it; this is not a
planning fallback. The previous head did not reject this unused field. Please
limit the check to fields actually required by the scan, including predicate
inputs. The normal route is established from source, including the Spark 4.0.4
comparison; this configuration was not
executed locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
