dwsmith1983 commented on code in PR #7130:
URL: https://github.com/apache/datafusion-comet/pull/7130#discussion_r4426785240
##########
native/core/src/execution/operators/iceberg_scan.rs:
##########
@@ -576,21 +576,36 @@ impl fmt::Debug for RedactedProperties<'_> {
}
}
+/// Exact name matching: iceberg-rust batches and Spark's attributes carry the
table's own names.
+fn iceberg_parquet_options() -> SparkParquetOptions {
+ let mut options = SparkParquetOptions::new(EvalMode::Legacy, "UTC");
+ options.case_sensitive = true;
Review Comment:
#7028 is on main now. I merged it here in 90fa7d026, and ab20cdee5 switches
"nested struct field renamed by case only" and "nested field renamed under A
beside a" to the native scan check.
The same commit adds "struct under A with no fields in common beside a".
#7028's check for structs that share no fields with the file only sees `A` if
`A` resolves to its own field id. With the old case-insensitive lookup it took
`a`'s id, the check skipped `A`, and the native read returned a struct of NULLs
where Spark returns NULL. With the exact lookup the scan falls back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
