dwsmith1983 opened a new issue, #7134: URL: https://github.com/apache/datafusion-comet/issues/7134
### Describe the bug The native Iceberg scan can return the values of the wrong struct child for a name-mapped file whose physical nested names differ in case from the table's. The values are silently wrong; there is no error. iceberg-rust applies name mapping to top-level fields only (`apply_name_mapping_to_arrow_schema` in `crates/iceberg/src/arrow/reader/projection.rs` at the pinned 1f3bc34). With nested schema pruning, the projected target is `Struct<y>`, which doesn't match the file's `Struct<X, Y>`, so the `RecordBatchTransformer` promotes the column with arrow's `cast`. arrow-cast's `cast_struct_to_struct` (59.3) finds no child named exactly `y` and falls back to matching children by position, so `y` takes `X`'s data. This is separate from #6790, where nested name-mapped fields read as null whenever a top-level column is projected alongside them, but it has the same root cause: nested name mapping in iceberg-rust (iceberg-rust#1845) would give the nested children field ids and avoid the positional cast. Found while checking #7130. It behaves the same on main (39380cca8). ### Steps to reproduce 1. Write a Parquet file with schema `id INT, S STRUCT<X INT, Y INT>` and rows `(1, (10, 100))`, `(2, (20, 200))`, `(3, (30, 300))`. 2. Create an Iceberg table `id INT, s STRUCT<x INT, y INT>`, set `schema.name-mapping.default` to map `S`, `X` and `Y` to the table's field ids, and import the file with `SparkTableUtil.importSparkTable`. 3. With nested schema pruning on and the native Iceberg scan enabled, run `SELECT s.y FROM t ORDER BY s.y`. Spark returns `100, 200, 300`. Comet returns `10, 20, 30`. The result is the same with `spark.sql.caseSensitive` true or false. `SELECT s.x` happens to return the right values only because `X` is the first child. A file where only the child differs in case (`s STRUCT<X INT, y INT>`) reads correctly, since `y` then matches exactly. ### Expected behavior The values Spark returns, or a fallback to Spark for a name-mapped file with nested columns until the nested mapping is available. ### Additional context A conservative fallback could key on the table having `schema.name-mapping.default` and the scan projecting a nested column, since the JVM side can't cheaply tell which data files lack field ids. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
