dwsmith1983 commented on PR #7130: URL: https://github.com/apache/datafusion-comet/pull/7130#issuecomment-6290051428
> Could you add a test for that shape, compared with Spark? If it now reads nulls or fails, should nested matching stay case-insensitive, or should that shape fall back to Spark? Added in 60b4745fd: a name-mapped file imported with `importSparkTable` whose nested children are physically `S<X, Y>` or `s<X, y>` against table `s<x, y>`, with an exact-name control, under both case settings and with nested pruning on and off. The reads give the same results with and without this PR. The batch Comet's adapter sees already has the table's nested names: when the column isn't passed through as is, iceberg-rust's transformer builds the batch with its target schema, so `match_struct_fields` compares `x` with `x` either way and the fold had nothing to resolve. The shape does read wrong values, on main as well, from two problems in iceberg-rust, which maps names for top-level fields only. Any read that also projects a top-level column gets nulls for `s`, which is #6790. A pruned `s.y` over physical `X`, `Y` returns `X`'s values, because arrow's struct cast falls back to matching by position; I filed that as #7134. The test leaves out exactly those reads and cites both issues. > Could the missing-column error also be exercised against the metadata columns (`_file`, `_pos`, `_partition`)? Also in 60b4745fd: `_file`, `_pos`, `_spec_id`, `_partition` and `_partition.Region` next to data columns on a table partitioned by `Region`, under both case settings. The check doesn't fire for any of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
