dwsmith1983 opened a new issue, #6136: URL: https://github.com/apache/datafusion-comet/issues/6136
### Describe the bug DataFusion's Parquet opener builds the physical expression adapter only when a predicate is pushed or the file schema differs from the requested schema. A file written without key-value metadata whose schema equals the requested schema is therefore read positionally, and none of the native scan's duplicate-name resolution runs for it. For a nested struct with two sibling names that collide case-insensitively, read under Spark's default `spark.sql.caseSensitive=false`, Spark's `matchCaseInsensitiveField` raises `foundDuplicateFieldInCaseInsensitiveModeError` at read time. The native scan returns rows, with each requested child bound by position. #6004 covers the case where the requested schema itself repeats a field id. It does not cover duplicate names, and Spark's analysis-time `COLUMN_ALREADY_EXISTS` check only covers the root columns of the output, not nested struct fields. ### Steps to reproduce Write a Parquet file with parquet-mr and no key-value metadata whose schema is `s: struct<x int, X int>`, then read it with the same requested schema and `spark.sql.caseSensitive=false`. Spark raises; the native scan reads. ### Expected behavior The native scan raises the same error as Spark, or falls back to Spark for the file. ### Additional context The same opener behaviour means any nested duplicate-name shape in such a file bypasses the resolver, not only the case-insensitive one. A check at the point where the footer is read, or a planning-time decline when the requested schema holds nested names that collide under the session's case sensitivity, are the two candidate fixes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
