dwsmith1983 commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5536607470
Thanks, all three are addressed in a69b5f5c0, following the once-per-file design you suggested. The physical expression adapter factory already runs once per file schema, so it now resolves a small `FieldMapping` tree (struct sources, list and large list elements, map key and value, leaf) for every logical field that holds a struct, mirroring Spark's `clipParquetSchema` at each nesting level. Duplicate requested ids and ambiguous case-insensitive names are detected there, stored per logical field, and raised when the column is referenced, whether or not a cast is later emitted. The per-batch conversion receives the resolved mapping and applies it positionally, so there is no hashing or per-id allocation on the batch path; the index type is a compact entry that records an index and an ambiguity flag, and matching names are collected only when the error is built. The `contains_field_id_metadata` predicate is gone; the relabel shortcut is gated on the mapping being positional instead. Your repros: the identical `s<x: long id=1, y: long id=1>` schema now raises the duplicate id error with no cast in the plan, pinned in Rust through the exec path and in `ParquetReadV1Suite` (the Scala case fails against the previous native library and passes now). The folded placeholder collision returns the configured default again, pinned in the adapter tests with Spark-style key-value metadata on the file. One residual worth naming: DataFusion's opener skips the adapter entirely when the logical and physical schemas compare equal and no predicate exists. Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they always go through the adapter, but a file with no metadata at all and duplicated ids inside a struct would still read positionally. Happy to cover that in a follow-up if you think it matters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
