dwsmith1983 opened a new pull request, #5654: URL: https://github.com/apache/datafusion-comet/pull/5654
## Which issue does this PR close? Part of the restructuring of #5365 requested in review: this extracts the duplicate field and field id matching semantics that previously traveled with the Delta contrib work, re-derived on top of the folding that #5602 added. ## Rationale for this change Three places where the native parquet field lookup diverges from Spark: 1. `remap_physical_schema` only shields id-bearing logical fields whose id is missing from the file. When a logical field's id matches one physical field but a stray physical column carries that logical field's name, the stray column can still name-match through the expression adapter fallback and hijack the read. 2. `parquet_convert_struct_to_struct` silently resolves a requested field id that matches more than one physical field to the first match. Spark raises the duplicate field error in field id lookup mode. 3. Case sensitive exact name lookup on duplicate names resolves to the first field, while Spark builds its name map with `.toMap`, where the last entry wins. ## What changes are included in this PR? - All id-bearing logical fields are shielded from name matching, the shield runs after the name match pass so a successful match claims the field first, and fake placeholder names skip a reserved set built from both schemas so they can never collide with real columns. - A requested field id resolving to more than one physical field raises `SparkError::DuplicateFieldByFieldId` (`_LEGACY_ERROR_TEMP_2094`). Duplicate ids that no requested field references remain harmless. - Exact name lookup on duplicate names resolves to the last field. ## How are these changes tested? Six tests written first; four failed on unmodified main (stray column hijacking the remap, case insensitive sibling null-filled, duplicate id silently reading the first match, first-wins name resolution), two pass on main and pin behavior that must not change. Full native suite 246 passed, clippy with `-D warnings` and fmt clean, and `CometNativeReaderSuite` on Spark 3.5 (58 succeeded) against the rebuilt native library. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
