dwsmith1983 commented on code in PR #5654:
URL: https://github.com/apache/datafusion-comet/pull/5654#discussion_r3921345726
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -301,15 +304,35 @@ fn parquet_convert_struct_to_struct(
let from_index = match (should_match_by_id,
field_id(to_field)) {
// Spark treats a missing ID match as a missing column
rather than
// falling back to name match.
- (true, Some(id)) => from_id_to_index.get(&id).copied(),
+ (true, Some(id)) => match from_id_to_indices.get(&id) {
+ None => None,
+ Some(indices) if indices.len() == 1 =>
Some(indices[0]),
+ // Mirror Spark's
`foundDuplicateFieldInFieldIdLookupModeError`
+ // (`_LEGACY_ERROR_TEMP_2094`): a requested ID
resolving to more
+ // than one file field is ambiguous.
+ Some(indices) => {
Review Comment:
Good catch, the shortcut sailed right past the new validation. Fixed in
3d68f22f9: the relabel arm is now guarded so that when use_field_id is set and
the requested type carries field id metadata, evaluation falls through to the
struct conversion where the duplicate id lookup runs. Chose the guard at the
call site rather than inside types_differ_only_in_field_names since that
predicate is a pure structural comparison with no access to the parquet
options. Your exact probe is now a regression test (unchanged child names,
duplicate id 1, asserts the 2094 error) plus a companion pinning that the fast
path survives for name only differences without ids and for the flag alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
