sunchao commented on code in PR #5654:
URL: https://github.com/apache/datafusion-comet/pull/5654#discussion_r3921155697
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -301,15 +304,35 @@ fn parquet_convert_struct_to_struct(
let from_index = match (should_match_by_id,
field_id(to_field)) {
// Spark treats a missing ID match as a missing column
rather than
// falling back to name match.
- (true, Some(id)) => from_id_to_index.get(&id).copied(),
+ (true, Some(id)) => match from_id_to_indices.get(&id) {
+ None => None,
+ Some(indices) if indices.len() == 1 =>
Some(indices[0]),
+ // Mirror Spark's
`foundDuplicateFieldInFieldIdLookupModeError`
+ // (`_LEGACY_ERROR_TEMP_2094`): a requested ID
resolving to more
+ // than one file field is ambiguous.
+ Some(indices) => {
Review Comment:
[P2] Run duplicate-ID validation before metadata-only struct relabeling
Could you route metadata-only struct adaptations through this validation
too? For file struct `s<x: int id=1, y: int id=1, z: int id=2>` and requested
`s<x: int id=1, y: int id=3, z: int id=2>`, Spark rejects requested ID 1 as
ambiguous. DataFusion emits a struct cast, but `CometCastColumnExpr::evaluate`
takes `types_differ_only_in_field_names` and calls `relabel_array`, because
that predicate ignores field-ID metadata. The new lookup never runs and leaves
all three physical values in place. A focused probe using the current cast
expression and a real Arrow/Parquet round trip returned `[42, 43, 44]`, while
renaming requested `x` made the same input reach the duplicate-ID error. Could
you guard the relabel shortcut for ID-based reads and add a cast-expression or
scan regression with unchanged child names?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
