voonhous commented on code in PR #19834:
URL: https://github.com/apache/hudi/pull/19834#discussion_r3955672542
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -574,7 +574,12 @@ private static HoodieSchema
pruneDataSchemaInternal(HoodieSchema dataSchema, Hoo
return
HoodieSchema.createMap(pruneDataSchema(dataSchema.getValueType(),
requiredSchema.getValueType(), Collections.emptySet()));
case UNION:
- throw new IllegalArgumentException("Data schema is a union");
+ // A union is a leaf as far as pruning goes: Avro resolves a branch by
its type, so dropping a
+ // branch changes the column's type instead of narrowing it. Hand back
the data schema unpruned,
+ // which is what the default arm below already does when Spark
projects a single member out of
+ // the member-struct encoding of a union. This also covers a plain
record whose fields happen to
+ // be named member0..memberN, which HoodieSparkSchemaConverters reads
back as a union.
+ return dataSchema;
Review Comment:
Confirmed. The arm switched on the required schema, so it only fired while a
union survived there; a one-member projection of a record, array or map branch
comes back as that branch's own type and fell into the RECORD/ARRAY/MAP arms.
Hoisted above the switch: either side still holding a union hands the data
schema back whole.
`TestNestedSchemaPruningOptimization` now writes a record branch and reads
`nested.member1.x`; reverting the hoist fails it with exactly `Data schema is
not a record`.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/internal/convert/InternalSchemaConverter.java:
##########
@@ -130,7 +130,7 @@ private static void collectColNamesFromSchema(HoodieSchema
schema, Deque<String>
return;
case UNION:
- collectColNamesFromSchema(schema.getNonNullType(), visited, resultSet);
+ schema.getTypes().forEach(branch -> collectColNamesFromSchema(branch,
visited, resultSet));
Review Comment:
Right, `visitSchemaToBuildType` keeps `fTypes.get(0) == null ? fTypes.get(1)
: fTypes.get(0)`, so only that branch has ids. The collector walks that branch
alone now, and a branch is never itself a union so it still terminates.
`TestInternalSchemaConverter` pins the names for a union whose record branch
is not first and round-trips it through `pruneHoodieSchemaToInternalSchema`,
which threw `cannot prune col: field1.x which does not exist in hudi table`
before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
