danny0405 commented on code in PR #20561:
URL: https://github.com/apache/hudi/pull/20561#discussion_r4520459956


##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieArrayWritableSchemaUtils.java:
##########
@@ -147,6 +147,12 @@ private static Writable 
rewriteRecordWithNewSchemaInternal(Writable writable, Ho
         return map;
 
       case UNION:
+        // Avro/Parquet stores a union column on disk as a record of optional 
member0..memberN
+        // fields, which is also the shape Hive expects for the column. When 
the file schema
+        // (oldSchema) is that record, the writable already has the required 
representation.
+        if (oldSchema.getType() == HoodieSchemaType.RECORD && writable 
instanceof ArrayWritable) {
+          return writable;

Review Comment:
   Could we restrict this pass-through to verified equivalent member layouts? 
`RECORD` + `ArrayWritable` does not establish that the nested values already 
match the target schema, and this return skips their recursive rewrite.
   
   I reproduced this using this PR's utility source with a union `[long, 
Member]`: evolve `Member` from `{a:string, b:string}` to `{b:string, 
a:string}`. Both `areSchemasCompatible` and `checkValidEvolution` accept the 
evolution. The file's `member1` contains `[A, B]`, but this method leaves it as 
`[A, B]` instead of rewriting it to `[B, A]`. Reading it with the target field 
positions silently swaps the values of `a` and `b`.
   
   One approach is to derive the target's physical `memberN` record schema and 
reuse the existing recursive record rewrite, preserving the projection mask. 
Skip null branches when numbering members and make the member fields nullable. 
Branch correspondence also needs validation: matching `memberN` names alone is 
insufficient if union branches themselves change order. For a narrower fix, 
only pass through verified equivalent layouts and explicitly reject unsupported 
evolution.
   
   Please add a regression test with a record-valued union branch whose fields 
are reordered; the new tests only cover unchanged primitive branches. 
Validation here was an isolated utility-level reproduction, not an end-to-end 
Hive query.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to