nsivabalan opened a new pull request, #20764:
URL: https://github.com/apache/hudi/pull/20764

   ### Describe the issue this Pull Request addresses
   
   Closes #20762
   
   A MERGE INTO that updates an existing key on a COW table, merged through 
`HoodieWriteMergeHandle` (or its subclasses `HoodieSortedMergeHandle` / 
`HoodieMergeHandleWithChangeLog`), can fail or write wrong data. The handle 
wraps the merge result in a new `ExpressionPayload` via 
`constructHoodieRecord`. `shouldIgnore` then calls 
`ExpressionPayload.getInsertValue` on the already-merged row, which causes two 
bugs:
   
   1. **Decode failure / shifted values.** `getInsertValue` decodes with the 
MERGE source schema (`hoodie.payload.record.schema`), not the table schema. 
`BaseAvroPayload.getRecord` re-encodes the merged row with the table schema 
(nullable unions) and decodes those bytes with the source schema. When the 
source columns are NOT NULL (e.g. a subquery of constants), the union branch 
bytes shift the decode: `AvroRuntimeException: Malformed data. Length is 
negative`, `EOFException`, `Invalid int encoding`, or silently shifted values.
   2. **Matched update silently dropped (also with matching schemas).** On COW, 
`getInsertValue` runs `processNotMatchedRecord`, so the `WHEN NOT MATCHED` 
clause is re-applied to the merged row. If its condition is false for that row 
(e.g. `WHEN NOT MATCHED AND s.rating > 10 THEN INSERT *`), it returns 
`SENTINEL`, `shouldIgnore` is true, and the old row is kept, with no error.
   
   The default `FileGroupReaderBasedMergeHandle` is not affected; it already 
writes merge results via `constructFinalHoodieRecord` (`HoodieFileGroupReader`).
   
   ### Summary and Changelog
   
   MERGE INTO through `HoodieWriteMergeHandle` now writes the merged row as-is, 
matching the default merge handle.
   
   - `HoodieWriteMergeHandle#write`: build the combined record with 
`constructFinalHoodieRecord` instead of `constructHoodieRecord`. For Avro this 
yields a `HoodieAvroIndexedRecord` (or `HoodieEmptyRecord` for deletes) rather 
than a payload-wrapped record. Spark/Flink/Hive record contexts do not override 
`constructFinalHoodieRecord`, so they are unchanged.
     - Rows no update condition applies to are still kept: 
`HoodieAvroRecordMerger` maps `SENTINEL` to the older record, and 
`writeUpdateRecord`'s identity check copies the old record.
     - Deletes are unchanged: an empty merge result becomes a delete record.
   - New `TestMergeIntoWithWriteMergeHandle` (COW, `HoodieWriteMergeHandle` 
pinned):
     - NOT NULL source, update + insert: failed before the fix with `Malformed 
data. Length is negative: -49`.
     - Nullable source, update + insert: control, passed before and after.
     - Nullable source, matched update + false `WHEN NOT MATCHED` condition: 
before the fix the update was silently dropped (`[a1,phoenix,1,1000,US]` 
instead of `[a1,scottsdale,9,2000,US]`).
   
   ### Impact
   
   Fixes MERGE INTO correctness for users who set 
`hoodie.write.merge.handle.class=org.apache.hudi.io.HoodieWriteMergeHandle` (or 
hit the sorted / CDC subclasses on the non-file-group-reader paths). No public 
API or config change.
   
   ### Risk Level
   
   low
   
   One-line change on the legacy merge handle, aligning it with the default 
file group reader path. Verified locally: the new suite (fails before, passes 
after), merge handle unit tests in hudi-client-common / hudi-spark-client / 
hudi-java-client, `TestMergeHandle`, and the Spark SQL MERGE INTO / INSERT 
suites.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to