nsivabalan opened a new pull request, #20764:
URL: https://github.com/apache/hudi/pull/20764
### Describe the issue this Pull Request addresses
Closes #20762
A MERGE INTO that updates an existing key on a COW table, merged through
`HoodieWriteMergeHandle` (or its subclasses `HoodieSortedMergeHandle` /
`HoodieMergeHandleWithChangeLog`), can fail or write wrong data. The handle
wraps the merge result in a new `ExpressionPayload` via
`constructHoodieRecord`. `shouldIgnore` then calls
`ExpressionPayload.getInsertValue` on the already-merged row, which causes two
bugs:
1. **Decode failure / shifted values.** `getInsertValue` decodes with the
MERGE source schema (`hoodie.payload.record.schema`), not the table schema.
`BaseAvroPayload.getRecord` re-encodes the merged row with the table schema
(nullable unions) and decodes those bytes with the source schema. When the
source columns are NOT NULL (e.g. a subquery of constants), the union branch
bytes shift the decode: `AvroRuntimeException: Malformed data. Length is
negative`, `EOFException`, `Invalid int encoding`, or silently shifted values.
2. **Matched update silently dropped (also with matching schemas).** On COW,
`getInsertValue` runs `processNotMatchedRecord`, so the `WHEN NOT MATCHED`
clause is re-applied to the merged row. If its condition is false for that row
(e.g. `WHEN NOT MATCHED AND s.rating > 10 THEN INSERT *`), it returns
`SENTINEL`, `shouldIgnore` is true, and the old row is kept, with no error.
The default `FileGroupReaderBasedMergeHandle` is not affected; it already
writes merge results via `constructFinalHoodieRecord` (`HoodieFileGroupReader`).
### Summary and Changelog
MERGE INTO through `HoodieWriteMergeHandle` now writes the merged row as-is,
matching the default merge handle.
- `HoodieWriteMergeHandle#write`: build the combined record with
`constructFinalHoodieRecord` instead of `constructHoodieRecord`. For Avro this
yields a `HoodieAvroIndexedRecord` (or `HoodieEmptyRecord` for deletes) rather
than a payload-wrapped record. Spark/Flink/Hive record contexts do not override
`constructFinalHoodieRecord`, so they are unchanged.
- Rows no update condition applies to are still kept:
`HoodieAvroRecordMerger` maps `SENTINEL` to the older record, and
`writeUpdateRecord`'s identity check copies the old record.
- Deletes are unchanged: an empty merge result becomes a delete record.
- New `TestMergeIntoWithWriteMergeHandle` (COW, `HoodieWriteMergeHandle`
pinned):
- NOT NULL source, update + insert: failed before the fix with `Malformed
data. Length is negative: -49`.
- Nullable source, update + insert: control, passed before and after.
- Nullable source, matched update + false `WHEN NOT MATCHED` condition:
before the fix the update was silently dropped (`[a1,phoenix,1,1000,US]`
instead of `[a1,scottsdale,9,2000,US]`).
### Impact
Fixes MERGE INTO correctness for users who set
`hoodie.write.merge.handle.class=org.apache.hudi.io.HoodieWriteMergeHandle` (or
hit the sorted / CDC subclasses on the non-file-group-reader paths). No public
API or config change.
### Risk Level
low
One-line change on the legacy merge handle, aligning it with the default
file group reader path. Verified locally: the new suite (fails before, passes
after), merge handle unit tests in hudi-client-common / hudi-spark-client /
hudi-java-client, `TestMergeHandle`, and the Spark SQL MERGE INTO / INSERT
suites.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]