yihua opened a new pull request, #20658:
URL: https://github.com/apache/hudi/pull/20658

   ### Describe the issue this Pull Request addresses
   
   Closes #19636.
   
   A table with a nullable event-time ordering field can hold a record whose 
ordering value is null (for example a base file written before the field was 
added). Every later merge on that key fails: the file group reader maps the 
null to the `Integer` 0 default and `Long.compareTo(Integer)` throws a 
`ClassCastException`, or an NPE where the raw null survives. This breaks COW 
upserts, MOR snapshot reads and compaction, and the global index merge. #19637 
takes a different approach (keeping nulls through the read path); this PR keeps 
the change at the comparison sites.
   
   ### Summary and Changelog
   
   A stored null ordering value now ranks below any incoming value, so the 
incoming record wins, which matches how `DefaultHoodieRecordPayload` already 
treats a persisted null.
   
   - Add `OrderingValues.isBaseOrderingHigher` and use it in 
`BufferedRecordMergerFactory` (event-time and partial-update mergers), 
`HoodieRecordMerger.maxOrderingValue`, and the Spark, Flink and Hive event-time 
mergers. A base value that is null, or the `Integer` 0 default while the 
incoming value has another class, ranks lowest; multi-field values apply this 
per field.
   - Make `OrderingValues.isSameClass` and `ArrayComparable.isValueSameClass` 
null-safe, so the delete merges in `BufferedRecordMergerFactory` and 
`HoodieMergedLogRecordScanner` let the delete win instead of throwing an NPE.
   - A stored `Integer` 0 against an incoming `Integer` still compares as a 
real value, so int ordering fields behave as before. Two non-null values of 
different classes, or an incoming null against a real stored value, still fail, 
now with an `IllegalArgumentException` naming both types.
   - Tests: unit tests for the helper, the buffered mergers, the delete paths, 
the log scanner and each engine merger, plus a Spark functional case table 
(COW, MOR, MOR with `GLOBAL_SIMPLE`, and the partial-update merger) that seeds 
stored nulls through a row-writer bulk insert and asserts the rows after an 
upsert with an update and a delete, before and after compaction. Every 
stored-null case fails without the fix.
   
   ### Impact
   
   Tables holding a null in a nullable ordering field can be upserted, read and 
compacted again. No behavior change for non-null ordering values.
   
   ### Risk Level
   
   low
   
   Only inputs that threw a `ClassCastException` or NPE before change behavior; 
same-class comparisons are untouched.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to