ad1happy2go opened a new pull request, #20600:
URL: https://github.com/apache/hudi/pull/20600

   ### Describe the issue this Pull Request addresses
   
   closes #20599
   
   With 
`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true`
 and a timestamp ordering field, upserts that update existing keys fail on 1.x. 
The incoming ordering value becomes a `java.sql.Timestamp`, but the base-file 
value is a `Long`. COW fails with a `ClassCastException` in 
`shouldKeepNewerRecord`, MOR fails writing the delete block, and payload 
comparisons lose sub-millisecond ordering.
   
   ### Summary and Changelog
   
   Ordering values no longer take the key-generator timestamp conversion and 
always stay in their stored form, as the Spark record path already does. The 
flag still applies to record keys and partition paths.
   
   - The Spark datasource (`HoodieCreateRecordUtils`), HoodieStreamer 
(`HoodieStreamerUtils`) and MERGE INTO custom-payload (`ExpressionPayload`) 
record builders, plus `HoodieAvroIndexedRecord`, `DefaultHoodieRecordPayload` 
and `PartialUpdateAvroPayload`, read the ordering field without the conversion.
   - New tests: a COW/MOR upsert in `TestGenericRecordAndRowConsistency` 
covering newer, older, sub-millisecond and delete cases, plus unit tests for 
HoodieStreamer, the two payloads and `HoodieAvroIndexedRecord`. Each fails on 
master and passes with this change.
   
   ### Impact
   
   Upserts with this flag and a timestamp ordering field work again. With the 
flag on, payload comparisons on timestamp ordering fields now use the full 
stored precision (micros) instead of milliseconds, so records less than a 
millisecond apart no longer tie.
   
   ### Risk Level
   
   low. Only the ordering value path changes when the flag is on; record keys, 
partition paths and behavior with the flag off are unchanged.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(writer): keep ordering... ad1happy2go (via GitHub) via commits

Reply via email to