yihua opened a new pull request, #882:
URL: https://github.com/apache/hudi-rs/pull/882
## Description
closes #881
With an ordering field, a copy-on-write upsert merged each touched file
group by passing every stored row, together with the incoming rows, through
`RecordMerger::merge_record_batches`. Only stored rows whose key is incoming
can change, so `merge_with_event_time` now gives the merger those rows plus the
incoming ones and keeps every other stored row in place, using the `old_by_key`
map the caller already builds. A file group whose stored keys repeat still goes
through the full merge, which is what collapses the repeats today. Final rows
are unchanged; rows are now written in the order the commit-time path already
uses (stored rows in file order, then incoming winners) rather than in the
merger's descending key order.
On a 1M-row table with 10,000 updates per commit, the median upsert went
from about 1.16 s to about 0.69 s (local SSD, same final table by digest).
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_merge_with_event_time_matches_full_merge` compares the new selection
with the full merge for lower, equal and higher ordering values, repeated
incoming and stored keys, and an untouched key.
`test_cow_event_time_upsert_merges_only_touched_keys` runs upserts with delete
markers, inserts and a partial upsert end to end and compares every data
column. The existing write suites pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
