yihua opened a new issue, #881: URL: https://github.com/apache/hudi-rs/issues/881
### Feature Description A copy-on-write upsert on a table with an ordering field merges each touched file group by passing every stored row of the group, together with the incoming rows, through `RecordMerger::merge_record_batches`: a concatenation, a row-format sort over key, ordering and sequence number, and a hash map over all combined rows. Only stored rows whose key is incoming can change, so for a typical upsert that touches a small fraction of a file group almost all of that work reproduces rows that were already final. Profiling a 1M-row table with 10,000 updates per commit puts this at about 40 percent of the upsert's time. Proposed: run the merger only over the stored rows whose key is incoming plus the incoming rows, and keep every other stored row in place. Final logical rows stay the same; rows are written in the same order the commit-time path already uses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
