yihua opened a new pull request, #869:
URL: https://github.com/apache/hudi-rs/pull/869
## Description
closes #865
Every upsert and delete tagged its keys by loading the whole `record_index`
metadata partition and Avro-decoding every entry, so the cost grew with the
table rather than with the batch. The lookup now lists the partition's live
slices once, hashes each key to its shard the way the writer placed it, reads
each touched shard's base through its HFile block index and its log files with
the keys pushed into the HFile block decoder, and decodes only the requested
entries. Shards are read with the table's file slice read concurrency, and a
key routed to a shard the listing does not have is an error instead of a silent
miss that would turn an upsert into a duplicate insert.
On a 1M-row table with the default ten shards, a one-key lookup went from
about 11 s to about 0.01 s, and a 10,000-key lookup takes about 0.16 s (local
SSD, measured with keys that do not exist so the time is the lookup alone).
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
New tests cover a lookup against a Spark-written nine-block record index
HFile installed as a committed compaction base (every block's first and last
key, all keys at once, and misses between blocks), the key predicate narrowing
HFile log blocks while leaving delete blocks whole, the missing-shard error,
and a 20,000-key lifecycle across all shards with updates, deletes, and
re-inserts of deleted keys. The existing write suites pass unchanged.
https://claude.ai/code/session_01PpvixgKyY2VqVo8UAdtUeF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
