yihua opened a new issue, #865:
URL: https://github.com/apache/hudi-rs/issues/865
### Describe the bug
`RecordIndex::tag_location` loads the entire `record_index` metadata
partition on every call: every shard's base HFile and every log block are
parsed and every entry is Avro-decoded into a hash map, and only then are the
requested keys looked up. The cost of tagging one key is therefore proportional
to the table size, not to the number of keys, and it is paid again on every
upsert and delete.
Measured on an M3 Max against a copy-on-write table with the default record
index (10 shards, 103 MiB of index log files), timing `delete_keys` with a
single key that does not exist, which runs the lookup and nothing else:
| table rows | one-key lookup |
|---|---|
| 100k | about 1.0 s |
| 1M | 10.5 to 11.8 s |
`sample` on the process shows the time in `apache_avro` decoding, string
allocation and hash-map inserts for all 1M entries.
### To Reproduce
```python
from hudi import HudiTable
t = HudiTable.create("/tmp/t", record_key="id") # record index on by
default
t.append(one_million_rows) # any schema with a
string id
t.delete_keys(["missing"]) # ~11 s, no commit is
written
```
### Expected behavior
A lookup reads only the shards the requested keys hash to, only the HFile
blocks those keys can fall in (the block index is already in hand), and decodes
only matching entries, so an upsert of a few keys costs milliseconds regardless
of table size.
### Environment
main at 4dd65422 (after #666), macOS arm64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
