yihua opened a new pull request, #873:
URL: https://github.com/apache/hudi-rs/pull/873
## Description
closes #872
Every row written to a table with the record-level index adds an index
entry, and each entry was encoded by building an `apache_avro::Value` tree and
validating it against the full `HoodieMetadataRecord` schema. The entry's shape
is fixed, so `encode_record_index_entry` now writes it straight into Avro
binary. The generic encoder stays as a test-only reference.
Appending 1M rows with the record index on went from about 21.6 s to about
3.2 s (about 1.6 s with the index off), measured through the Python binding on
a local SSD.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`encode_record_index_matches_generic_avro_encoding` checks byte equality
with the generic encoding and a decode round trip for UUID and non-UUID file
ids, a multi-byte-length partition path, non-ASCII text, and instants including
`i64::MIN`, 0 and -1. The existing record index write and lifecycle suites pass
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
