linliu-code opened a new pull request, #690:
URL: https://github.com/apache/hudi-rs/pull/690
## Description
The version two file group reader already selects a base file reader by
format, but `create_base_file_reader` refuses HFile, so a metadata table file
slice cannot be read through it. This adds an `HFileBaseFileReader` and selects
it inside the version two reader, leaving the shared factory and the legacy
reader untouched, so HFile stays refused everywhere it was refused before.
The file is buffered rather than read block by block, because the existing
decoder is constructed from a byte buffer. That is a real limit, not a
preference, so a size bound refuses a file too large to hold instead of
exhausting the heap. Reading a block at a time, which is what a key seeking
reader needs, is separate work.
Scope is the base file only. Merging HFile log blocks, key and prefix
pushdown, and decoding the record value into typed columns are each their own
change; this reader emits the key and the raw value bytes as an HFile stores
them.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
A new test reads the metadata table's `files` partition base file through
the version two reader and asserts its key set equals what
`MetadataTableFileGroupReader` returns for the same base file only slice. It
fails before this change with `UnsupportedBaseFileFormat` and passes after.
Both CI legs were run locally: default features, and `--no-default-features`
with clippy denying warnings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
