linliu-code opened a new pull request, #682:
URL: https://github.com/apache/hudi-rs/pull/682
> **Stacked on #678 — review that first.** This targets `main`, so it
currently shows #678's 8 commits alongside its own 3. The delta that belongs to
this PR is 5 files, +234/-39: the last three commits. It will rebase down to
just those once #678 merges.
## Description
Pass 1 read every log file whole, so every block was fetched and fully
Arrow-decoded before the five gates chose which ones to keep. Two consequences:
a block that could not decode failed the entire read even when its instant was
rolled back, uncommitted, or outside the requested range, and a slice's decoded
batches all sat in memory at once, where the merge map they feed spills but
this intermediate `Vec` does not.
The header-only tier that avoids both already existed with no production
caller. `LogFileReader::new_streaming` and `read_all_blocks_metadata_only` were
reachable only from the memory bench, and
`Storage::get_streaming_storage_reader` only from `new_streaming` itself. Pass
1 now uses it, which is also what Java does (`tryReadContent(readLazily =
true)` seeks past the content) and what the internal reader already runs in
production.
Passes 1 and 2 read nothing but a block's header, so the range stays
unbounded and the gates alone decide, and the sweep still sees every rollback
command block: `instant_time()` and `target_instant_time()` are both header
lookups. Pass 3 already loaded a deferred block's content through a decoder
configured with the same row filter and reader schema the eager sweep used, so
an inflated block is byte-identical to an eagerly read one.
Two robustness fixes ported from the internal reader come with it. A ranged
read whose end runs past the file is clamped rather than refused
(`object_store`'s `GetRange::as_range` returns `start..len`), so an overlong
content length came back as a short buffer and failed with "failed to fill
whole buffer" from inside the decoder; it now names the truncation. And a
loaded block releases the location it needed to fetch itself, except when the
decode produced no content, since a command block decodes to `Empty` and
releasing there would make a second load fail rather than no-op.
One behavior change worth noting: `hoodie.memory.dfs.buffer.max.size` was
previously inert, because nothing reached the tier it configures. It now takes
effect.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
`test_undecodable_block_in_a_discarded_instant_does_not_fail_the_read`, with
an admitted-block control so it cannot pass because the bytes still decode.
Reverting Pass 1 to the eager reader fails it with `Invalid log block version:
4294967295`, the corrupted bytes the fixture writes.
`test_load_content_reports_a_block_running_past_the_file_end` and
`test_load_content_is_idempotent_for_every_block_type` cover the two robustness
fixes, and each was mutation-checked the same way. #678's uncommitted-log
fixtures run against this path on both reader versions, and the streaming read
is covered by
`test_stream_merges_a_slice_incrementally_and_matches_the_eager_read` on a
merge-on-read fixture.
The memory claim is measured rather than argued, using the bench already in
the tree (`HUDI_BENCH=eager|lazy ... --lib memory_bench -- --ignored`). On a 91
MB log file decoding 1.2M rows, peak resident memory over baseline goes from
+200 MB to +40 MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
