yihua commented on code in PR #686:
URL: https://github.com/apache/hudi-rs/pull/686#discussion_r3910873614
##########
crates/core/src/storage/reader.rs:
##########
@@ -240,62 +223,113 @@ impl StorageReader {
LogBlockFetcher::new(self.object_store.clone(), self.location.clone())
}
- /// Refill the window if the cursor has moved outside it. No-op past the
end.
- fn ensure_window(&mut self) -> Result<()> {
- if self.pos >= self.file_len {
- return Ok(());
- }
- let in_window = self.pos >= self.window_start
- && self.pos < self.window_start + self.window.len() as u64;
- if in_window {
- return Ok(());
- }
- let start = self.pos;
- let end = (start + self.window_size).min(self.file_len);
- self.window = get_range_blocking(&self.object_store, &self.location,
start, end - start)?;
- self.window_start = start;
+ /// Length of the whole file, known without reading any of it.
+ pub fn file_len(&self) -> u64 {
+ self.file_len
+ }
+
+ /// Where the cursor sits.
+ pub fn position(&self) -> u64 {
+ self.pos
+ }
+
+ /// Move the cursor. No I/O, and a position past the end of the file is
+ /// allowed: it fails at the next read, which is what lets a caller skip
past
+ /// a block's content without paying for it.
+ pub fn seek_to(&mut self, pos: u64) {
+ self.pos = pos;
+ }
+
+ /// Whether `[start, end)` lies inside the resident window.
+ fn window_covers(&self, start: u64, end: u64) -> bool {
+ start >= self.window_start && end <= self.window_start +
self.window.len() as u64
+ }
+
+ /// Fetch a window starting at the cursor.
+ async fn fill_window(&mut self) -> Result<()> {
Review Comment:
non-blocking: Two fetch patterns here look worth a measurement before the
merge-down: on a file whose blocks are smaller than the window, the walk's
refills sweep essentially the whole file and Pass 3 then re-fetches the
admitted content, so total bytes read are ~2x the old whole-file GET; and for a
block larger than the window, `is_block_corrupted`'s trailing-pointer probe
evicts and re-fetches the header-side window (seek to trailing → refill → seek
back → refill again), costing ~2 extra windows per large block. Have you
considered a bytes-fetched counter in the object-storage bench you already
plan, or restoring/keeping the walk window across the corruption probe? The
memory win is measured and real — this is only about knowing what it costs in
bandwidth where round trips were the justification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
