vishnuprakaz opened a new pull request, #17418:
URL: https://github.com/apache/iceberg/pull/17418
`PuffinReader.readAll` currently issues one `seek` + read per blob. This
resolves
the long-standing TODO from #4537:
Contiguous (or overlapping) blobs are now read in a single request and
handed back
as no-copy ByteBuffer views over that region. Since Puffin stats/index
blobs are
written contiguously, a readAll over many blobs collapses from N reads to
one
the win matters most on object stores, where each read is a round trip.
### Approach
This coalesces contiguous blob reads: after sorting by offset,
adjacent/overlapping blobs are merged into a single read, and a run is split
once it would exceed a max read size.
Also `RangeReadable.readVectored(...)` was considered as an alternative,
but its default implementation doesn't coalesce on S3/Hadoop (only some
FileIOs, e.g. GCS analytics, optimize it) and eagerly buffers every range.
Coalescing contiguous reads captures the benefit for real Puffin layouts
without reading bytes the caller didn't request, and reads stay lazy (a region
is fetched only when iterated).
The max read size is capped at 8 MiB the same value as
`MANIFEST_TARGET_SIZE`, since Puffin blobs are metadata. The cap bounds the
extra memory coalescing holds compared to the original per-blob reads (a single
blob larger than the cap is still read on its own).
### Are these changes tested?
- Existing testReadMetricDataUncompressed /
testReadMetricDataCompressedZstd cover
readAll end-to-end for both codecs (contiguous blobs, nonzero-offset
slice).
- New end-to-end readAll test with blobs far apart, asserting both decode
across two
separate reads.
- New unit tests for the grouping logic: contiguous/overlap merge, gap
split, region-size
split, and an oversized single blob kept in its own read.
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
