I think it's fair to say that it may not benefit all reads equally (puffin files are a good example), but the issue isn't just confined to parquet (ORC would also benefit).
I felt this was general enough that we wouldn't necessarily need to make the decision at every point of use, which I believe is roughly the same approach some accelerator libraries are taking. It looks like we could integrate this at one of three points: 1. The FileReader API, 2. The FileIO layer, 3. Directly from on the InputStream at the point of use. The third options seems like the tightest coupling and maybe the first is a better place to see if there's an obvious integration path. -Dan On Mon, Aug 10, 2026 at 7:53 AM Russell Spitzer <[email protected]> wrote: > I'd really like to understand the pros of putting this in FileIO a bit > better. > > If we want to achieve global coverage immediately, we could just jump to > Parquet.java > <https://github.com/apache/iceberg/blob/3d682a3b65/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L1269-L1271> > instead > of doing > a manifest specific code change. That would cover all usages we care about > and avoid adding > complexity to the IO implementation for what is essentially a performance > fix for parquet-java. Ideally, this should be a fix in parquet-java > directly, > so keeping it contained to our Parquet reader code feels like the right > place for me. There's no > reason for us to trigger the same path for, say, a Puffin file or > metadata.json. > > On Wed, Aug 5, 2026 at 7:30 PM Daniel Weeks <[email protected]> wrote: > >> We almost always have the file length from metadata (we already plumbed >> this through to avoid additional head requests). >> >> We can infer the type from the path, but it doesn't seem like it would be >> necessary if Avro is a wash and parquet is faster. >> >> >> >> On Wed, Aug 5, 2026, 2:46 PM Russell Spitzer <[email protected]> >> wrote: >> >>> It just seems a lot more complicated to me that the decision on whether >>> to prefetch and catch the file would be made by the fileio and not the tool >>> opening the file. I wasn’t saying it would be slower, just unnecessary. If >>> you check the pr, Varun actually did this benchmark already and Avro is >>> basically unchanged. >>> >>> Are you saying fileio should know the size of the file and its type when >>> opening? At the moment it really is pretty opaque about that sort of thing. >>> >>> On Wed, Aug 5, 2026 at 1:43 PM Daniel Weeks <[email protected]> wrote: >>> >>>> I'm not convinced it's actually worse for Avro. With Avro, you >>>> typically read the entire file if you intend to open it at all. There's no >>>> real skipping or ranged projection happening, so I'm not convinced that a >>>> full file fetch would be net slower than the typical read path, which may >>>> require multiple incremental fetches (it might even help some of the byte >>>> skipping paths). >>>> >>>> That would be good to benchmark, but I know multiple implementations >>>> have this exact type of optimization. If we think we need file type >>>> specific behavior, we could incorporate that into the FileIO abstraction as >>>> well. >>>> >>>> On Wed, Aug 5, 2026 at 8:02 AM Russell Spitzer < >>>> [email protected]> wrote: >>>> >>>>> Yeah one of the worries here is that while this makes a lot of sense >>>>> for parquet, it doesn’t make any sense for Avro. We were mostly targeting >>>>> a >>>>> more conservative set of changes and then more global modifications later. >>>>> >>>>> On Wed, Aug 5, 2026 at 9:41 AM Daniel Weeks <[email protected]> wrote: >>>>> >>>>>> Sorry about the confusion, I think my phrasing wasn't quite accurate, >>>>>> but the point remains. >>>>>> >>>>>> The referenced PR for integrating this directly in the ManifestFiles >>>>>> class, which narrowly targets the metadata path, not making this a >>>>>> general >>>>>> FileIO capability, but rather a bespoke shortcircut in the metadata path. >>>>>> >>>>>> Prefetching like this would potentially benefit any file operation >>>>>> (particularly with small parquet files), but that's not what's proposed. >>>>>> >>>>>> I'm suggesting integrating this into the existing FileIO >>>>>> implementations so any InputFile::newStream would be accelerated. The >>>>>> difference is making it native to the FileIO as opposed to wrapping the >>>>>> stream after. >>>>>> >>>>>> Hopefully that clarifies, >>>>>> -Dan >>>>>> >>>>>> On Wed, Aug 5, 2026 at 5:13 AM Russell Spitzer < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Could you elaborate a bit more Dan? This isn’t part of the fileio, >>>>>>> it’s a separate file class (much like encrypting file) which is used by >>>>>>> all >>>>>>> IO like the encryption implementation. >>>>>>> >>>>>>> On Tue, Aug 4, 2026 at 1:34 PM Daniel Weeks <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Varun, >>>>>>>> >>>>>>>> I noticed a couple weeks ago when this went in and was a little >>>>>>>> confused at the time because it wasn't wired in and didn't appear to be >>>>>>>> generically useable, so I'm excited to see this follow up. >>>>>>>> >>>>>>>> I'm a little concerned based on my impression from the PRs that >>>>>>>> we're thinking about integrating this in the wrong way. Rather than >>>>>>>> inserting a specific FileIO implementation in sections of the code >>>>>>>> path, it >>>>>>>> seems this would more generically apply as a base implementation for >>>>>>>> any >>>>>>>> FileIO. >>>>>>>> >>>>>>>> I think my preference would be that if you enable eager loading, it >>>>>>>> would apply to any existing FileIO implementation and any access. Then >>>>>>>> it's just a matter of tuning the threshold where the implementation >>>>>>>> switches from eager fetching to standard/vectored IO paths. >>>>>>>> >>>>>>>> -Dan >>>>>>>> >>>>>>>> On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> This is awesome and really valuable, request others to review and >>>>>>>>> conclude ,if needs plz perform own benchmark. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Viquar Khan >>>>>>>>> >>>>>>>>> On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the great work, this is super exciting. I've been >>>>>>>>>> looking at object storage optimizations lately, I'll find some time >>>>>>>>>> this >>>>>>>>>> week to take a look. >>>>>>>>>> >>>>>>>>>> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I am looking for views on keeping the one flag true by default >>>>>>>>>>> to use EagerInputFile in ManifestFiles read path (Major benefits in >>>>>>>>>>> v4 >>>>>>>>>>> Parquet Manifests). >>>>>>>>>>> We have ready to run benchmark [1] and two independent sets of >>>>>>>>>>> results showing similar benefits ~25 - 55% reduction in reading >>>>>>>>>>> Parquet >>>>>>>>>>> manifest depending on machine and file. >>>>>>>>>>> >>>>>>>>>>> I would appreciate the community's look and feedback on this - >>>>>>>>>>> PR having changes [2] and whether we can enable this path by >>>>>>>>>>> default. >>>>>>>>>>> >>>>>>>>>>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>>>>>>>>>> [2] https://github.com/apache/iceberg/pull/17284 >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I'm not sure if anyone else has checked this out yet, but I >>>>>>>>>>>> think it's a really exciting improvement. We should seriously >>>>>>>>>>>> consider >>>>>>>>>>>> making this a default feature in the next release. Or potentially >>>>>>>>>>>> just have >>>>>>>>>>>> it always be on. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think we can flag property to enable this as default true, >>>>>>>>>>>>> It would be a great improvement for v4 parquet manifest. >>>>>>>>>>>>> Tried to do benchmarkings as extensive as possible. >>>>>>>>>>>>> I would love to hear thoughts on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Please review the PR once and would appreciate feedback. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello Everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I evaluated using EagerInputFile in v4 manifest reader path >>>>>>>>>>>>>> (parquet) using ManifestBenchmark and S3 as object store and EC2 >>>>>>>>>>>>>> machine >>>>>>>>>>>>>> for jmh benchmarking. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is to make sure we are doing a proper checklist while >>>>>>>>>>>>>> wiring this EagerInputFile path here, >>>>>>>>>>>>>> Detailed benchmarking setup and results are commented on in >>>>>>>>>>>>>> PR[1]. >>>>>>>>>>>>>> Using an EC2 machine in the same region (ap-south-1) gives >>>>>>>>>>>>>> *~28.4%* to *~42.7%* betterment while comparing against >>>>>>>>>>>>>> baseline default while benchmarking. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please provide review and feedback. >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Lakhyani Varun >>>>>>>>>>>>>> Indian Institute of Technology Roorkee >>>>>>>>>>>>>> Contact: +91 96246 46174 >>>>>>>>>>>>>> >>>>>>>>>>>>>>
