Hey Varun, I noticed a couple weeks ago when this went in and was a little confused at the time because it wasn't wired in and didn't appear to be generically useable, so I'm excited to see this follow up.
I'm a little concerned based on my impression from the PRs that we're thinking about integrating this in the wrong way. Rather than inserting a specific FileIO implementation in sections of the code path, it seems this would more generically apply as a base implementation for any FileIO. I think my preference would be that if you enable eager loading, it would apply to any existing FileIO implementation and any access. Then it's just a matter of tuning the threshold where the implementation switches from eager fetching to standard/vectored IO paths. -Dan On Tue, Aug 4, 2026 at 11:27 AM vaquar khan <[email protected]> wrote: > This is awesome and really valuable, request others to review and conclude > ,if needs plz perform own benchmark. > > Regards, > Viquar Khan > > On Mon, Aug 3, 2026, 11:15 AM Kevin Liu <[email protected]> wrote: > >> Thanks for the great work, this is super exciting. I've been looking at >> object storage optimizations lately, I'll find some time this week to take >> a look. >> >> On Mon, Aug 3, 2026 at 8:19 AM Varun Lakhyani <[email protected]> >> wrote: >> >>> I am looking for views on keeping the one flag true by default to use >>> EagerInputFile in ManifestFiles read path (Major benefits in v4 Parquet >>> Manifests). >>> We have ready to run benchmark [1] and two independent sets of results >>> showing similar benefits ~25 - 55% reduction in reading Parquet manifest >>> depending on machine and file. >>> >>> I would appreciate the community's look and feedback on this - PR having >>> changes [2] and whether we can enable this path by default. >>> >>> [1] https://github.com/varun-lakhyani/iceberg/pull/1 >>> [2] https://github.com/apache/iceberg/pull/17284 >>> >>> On Wed, Jul 29, 2026 at 6:29 AM Russell Spitzer < >>> [email protected]> wrote: >>> >>>> I'm not sure if anyone else has checked this out yet, but I think it's >>>> a really exciting improvement. We should seriously consider making this a >>>> default feature in the next release. Or potentially just have it always be >>>> on. >>>> >>>> On Tue, Jul 28, 2026 at 4:58 PM Varun Lakhyani < >>>> [email protected]> wrote: >>>> >>>>> I think we can flag property to enable this as default true, It would >>>>> be a great improvement for v4 parquet manifest. >>>>> Tried to do benchmarkings as extensive as possible. >>>>> I would love to hear thoughts on this. >>>>> >>>>> Please review the PR once and would appreciate feedback. >>>>> >>>>> On Mon, Jul 20, 2026 at 10:02 PM Varun Lakhyani < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello Everyone, >>>>>> >>>>>> I evaluated using EagerInputFile in v4 manifest reader path (parquet) >>>>>> using ManifestBenchmark and S3 as object store and EC2 machine for jmh >>>>>> benchmarking. >>>>>> >>>>>> This is to make sure we are doing a proper checklist while wiring >>>>>> this EagerInputFile path here, >>>>>> Detailed benchmarking setup and results are commented on in PR[1]. >>>>>> Using an EC2 machine in the same region (ap-south-1) gives *~28.4%* >>>>>> to *~42.7%* betterment while comparing against baseline default >>>>>> while benchmarking. >>>>>> >>>>>> Please provide review and feedback. >>>>>> Thanks >>>>>> >>>>>> [1] https://github.com/apache/iceberg/pull/17284 >>>>>> -- >>>>>> Lakhyani Varun >>>>>> Indian Institute of Technology Roorkee >>>>>> Contact: +91 96246 46174 >>>>>> >>>>>>
