Hey Russell, Thanks for sharing the proposal! A few of us (Ryan, Dan, Anoop and I) have also been working on a proposal for an adaptive metadata tree structure as part of enabling more efficient one file commits. From a read of the summary, it's great to see that we're thinking along the same lines about how to tackle this fundamental area!
Here is our proposal: https://docs.google.com/document/d/1q2asTpq471pltOTC6AsTLQIQcgEsh0AvEhRWnCcvZn0 <https://docs.google.com/document/d/1q2asTpq471pltOTC6AsTLQIQcgEsh0AvEhRWnCcvZn0> Thanks, Amogh Jahagirdar On Mon, Jul 14, 2025 at 8:08 PM Russell Spitzer <russell.spit...@gmail.com> wrote: > Hey y'all! > > We (Yi Fang, Steven Wu and Myself) wanted to share some > of the thoughts we had on how one-file commits could work in Iceberg. This > is pretty > much just a high level overview of the concepts we think we need and how > Iceberg would behave. > We haven't gone very far into the actual implementation and changes that > would need to occur in the > SDK to make this happen. > > The high level summary is: > > Manifest Lists are out > Root Manifests take their place > A Root manifest can have data manifests, delete manifests, manifest > delete vectors, data delete vectors and data files > Manifest delete vectors allow for modifying a manifest without deleting > it entirely > Data files let you append without writing an intermediary manifest > Having child data and delete manifests lets you still scale > > Please take a look if you like, > > https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.0 > > I'm excited to see what other proposals and Ideas are floating around the > community, > Russ > > On Wed, Jul 2, 2025 at 6:29 PM John Zhuge <jzh...@apache.org> wrote: > >> Very excited about the idea! >> >> On Wed, Jul 2, 2025 at 1:17 PM Anoop Johnson <anoop.k.john...@gmail.com> >> wrote: >> >>> I'm very interested in this initiative. Micah Kornfield and I presented >>> <https://youtu.be/4d4nqKkANdM?si=9TXgaUIXbq-l8idi&t=1405> on >>> high-throughput ingestion for Iceberg tables at the 2024 Iceberg Summit, >>> which leveraged Google infrastructure like Colossus for efficient appends. >>> >>> This new proposal is particularly exciting because it offers significant >>> advancements in commit latency and metadata storage footprint. Furthermore, >>> a consistent manifest structure promises to simplify the design and >>> codebase, which is a major benefit. >>> >>> A related idea I've been exploring is having a loose affinity between >>> data and delete manifests. While the current separation of data and delete >>> manifests in Iceberg is valuable for avoiding data file rewrites (and stats >>> updates) when deletes change, it does necessitate a join operation during >>> reads. I'd be keen to discuss approaches that could potentially reduce this >>> read-side cost while retaining the benefits of separate manifests. >>> >>> Best, >>> Anoop >>> >>> >>> >>> On Fri, Jun 13, 2025 at 11:06 AM Jagdeep Sidhu <sidhujagde...@gmail.com> >>> wrote: >>> >>>> Hi everyone, >>>> >>>> I am new to the Iceberg community but would love to participate in >>>> these discussions to reduce the number of file writes, especially for small >>>> writes/commits. >>>> >>>> Thank you! >>>> -Jagdeep >>>> >>>> On Thu, Jun 5, 2025 at 4:02 PM Anurag Mantripragada >>>> <amantriprag...@apple.com.invalid> wrote: >>>> >>>>> We have been hitting all the metadata problems you mentioned, Ryan. >>>>> I’m on-board to help however I can to improve this area. >>>>> >>>>> >>>>> ~ Anurag Mantripragada >>>>> >>>>> On Jun 3, 2025, at 2:22 AM, Huang-Hsiang Cheng >>>>> <hua...@apple.com.INVALID> wrote: >>>>> >>>>> I am interested in this idea and looking forward to collaboration. >>>>> >>>>> Thanks, >>>>> Huang-Hsiang >>>>> >>>>> On Jun 2, 2025, at 10:14 AM, namratha mk <nmk...@gmail.com> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I am interested in contributing to this effort. >>>>> >>>>> Thanks, >>>>> Namratha >>>>> >>>>> On Thu, May 29, 2025 at 1:36 PM Amogh Jahagirdar <2am...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks for kicking this thread off Ryan, I'm interested in helping >>>>>> out here! I've been working on a proposal in this area and it would be >>>>>> great to collaborate with different folks and exchange ideas here, since >>>>>> I >>>>>> think a lot of people are interested in solving this problem. >>>>>> >>>>>> Thanks, >>>>>> Amogh Jahagirdar >>>>>> >>>>>> On Thu, May 29, 2025 at 2:25 PM Ryan Blue <rdb...@gmail.com> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> Like Russell’s recent note, I’m starting a thread to connect those >>>>>>> of us that are interested in the idea of changing Iceberg’s metadata in >>>>>>> v4 >>>>>>> so that in most cases committing a change only requires writing one >>>>>>> additional metadata file. >>>>>>> >>>>>>> *Idea: One-file commits* >>>>>>> >>>>>>> The current Iceberg metadata structure requires writing at least one >>>>>>> manifest and a new manifest list to produce a new snapshot. The goal of >>>>>>> this work is to allow more flexibility by allowing the manifest list >>>>>>> layer >>>>>>> to store data and delete files. As a result, only one file write would >>>>>>> be >>>>>>> needed before committing the new snapshot. In addition, this work will >>>>>>> also >>>>>>> try to explore: >>>>>>> >>>>>>> - Avoiding small manifests that must be read in parallel and >>>>>>> later compacted (metadata maintenance changes) >>>>>>> - Extend metadata skipping to use aggregated column ranges that >>>>>>> are compatible with geospatial data (manifest metadata) >>>>>>> - Using soft deletes to avoid rewriting existing manifests >>>>>>> (metadata DVs) >>>>>>> >>>>>>> If you’re interested in these problems, please reply! >>>>>>> >>>>>>> Ryan >>>>>>> >>>>>> >>>>> >>>>> >> >> -- >> John Zhuge >> >
