Hi Shawn, Clear documentation will definitely help. However, I am wondering if these concerns might be premature, given that the V4 spec is still in its early stages [1].
In the meantime, what are the user stories for adopting V3? Are there upgrade lessons we can learn from that transition, and did we provide clear documentation for it? Best regards, Manu [1] [DISCUSS] Iceberg v4 Spec Scope and Priorities <https://lists.apache.org/thread/ko8cs3tgol97f0m20yozchpxlotzl1mj> On Thu, Aug 13, 2026 at 2:03 AM Shawn Chang <[email protected]> wrote: > Thanks everyone for the thoughtful discussion! This helped clarify the > intended direction quite a bit. > > > my recommendation is that we keep the spec open enough to enable > implementations to move into the newer structures, whenever and however > they choose. > This makes sense to me. The implementations can decide if/when/how they > want to do a full upgrade, potentially even adapting the strategy based > on the size and shape of the metadata: an adaptive metadata upgrade, > perhaps :) > > > I would definitely be open to considering V5 a breaking point and say > you can only go from Pure V4 to V5 or something like that. > +1, using a more additive format version as the point to draw a clearer > boundary between pre-V4 and V4+ metadata sounds reasonable to me. > > Just to clarify my intent here, I'm not trying to overturn the existing > upgrade design. I mainly wanted to understand and align on the expected > user experience. Without a clear message around what "upgraded to V4" > guarantees versus what still depends on metadata migration, I think > different Iceberg users and implementations could develop very different > expectations. > > Given the discussion here, I think it makes sense to separate this into > two concerns: *1) V4 implementations and vendors should state clearly > if/when/how they migrate users' existing metadata to V4 2) the Iceberg > community should clearly document the common limitations and expected > behavior of a table with a mixed pre-V4/V4 metadata layer.* > > Would be interested to hear more opinions! > > Best, > Shawn > > > > On Wed, Aug 12, 2026 at 9:50 AM Anoop Johnson <[email protected]> wrote: > >> That sounds like a good plan to me. Even though v4 introduces a different >> metadata structure, we have been trying to minimize the downstream impact >> through the use of an adapter layer. >> <https://github.com/apache/iceberg/pull/16100> The new metadata structs >> used in the AMT are abstracted by the lower-level v4 reader/writers in >> iceberg-core. >> >> On Wed, Aug 12, 2026 at 8:26 AM Russell Spitzer < >> [email protected]> wrote: >> >>> I agree with the previous comments. A V4 reader must be able to read >>> V1,V2, and V3 manifest-lists and manifests. I do think in the future we may >>> want to have a break just to give implementations a rest, but that's >>> probably not good to do in the same version we introduce brand new metadata >>> layouts. Updating metadata and similar items to V4 can be an implementation >>> decision or process; no spec work is required. >>> >>> I would definitely be open to considering V5 a breaking point and say >>> you can only go from Pure V4 to V5 or something like that. >>> >>> On Wed, Aug 12, 2026 at 9:24 AM Amogh Jahagirdar <[email protected]> >>> wrote: >>> >>>> Hey all, >>>> >>>> I largely agree with Anoop here, the format largely has biased towards >>>> cheap upgrades, where readers just have to be able to handle the older >>>> structures as well. The spec defines how to assign first row IDs for older >>>> manifest entries etc. I do think that this is the right tradeoff to >>>> continue making but we can always have implementations which do "eager" >>>> upgrades on write, which I think is a good idea for "not huge" tables. >>>> >>>> In other words, while the format *enables* cheap upgrades, it's not a >>>> requirement for implementations to *have to perform* a "cheap" upgrade. >>>> >>>> >>>> I do think it's valid to jump from v1/v2 to v4; I can't see an obvious >>>> reason to prevent that. >>>> >>>> A few syncs ago, we also talked about having leaf manifests for v3 and >>>> older manifests in a v4 root manifest, having the ability to also reference >>>> manifest DVs. This was so that for large updates/deletes which touch older >>>> manifests, there's not a significant post-upgrade performance cliff from >>>> metadata write amplification when trying to produce the combined entry >>>> structure that v4 requires. This is more of an issue when many delete >>>> manifests are touched, because finding the corresponding data file entries >>>> is a potentially large scan over larger data manifests. >>>> >>>> I think the concerns around having a proliferation of older manifests >>>> are somewhat valid but practically I think with nearly every write pattern >>>> there will be ways to age out the older manifest structures, either on the >>>> commit path of an implementation or just in the background as part of >>>> rewrite manifests. In the update/delete case I referenced, at some point >>>> we'd need to compact the MDV's anyways for those older manifests, and an >>>> implementation could produce the combined entries for those manifests at >>>> that point where the cost of producing combined entry is reduced. In an >>>> append only case, older manifests would largely just need to be rewritten >>>> into newer manifests (probably in the background), and there's no 2 phase >>>> planning that's needed anyways in this case. >>>> >>>> Again though, all of the above are just implementation choices and I >>>> think we have all the tools to address these cases; my recommendation is >>>> that we keep the spec open enough to enable implementations to move into >>>> the newer structures, whenever and however they choose. I don't think >>>> the spec needs to define anything here so long as we know post-upgrade >>>> there's no degenerate cases that we don't have a path through. >>>> >>>> Thanks, >>>> >>>> Amogh Jahagirdar >>>> >>>> On Tue, Aug 11, 2026 at 11:03 PM <[email protected]> wrote: >>>> >>>>> Hi Shawn & Anoop, >>>>> >>>>> Thank you Shawn for the clear write up. I am also worried that if we >>>>> don’t provide new ways to more intentionally upgrade tables sufficiently >>>>> significantly large pre-v4 tables may never fully complete their >>>>> migrations. >>>>> >>>>> This concern for me personally is mostly forward looking. I am a >>>>> little anxious about what corners that would put us into come Table format >>>>> v6 or v7. The scenario I’m imaging is where query performance is no longer >>>>> consistent because a user will never know if their query is about to hit a >>>>> v3, or v4, or v5, or v6, etc etc manifest/data file. >>>>> >>>>> With Iceberg definitely becoming a solution for multi Petabyte sized >>>>> tables and ideally aiming to support multi Exabyte scale tables. Waiting >>>>> for the current maintenance options to rewrite (compact) files leaves us >>>>> with few options for more intentional, incremental and asynchronous ways >>>>> of >>>>> catching a table up to the intended format version for all files. >>>>> >>>>> Anoop, question because I missed a few of the AMT syncs. Have we >>>>> discussed if it will be valid to jump straight from V1/V2 to V4 in the >>>>> proposed reference implementation or will users need to go V3 to V4? I’m >>>>> expecting that it would be valid for a V1 -> V4 jump, but I don’t want to >>>>> assume. >>>>> >>>>> Best Regards, >>>>> Kurtis >>>>> >>>>> On Aug 11, 2026, at 18:51, Anoop Johnson <[email protected]> wrote: >>>>> >>>>> >>>>> Hi, Shawn - >>>>> >>>>> Thanks for starting the discussion on this. Your summary of our >>>>> current thinking on upgrades is accurate. The upgrade is an O(1) operation >>>>> that leaves prior manifest files intact. New table writes will produce the >>>>> v4 metadata. This is by design because table upgrades that require >>>>> expensive metadata rewrites add a lot of friction to upgrade to v4. >>>>> Indeed, >>>>> tables with older manifests don't get the performance benefits of v4, but >>>>> they produce the correct results and are no worse in performance than the >>>>> v3 table pre-upgrade. The table converges to the v4 metadata >>>>> representation over time as older data age out and maintenance kicks in. >>>>> >>>>> This is no different than any existing version upgrade: v1 to v2 or v2 >>>>> to v3 upgrades didn't require expensive metadata or data rewrites. V4 >>>>> doesn't make that upgrade process any worse. >>>>> >>>>> On the spec vs guidance. I lean towards guidance plus maintenance >>>>> utilities rather than a prescriptive upgrade protocol that forces >>>>> metadata rewrites. >>>>> >>>>> Best, >>>>> Anoop >>>>> >>>>> On Tue, Aug 11, 2026 at 5:45 PM Shawn Chang <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> As V4 takes better shape daily, I think some items regarding the >>>>>> upgrade path remain unclear. >>>>>> >>>>>> I'd like to better understand the expected V3 -> V4 upgrade path, >>>>>> particularly the operational expectations around pre-V4 manifests after a >>>>>> table is upgraded. >>>>>> >>>>>> From the current V4 work, my understanding is that an implementation >>>>>> can perform a lightweight upgrade by creating a V4 root manifest that >>>>>> references existing pre v4 manifests, and then writes new metadata in the >>>>>> V4 format going forward. >>>>>> >>>>>> What I'm less clear about is whether we have an expected or >>>>>> recommended lifecycle for those pre-V4 manifests after the upgrade. >>>>>> >>>>>> For example, is the expectation that an upgraded table may >>>>>> indefinitely retain the structure below, until the user explicitly >>>>>> runs a manifest rewrite/maintenance job? >>>>>> >>>>>> V4 root >>>>>> -> pre-V4 data/delete manifests >>>>>> -> V4 manifests >>>>>> >>>>>> If so, I worry that we may be shifting significant migration >>>>>> responsibility onto users. In practice, users often don't run optional >>>>>> table maintenance consistently. Leaving this undefined reminds me of >>>>>> equality deletes. >>>>>> >>>>>> In short, I believe we can either 1) spec it or 2) provide general >>>>>> guidance to avoid a situation where the format technically supports a >>>>>> clean >>>>>> migration path, but the practical path relies on users understanding V4. >>>>>> >>>>>> Have we considered solutions where the implementation eagerly >>>>>> converts metadata that is cheap to convert while explicitly leaving >>>>>> legacy >>>>>> data that is expensive to migrate alone?(applying EqDels) >>>>>> >>>>>> >>>>>> Best, >>>>>> >>>>>> Shawn >>>>>> >>>>>>
