I have been reading through the thread and I think people overwhelmingly support an O(1) upgrade story. But I want to point out some realistic challenges -
1. V1's row count can be optional, but it is required by v3/v4 row lineage. So either we re-populate these field, or we check and reject upgrades on such v1 tables. 2. File path ambiguity. V4 supports relative paths, but v1/v2/v3 only use absolute paths. Need to normalize to absolute path, and carefully handle all the path references (such as DV and positional delete). Also relocation won’t work. 3. Positional delete to DV convert. When we delete on a pre V3 data file and if the data file already has positional deletes, we will need an extra step to convert them into V4 DVs, touch and re-write the old manifest. 4. The `first_row_id` back filled. The v3 spec requires to fill in `first_row_id` for the old manifests when the first time a v3 snapshot is committed. And I believe we will need to do the same for v4 upgrade. So we actually don't save too much by avoiding the re-write of manifests as we will have to re-write them anyway at the first commit. These are just a few things I can think of. They may be more when we get into the real implementation. I agree that eventually this is up to the vendor for implementation. One could choose to do a synchronized manifest convert during upgrade. But from spec perspective, I agree we should have clearly definition of the semantics when multiple versions of metadata co-exist. On 2026/08/12 00:45:01 Shawn Chang wrote: > Hi all, > > As V4 takes better shape daily, I think some items regarding the upgrade > path remain unclear. > > I'd like to better understand the expected V3 -> V4 upgrade path, > particularly the operational expectations around pre-V4 manifests after a > table is upgraded. > > From the current V4 work, my understanding is that an implementation can > perform a lightweight upgrade by creating a V4 root manifest that > references existing pre v4 manifests, and then writes new metadata in the > V4 format going forward. > > What I'm less clear about is whether we have an expected or recommended > lifecycle for those pre-V4 manifests after the upgrade. > > For example, is the expectation that an upgraded table may indefinitely > retain the structure below, until the user explicitly runs a manifest > rewrite/maintenance job? > > V4 root > -> pre-V4 data/delete manifests > -> V4 manifests > > If so, I worry that we may be shifting significant migration responsibility > onto users. In practice, users often don't run optional table maintenance > consistently. Leaving this undefined reminds me of equality deletes. > > In short, I believe we can either 1) spec it or 2) provide general guidance > to avoid a situation where the format technically supports a clean > migration path, but the practical path relies on users understanding V4. > > Have we considered solutions where the implementation eagerly converts > metadata that is cheap to convert while explicitly leaving legacy data that > is expensive to migrate alone?(applying EqDels) > > > Best, > > Shawn >
