Hi Ashok, Your proposal sounds reasonable to me, +1.
Thanks, Alex On Fri, Oct 2, 2026 at 6:28 PM Ashok Borra via dev <[email protected]> wrote: > > Hi All, > > Following the review discussion on #5582 > <https://github.com/apache/polaris/pull/5582>, I'd like to bring this to > the list. > > *Problem* > > Since #5115 <https://github.com/apache/polaris/pull/5115>, every table > commit validates the locations of all direct metadata references in the > table: the manifest list for every snapshot, plus the statistics and > partition statistics files. Before that change, a regular append did not > validate these locations. > > As a result, commit work grows with the table's snapshot history, even when > a commit adds just one snapshot. Tables with thousands of snapshots can > trigger thousands of repeated path checks per commit. The full set of paths > is also included in each DEBUG message and in rejection errors. Together, > this can lead to slow commits, disk pressure from log volume, and OOMs. > > *Proposal* > > References already accepted under the same storage restrictions do not need > to be checked again on every commit. I propose: > > 1. On a regular commit, validate only direct metadata references newly > introduced relative to the base metadata: manifest lists for new snapshots, > new statistics and partition statistics files, and the new metadata file > (as today). > 2. Keep full validation for *create* and *register* operations, and for > commits that change the table location - *write.metadata.path*. > 3. Keep the parent-directory deduplication from #5582 > <https://github.com/apache/polaris/pull/5582> for full-validation paths, > with provider-aware parent path handling in StorageLocation, as suggested > in the review. > > Thanks, > Ashok
