Pls ignore my previous email. Wrong thread. Yufei
On Thu, Sep 17, 2026 at 9:37 PM Yufei Gu <[email protected]> wrote: > Hi Vignesh, > > Thanks for reporting this. Based on an initial code review, the > table-level endpoint redirection appears to be a valid issue. > > Please share your proposed fix privately on this thread when ready. A > regression test covering the table-level permissions and server-side > request destination would help confirm the behavior and validate the fix. > We should also confirm the affected versions before finalizing severity and > advisory handling. > Yufei > > > On Thu, Sep 17, 2026 at 9:28 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Prithvi, >> >> Thanks for starting this discussion! >> >> I think it's a good call, and building on the existing >> dropTimestamp/purgeTimestamp/isDropped() fields instead of reviving >> entitiesDropped is a low risk in a first phase. >> >> Regarding your two questions: >> 1. Reserving the identifier during the hold: yes, I agree, this is the >> safer by default. It does mean drop-and-recreate ETL changes behavior on >> catalogs that opt in, but I believe it's reasonable cost given the feature >> is opt in and DROP PURGE is still available (as an explicit "give me the >> name back now"). Freeing the name immediately would reopen the >> shared-location/staged-create-credential-refresh race that we changed. >> Since unique locations are not default on yet, that invariant doesn't hold >> today. Gating name reusing on a second phase (once unique locations are >> actually universal) rather than blocking the whole feature on that >> "migration" seems right. >> 2. DROP PURGE staying "destroy now" with only plain DROP entering the >> hold: >> I also agree, that's a clean two-tier model (recoverable vs destructive) >> and matches how DROP_WITH_PURGE_ENABLED already behaves today. >> >> I just have one question: "the next list/create/drop-namespace in that >> namespace permanently deletes catalog state", is that a scan over >> soft-deleted entities in the namespace triggered on every list/create >> call? >> or something cheaper? If it's a scan, I would like to understand the cost >> on namespace with a lot of held tables, even with the feature off by >> default. I think it would be worth spelling out the eviction mechanism >> (lazy on access vs a scheduled sweep). >> >> Thanks, >> Regards >> JB >> >> On Fri, Sep 18, 2026 at 4:25 AM Prithvi S <[email protected]> >> wrote: >> >> > Hi all, >> > >> > I'd like to start a discussion on opt-in table soft-delete, hold period, >> > and later undrop: https://github.com/apache/polaris/issues/5054 >> > >> > Polaris DROP is hard from the catalog's point of view: the active name >> > disappears, and there is no hold period or undrop. Operators who need a >> > recoverable drop (compliance / legal hold, Nessie-to-Polaris migration) >> > invent workarounds that Iceberg REST clients can bypass. >> > >> > The change is opt-in and default-off, so catalogs that want today's hard >> > DROP keep it. >> > >> > Thanks to Dmitri for looking at this earlier. His suggestion was that >> name >> > reuse after soft-delete should be blocked until tables have unique >> > locations (PR #4606), because shared locations plus a recreated table >> > interact badly with staged-create credential refresh. >> > >> > #4606 has merged, but unique locations are still off by default, >> existing >> > tables are unchanged, and client-specified locations are still allowed. >> So >> > "all tables have unique locations" is not true today, and I do not want >> > #5054 to wait on that migration. >> > >> > The original GitHub issue freed the table name as soon as the table was >> > soft-deleted, so a drop-and-recreate ETL job could create the same >> > identifier again during the hold. I now think that is the wrong default >> for >> > an opt-in hold/recover feature: >> > * A legal-hold table should not share its identifier with a new live >> > table. >> > * Undrop is unambiguous if the name is still reserved. >> > * Drop-and-recreate ETL is unchanged on catalogs that leave the >> feature >> > off (the default). Catalogs that enable it can DROP PURGE when they >> truly >> > want the name back immediately. >> > >> > Phase 1 (default off): Catalog-scoped, INTERNAL Iceberg tables only. >> > Disabled catalogs keep >> > today's DROP behavior. >> > * DROP without purge -> soft-delete: entity retained, metadata >> location >> > retained, data/metadata files not deleted. >> > * Soft-deleted tables are not listable or loadable on normal Iceberg >> REST >> > paths. >> > * The identifier stays reserved for the hold period (default 7 days). >> > CREATE / register with the same name fails until permanent delete or a >> > later undrop. >> > * DROP PURGE on a live table remains immediate permanent delete, using >> > the existing cleanup path. >> > * After the hold, the next list/create/drop-namespace in that >> namespace >> > permanently deletes catalog state. File cleanup is opt-in (default off) >> and >> > uses today's ENTITY_CLEANUP_SCHEDULER. >> > * No new Iceberg REST undrop. No change to unique-location defaults. >> > * No new privileges in Phase 1: soft-drop uses TABLE_DROP; purge keeps >> > the current drop-with-purge authorization. >> > >> > Persistence: in-place dropped entity with dropTimestamp / >> toPurgeTimestamp, >> > filtered out of the active resolve/list path. Do not revive the old >> > entitiesDropped SPI. >> > >> > Phase 2: List soft-deleted tables and undrop, with new privileges. Name >> > reuse after soft-delete only if/when a catalog actually has unique >> > locations for every table. >> > >> > out of scope for this thread: >> > * Staged-create credential refresh >> > * Flipping DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED default >> > >> > questions: >> > 1. Is reserving the identifier during the hold acceptable for Phase 1, >> > instead of freeing the table name on soft-delete? >> > 2. Is it correct that DROP PURGE stays "destroy now", and only DROP >> > without purge enters the hold? >> > >> > I opened a draft PR for Phase 1: >> > https://github.com/apache/polaris/pull/5544 >> > >> > Thanks, >> > Prithvi S >> > >> >
