Hi Prithvi, Thanks for starting this discussion!
I think it's a good call, and building on the existing dropTimestamp/purgeTimestamp/isDropped() fields instead of reviving entitiesDropped is a low risk in a first phase. Regarding your two questions: 1. Reserving the identifier during the hold: yes, I agree, this is the safer by default. It does mean drop-and-recreate ETL changes behavior on catalogs that opt in, but I believe it's reasonable cost given the feature is opt in and DROP PURGE is still available (as an explicit "give me the name back now"). Freeing the name immediately would reopen the shared-location/staged-create-credential-refresh race that we changed. Since unique locations are not default on yet, that invariant doesn't hold today. Gating name reusing on a second phase (once unique locations are actually universal) rather than blocking the whole feature on that "migration" seems right. 2. DROP PURGE staying "destroy now" with only plain DROP entering the hold: I also agree, that's a clean two-tier model (recoverable vs destructive) and matches how DROP_WITH_PURGE_ENABLED already behaves today. I just have one question: "the next list/create/drop-namespace in that namespace permanently deletes catalog state", is that a scan over soft-deleted entities in the namespace triggered on every list/create call? or something cheaper? If it's a scan, I would like to understand the cost on namespace with a lot of held tables, even with the feature off by default. I think it would be worth spelling out the eviction mechanism (lazy on access vs a scheduled sweep). Thanks, Regards JB On Fri, Sep 18, 2026 at 4:25 AM Prithvi S <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on opt-in table soft-delete, hold period, > and later undrop: https://github.com/apache/polaris/issues/5054 > > Polaris DROP is hard from the catalog's point of view: the active name > disappears, and there is no hold period or undrop. Operators who need a > recoverable drop (compliance / legal hold, Nessie-to-Polaris migration) > invent workarounds that Iceberg REST clients can bypass. > > The change is opt-in and default-off, so catalogs that want today's hard > DROP keep it. > > Thanks to Dmitri for looking at this earlier. His suggestion was that name > reuse after soft-delete should be blocked until tables have unique > locations (PR #4606), because shared locations plus a recreated table > interact badly with staged-create credential refresh. > > #4606 has merged, but unique locations are still off by default, existing > tables are unchanged, and client-specified locations are still allowed. So > "all tables have unique locations" is not true today, and I do not want > #5054 to wait on that migration. > > The original GitHub issue freed the table name as soon as the table was > soft-deleted, so a drop-and-recreate ETL job could create the same > identifier again during the hold. I now think that is the wrong default for > an opt-in hold/recover feature: > * A legal-hold table should not share its identifier with a new live > table. > * Undrop is unambiguous if the name is still reserved. > * Drop-and-recreate ETL is unchanged on catalogs that leave the feature > off (the default). Catalogs that enable it can DROP PURGE when they truly > want the name back immediately. > > Phase 1 (default off): Catalog-scoped, INTERNAL Iceberg tables only. > Disabled catalogs keep > today's DROP behavior. > * DROP without purge -> soft-delete: entity retained, metadata location > retained, data/metadata files not deleted. > * Soft-deleted tables are not listable or loadable on normal Iceberg REST > paths. > * The identifier stays reserved for the hold period (default 7 days). > CREATE / register with the same name fails until permanent delete or a > later undrop. > * DROP PURGE on a live table remains immediate permanent delete, using > the existing cleanup path. > * After the hold, the next list/create/drop-namespace in that namespace > permanently deletes catalog state. File cleanup is opt-in (default off) and > uses today's ENTITY_CLEANUP_SCHEDULER. > * No new Iceberg REST undrop. No change to unique-location defaults. > * No new privileges in Phase 1: soft-drop uses TABLE_DROP; purge keeps > the current drop-with-purge authorization. > > Persistence: in-place dropped entity with dropTimestamp / toPurgeTimestamp, > filtered out of the active resolve/list path. Do not revive the old > entitiesDropped SPI. > > Phase 2: List soft-deleted tables and undrop, with new privileges. Name > reuse after soft-delete only if/when a catalog actually has unique > locations for every table. > > out of scope for this thread: > * Staged-create credential refresh > * Flipping DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED default > > questions: > 1. Is reserving the identifier during the hold acceptable for Phase 1, > instead of freeing the table name on soft-delete? > 2. Is it correct that DROP PURGE stays "destroy now", and only DROP > without purge enters the hold? > > I opened a draft PR for Phase 1: > https://github.com/apache/polaris/pull/5544 > > Thanks, > Prithvi S >
