Hi Prithvi, The proposed two phase approach sounds reasonable to me. I'll try and review the PR ASAP.
Cheers, Dmitri. On Thu, Sep 17, 2026 at 10:25 PM Prithvi S <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on opt-in table soft-delete, hold period, > and later undrop: https://github.com/apache/polaris/issues/5054 > > Polaris DROP is hard from the catalog's point of view: the active name > disappears, and there is no hold period or undrop. Operators who need a > recoverable drop (compliance / legal hold, Nessie-to-Polaris migration) > invent workarounds that Iceberg REST clients can bypass. > > The change is opt-in and default-off, so catalogs that want today's hard > DROP keep it. > > Thanks to Dmitri for looking at this earlier. His suggestion was that name > reuse after soft-delete should be blocked until tables have unique > locations (PR #4606), because shared locations plus a recreated table > interact badly with staged-create credential refresh. > > #4606 has merged, but unique locations are still off by default, existing > tables are unchanged, and client-specified locations are still allowed. So > "all tables have unique locations" is not true today, and I do not want > #5054 to wait on that migration. > > The original GitHub issue freed the table name as soon as the table was > soft-deleted, so a drop-and-recreate ETL job could create the same > identifier again during the hold. I now think that is the wrong default for > an opt-in hold/recover feature: > * A legal-hold table should not share its identifier with a new live > table. > * Undrop is unambiguous if the name is still reserved. > * Drop-and-recreate ETL is unchanged on catalogs that leave the feature > off (the default). Catalogs that enable it can DROP PURGE when they truly > want the name back immediately. > > Phase 1 (default off): Catalog-scoped, INTERNAL Iceberg tables only. > Disabled catalogs keep > today's DROP behavior. > * DROP without purge -> soft-delete: entity retained, metadata location > retained, data/metadata files not deleted. > * Soft-deleted tables are not listable or loadable on normal Iceberg REST > paths. > * The identifier stays reserved for the hold period (default 7 days). > CREATE / register with the same name fails until permanent delete or a > later undrop. > * DROP PURGE on a live table remains immediate permanent delete, using > the existing cleanup path. > * After the hold, the next list/create/drop-namespace in that namespace > permanently deletes catalog state. File cleanup is opt-in (default off) and > uses today's ENTITY_CLEANUP_SCHEDULER. > * No new Iceberg REST undrop. No change to unique-location defaults. > * No new privileges in Phase 1: soft-drop uses TABLE_DROP; purge keeps > the current drop-with-purge authorization. > > Persistence: in-place dropped entity with dropTimestamp / toPurgeTimestamp, > filtered out of the active resolve/list path. Do not revive the old > entitiesDropped SPI. > > Phase 2: List soft-deleted tables and undrop, with new privileges. Name > reuse after soft-delete only if/when a catalog actually has unique > locations for every table. > > out of scope for this thread: > * Staged-create credential refresh > * Flipping DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED default > > questions: > 1. Is reserving the identifier during the hold acceptable for Phase 1, > instead of freeing the table name on soft-delete? > 2. Is it correct that DROP PURGE stays "destroy now", and only DROP > without purge enters the hold? > > I opened a draft PR for Phase 1: > https://github.com/apache/polaris/pull/5544 > > Thanks, > Prithvi S >
