Hi Alex, > leave room for 3 values: STAGED, LIVE, DELETED [...]
This is a nice idea! I think it is a promising avenue to explore for supporting reliable location overlap checks and credential refresh for staged creates. Cheers, Dmitri. On Tue, Nov 17, 2026 at 7:16 AM Alex Dutra via dev <[email protected]> wrote: > Hi all, > > A correction to my previous message, and a few suggestions: > > 1) Spark can ask the catalog to purge. Since Iceberg 1.12, Spark has a > catalog option, rest-catalog-purge, off by default. When set to true, > Spark deletes no files itself. Without the option, which includes all > earlier Iceberg versions, my previous point stands though: Spark drops > without purge, then deletes the files itself. Recommending > rest-catalog-purge=true for catalogs with soft delete enabled seems > worth documenting in [5544]. > > 2) But catalogs can also ask clients not to delete files. Iceberg > Java's S3FileIO has a property, s3.delete-enabled. When a catalog sets > this property to false, S3FileIO silently skips deletes. > Interestingly, Lakekeeper does this by default for warehouses with > soft deletion enabled. Should Polaris do something similar? This > property has a few drawbacks, the biggest one being that it disables > all client-side deletes, including on live tables (i.e., when running > expire_snapshots or remove_orphan_files). Therefore, it only makes > sense to send that property **if the catalog handles all file > deletions server-side**, including snapshot expirations, which Polaris > doesn't do today. (Lakekeeper does that in its commercial offering > only, afaict; for the open-source edition, the documentation tells > users to override s3.delete-enabled=true in their Spark configuration > for maintenance jobs.) All of this to say, pushing > s3.delete-enabled=false could be an opt-in later, but for that Polaris > should first implement a proper snapshot expiry service. > > 3) I would like to introduce soft-creations too, for staged table > creations. IOW, the table would be in 3 possible states: STAGED, LIVE, > DELETED. Today a staged table isn't persisted until the transaction > commits, so Polaris has nothing to hold its locations or to authorize > against in the meantime. A persisted STAGED entity would: > > - hold its locations and reserve its name, like a soft-deleted table; > - let remote signing authorize against the entity instead of a token; > - let Polaris clean up abandoned creates. > > It would require keeping the entity invisible to list and load until > it commits, and an expiry service for abandoned entries, much like the > hold for soft-deleted tables in [5544]. It also changes concurrent > creates of the same name to fail at staging time rather than at commit > time, which I believe is a net improvement. > > I'm not suggesting we do all this for Phase 1, only that the state > model should leave room for 3 values: STAGED, LIVE, DELETED. > > WDYT? > > Thanks, > Alex > > [5544]: https://github.com/apache/polaris/pull/5544 > > On Fri, Nov 13, 2026 at 5:54 PM Alex Dutra <[email protected]> wrote: > > > > Hi Prithvi, hi all, > > > > Two points from the remote signing work that relate to this proposal. > > > > 1. Spark's DROP TABLE ... PURGE is a drop without purge. > > > > Spark doesn't ask the catalog to purge: > > > > - It loads the table and calls the catalog's drop with purge=false. > > - It then deletes the table's files itself, through the table's FileIO. > > > > With Spark, the catalog never sees a purge. On a catalog with > > soft-delete enabled, a Spark DROP TABLE ... PURGE would: > > > > - put the table on hold, with its name reserved, even though the user > > asked for a purge; > > - delete its files right away from the client. > > > > The hold would then keep a table whose files are gone, which matters > > once recovery is added. With vended credentials, Polaris can't see or > > prevent those deletes. This might be worth documenting. > > > > 2. Soft-delete would help remote signing. > > > > I'm reviving S3 remote signing. Polaris authorizes each sign request > > against the table, so it does see those deletes. Today, after a drop, > > the table and its grants are gone. My current approach records the > > principal's DROP_TABLE_WITH_PURGE decision in the remote signing token > > when the table is loaded. After the drop, it then allows only reads > > and deletes within the table's locations. > > > > In [5544], a soft-deleted table keeps its row, id and grants. So for > > those tables Polaris could: > > > > - authorize the deletes against the table itself when they happen, > > instead of relying on a decision recorded at load time; > > - choose a policy for the hold: either allow the deletes, since the > > client asked for a purge, or refuse them, so the table stays > > recoverable. Only remote signing makes the second option possible. > > > > Two questions: > > > > - Which policy makes sense for deletes during the hold? > > - The resolver hides dropped entities. Would an option to resolve a > > soft-deleted table, for this purpose only, be acceptable? > > > > Neither point should block Phase 1. I just want to make sure the two > > features fit together. > > > > Thanks, > > Alex > > > > [5544]: https://github.com/apache/polaris/pull/5544 > > > > > > On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev > > <[email protected]> wrote: > > > > > > Hi Prithvi and all, > > > > > > I did another review round in GH. > > > > > > Re-posting here the key comment: I'm very hesitant about > > > expiring soft-deleted entities during REST requests against other > entities. > > > I think the resulting behaviours are too "magical" from the end user's > > > perspective. > > > > > > I believe the expiry use case belongs more naturally in the Admin Tool > > > given the current codebase. Eventually it could also be handled by the > > > delegation service [3990]. > > > > > > WDYT? > > > > > > Thanks, > > > Dmitri. > > > [3990] https://github.com/apache/polaris/pull/3990 > > > > > > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected] > > > > > wrote: > > > > > > > Hi all, > > > > > > > > Quick follow-up on the cleanup gating for Phase 1: > > > > holding the identifier until the entity cleanup and its > > > > child tasks finish, then releasing it. > > > > > > > > While implementing this, I found that the coupling is > > > > in the metastore manager. dropEntityIfExists removes > > > > the entity (freeing the name) and schedules the cleanup > > > > task in the same transaction, by design. > > > > > > > > To keep the name reserved until cleanup completes, I'd > > > > need to separate these operations: keep the dropped > > > > entity, schedule cleanup, and remove the entity only > > > > after the cleanup task and its child tasks have finished. > > > > > > > > We don't currently have a way to detect when a task > > > > tree has completed. Tasks are fire-and-forget and > > > > self-remove on success, so I'd also need to add a small > > > > completion check. > > > > > > > > This touches PolarisMetaStoreManager, which has both > > > > JDBC (atomic and transactional) and NoSQL > > > > implementations. I wanted to check the direction before > > > > coding, in line with our interface-change guidance. > > > > > > > > Does keeping the entity reserved and gating its removal > > > > on cleanup completion sound right, or would you prefer > > > > a different approach, such as a dedicated purge task > > > > that handles the final entity removal? > > > > > > > > Thanks, > > > > Prithvi > > > > > > > > > > > > > >
