Hi all, A correction to my previous message, and a few suggestions:
1) Spark can ask the catalog to purge. Since Iceberg 1.12, Spark has a catalog option, rest-catalog-purge, off by default. When set to true, Spark deletes no files itself. Without the option, which includes all earlier Iceberg versions, my previous point stands though: Spark drops without purge, then deletes the files itself. Recommending rest-catalog-purge=true for catalogs with soft delete enabled seems worth documenting in [5544]. 2) But catalogs can also ask clients not to delete files. Iceberg Java's S3FileIO has a property, s3.delete-enabled. When a catalog sets this property to false, S3FileIO silently skips deletes. Interestingly, Lakekeeper does this by default for warehouses with soft deletion enabled. Should Polaris do something similar? This property has a few drawbacks, the biggest one being that it disables all client-side deletes, including on live tables (i.e., when running expire_snapshots or remove_orphan_files). Therefore, it only makes sense to send that property **if the catalog handles all file deletions server-side**, including snapshot expirations, which Polaris doesn't do today. (Lakekeeper does that in its commercial offering only, afaict; for the open-source edition, the documentation tells users to override s3.delete-enabled=true in their Spark configuration for maintenance jobs.) All of this to say, pushing s3.delete-enabled=false could be an opt-in later, but for that Polaris should first implement a proper snapshot expiry service. 3) I would like to introduce soft-creations too, for staged table creations. IOW, the table would be in 3 possible states: STAGED, LIVE, DELETED. Today a staged table isn't persisted until the transaction commits, so Polaris has nothing to hold its locations or to authorize against in the meantime. A persisted STAGED entity would: - hold its locations and reserve its name, like a soft-deleted table; - let remote signing authorize against the entity instead of a token; - let Polaris clean up abandoned creates. It would require keeping the entity invisible to list and load until it commits, and an expiry service for abandoned entries, much like the hold for soft-deleted tables in [5544]. It also changes concurrent creates of the same name to fail at staging time rather than at commit time, which I believe is a net improvement. I'm not suggesting we do all this for Phase 1, only that the state model should leave room for 3 values: STAGED, LIVE, DELETED. WDYT? Thanks, Alex [5544]: https://github.com/apache/polaris/pull/5544 On Fri, Nov 13, 2026 at 5:54 PM Alex Dutra <[email protected]> wrote: > > Hi Prithvi, hi all, > > Two points from the remote signing work that relate to this proposal. > > 1. Spark's DROP TABLE ... PURGE is a drop without purge. > > Spark doesn't ask the catalog to purge: > > - It loads the table and calls the catalog's drop with purge=false. > - It then deletes the table's files itself, through the table's FileIO. > > With Spark, the catalog never sees a purge. On a catalog with > soft-delete enabled, a Spark DROP TABLE ... PURGE would: > > - put the table on hold, with its name reserved, even though the user > asked for a purge; > - delete its files right away from the client. > > The hold would then keep a table whose files are gone, which matters > once recovery is added. With vended credentials, Polaris can't see or > prevent those deletes. This might be worth documenting. > > 2. Soft-delete would help remote signing. > > I'm reviving S3 remote signing. Polaris authorizes each sign request > against the table, so it does see those deletes. Today, after a drop, > the table and its grants are gone. My current approach records the > principal's DROP_TABLE_WITH_PURGE decision in the remote signing token > when the table is loaded. After the drop, it then allows only reads > and deletes within the table's locations. > > In [5544], a soft-deleted table keeps its row, id and grants. So for > those tables Polaris could: > > - authorize the deletes against the table itself when they happen, > instead of relying on a decision recorded at load time; > - choose a policy for the hold: either allow the deletes, since the > client asked for a purge, or refuse them, so the table stays > recoverable. Only remote signing makes the second option possible. > > Two questions: > > - Which policy makes sense for deletes during the hold? > - The resolver hides dropped entities. Would an option to resolve a > soft-deleted table, for this purpose only, be acceptable? > > Neither point should block Phase 1. I just want to make sure the two > features fit together. > > Thanks, > Alex > > [5544]: https://github.com/apache/polaris/pull/5544 > > > On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev > <[email protected]> wrote: > > > > Hi Prithvi and all, > > > > I did another review round in GH. > > > > Re-posting here the key comment: I'm very hesitant about > > expiring soft-deleted entities during REST requests against other entities. > > I think the resulting behaviours are too "magical" from the end user's > > perspective. > > > > I believe the expiry use case belongs more naturally in the Admin Tool > > given the current codebase. Eventually it could also be handled by the > > delegation service [3990]. > > > > WDYT? > > > > Thanks, > > Dmitri. > > [3990] https://github.com/apache/polaris/pull/3990 > > > > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Quick follow-up on the cleanup gating for Phase 1: > > > holding the identifier until the entity cleanup and its > > > child tasks finish, then releasing it. > > > > > > While implementing this, I found that the coupling is > > > in the metastore manager. dropEntityIfExists removes > > > the entity (freeing the name) and schedules the cleanup > > > task in the same transaction, by design. > > > > > > To keep the name reserved until cleanup completes, I'd > > > need to separate these operations: keep the dropped > > > entity, schedule cleanup, and remove the entity only > > > after the cleanup task and its child tasks have finished. > > > > > > We don't currently have a way to detect when a task > > > tree has completed. Tasks are fire-and-forget and > > > self-remove on success, so I'd also need to add a small > > > completion check. > > > > > > This touches PolarisMetaStoreManager, which has both > > > JDBC (atomic and transactional) and NoSQL > > > implementations. I wanted to check the direction before > > > coding, in line with our interface-change guidance. > > > > > > Does keeping the entity reserved and gating its removal > > > on cleanup completion sound right, or would you prefer > > > a different approach, such as a dedicated purge task > > > that handles the final entity removal? > > > > > > Thanks, > > > Prithvi > > > > > > > > > >
