Hi Alex > - The resolver hides dropped entities. [...]
Thanks for flagging this. I tend to think that the Resolver should not hide anything because it is a critical piece in ensuing catalog consistency with respect to the Entity Cache. If the Resolver is a convenient place to enforce soft-deleted states, perhaps we could do that during the "get" calls, while still allowing the soft-deleted entities to be resolved normally. The request signer could then use a different getter method returning all resolved entities. > - Which policy makes sense for deletes during the hold? I think we should block file removal for soft-deleted entities where we can. It seems possible with request signing. Cheers, Dmitri. On Fri, Nov 13, 2026 at 11:55 AM Alex Dutra via dev <[email protected]> wrote: > Hi Prithvi, hi all, > > Two points from the remote signing work that relate to this proposal. > > 1. Spark's DROP TABLE ... PURGE is a drop without purge. > > Spark doesn't ask the catalog to purge: > > - It loads the table and calls the catalog's drop with purge=false. > - It then deletes the table's files itself, through the table's FileIO. > > With Spark, the catalog never sees a purge. On a catalog with > soft-delete enabled, a Spark DROP TABLE ... PURGE would: > > - put the table on hold, with its name reserved, even though the user > asked for a purge; > - delete its files right away from the client. > > The hold would then keep a table whose files are gone, which matters > once recovery is added. With vended credentials, Polaris can't see or > prevent those deletes. This might be worth documenting. > > 2. Soft-delete would help remote signing. > > I'm reviving S3 remote signing. Polaris authorizes each sign request > against the table, so it does see those deletes. Today, after a drop, > the table and its grants are gone. My current approach records the > principal's DROP_TABLE_WITH_PURGE decision in the remote signing token > when the table is loaded. After the drop, it then allows only reads > and deletes within the table's locations. > > In [5544], a soft-deleted table keeps its row, id and grants. So for > those tables Polaris could: > > - authorize the deletes against the table itself when they happen, > instead of relying on a decision recorded at load time; > - choose a policy for the hold: either allow the deletes, since the > client asked for a purge, or refuse them, so the table stays > recoverable. Only remote signing makes the second option possible. > > Two questions: > > - Which policy makes sense for deletes during the hold? > - The resolver hides dropped entities. Would an option to resolve a > soft-deleted table, for this purpose only, be acceptable? > > Neither point should block Phase 1. I just want to make sure the two > features fit together. > > Thanks, > Alex > > [5544]: https://github.com/apache/polaris/pull/5544 > > > On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev > <[email protected]> wrote: > > > > Hi Prithvi and all, > > > > I did another review round in GH. > > > > Re-posting here the key comment: I'm very hesitant about > > expiring soft-deleted entities during REST requests against other > entities. > > I think the resulting behaviours are too "magical" from the end user's > > perspective. > > > > I believe the expiry use case belongs more naturally in the Admin Tool > > given the current codebase. Eventually it could also be handled by the > > delegation service [3990]. > > > > WDYT? > > > > Thanks, > > Dmitri. > > [3990] https://github.com/apache/polaris/pull/3990 > > > > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Quick follow-up on the cleanup gating for Phase 1: > > > holding the identifier until the entity cleanup and its > > > child tasks finish, then releasing it. > > > > > > While implementing this, I found that the coupling is > > > in the metastore manager. dropEntityIfExists removes > > > the entity (freeing the name) and schedules the cleanup > > > task in the same transaction, by design. > > > > > > To keep the name reserved until cleanup completes, I'd > > > need to separate these operations: keep the dropped > > > entity, schedule cleanup, and remove the entity only > > > after the cleanup task and its child tasks have finished. > > > > > > We don't currently have a way to detect when a task > > > tree has completed. Tasks are fire-and-forget and > > > self-remove on success, so I'd also need to add a small > > > completion check. > > > > > > This touches PolarisMetaStoreManager, which has both > > > JDBC (atomic and transactional) and NoSQL > > > implementations. I wanted to check the direction before > > > coding, in line with our interface-change guidance. > > > > > > Does keeping the entity reserved and gating its removal > > > on cleanup completion sound right, or would you prefer > > > a different approach, such as a dedicated purge task > > > that handles the final entity removal? > > > > > > Thanks, > > > Prithvi > > > > > > > > > > >
