Hi EJ, (replying separately to each item)
Very good point about "business rules" (although I personally do not like the term at all :) ) I believe this area of improvement is actually independent of the bigger Persistence discussion. If there's interest in refactoring that code to improve logic sharing, I think that work can proceed in parallel. Cheers, Dmitri. On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <[email protected]> wrote: > Thanks for the recap, Dmitri. I'd like to follow up on the three points I > raised during the sync. We didn't have much time to discuss them, so it > would help to confirm these before choosing an SPI design. > > 1. Which backend assumptions should we design for? > > Within one Polaris server instance, do we assume entities, grants and > secrets all use the same database backend, or must we support mixing > backends across them? > Must every supported persistence implementation provide transactional > guarantees? Does the database itself need to provide transactions, or can > Polaris build that capability on top of simpler database operations? > > I'm happy to work with a homogeneous assumption if that's the intended > scope. Existing NoSQL illustrates the second question: it can publish > changes to multiple objects under one reference using a single conditional > pointer update. > > 2. How do we protect the data used by validation? > > Your storage-configuration and location-overlap examples make the problem > concrete. Consider these races: > > A table creation passes validation against the catalog's allowed locations, > then another request changes that configuration before the table is > committed. > Two table-creation requests each check for overlapping locations, both see > no overlap, and then create tables at conflicting locations. > > Checking only the record being written cannot cover these dependencies. The > first also depends on the catalog configuration. The second depends on the > absence of another overlapping table. > > Could the persistence layer protect these reads together with the writes, > using backend-internal versions/tokens or database isolation? Shared code > would still need to identify the required reads and checks. Could we keep > the mechanism for protecting them inside each backend? > > 3. Business rules are duplicated across backend implementations > > Today, the TreeMap > < > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > and JDBC > < > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > implementations each load a secret, check its principal, apply the > rotation/reset steps and write it back. > > Changing these business rules therefore requires keeping multiple > implementations in sync. A fix applied to one backend can be missed in > another, causing the same Polaris operation to behave differently depending > on the database. > > My proposal is to implement each business workflow once, against a common > set of storage operations with defined consistency and failure semantics. > The shared workflow should not need to know whether those operations are > implemented through JDBC transactions, NoSQL reference CAS or another > mechanism. Each backend would implement the same storage contract, keeping > its execution details internal. A new database backend could then reuse the > existing business workflows. > > Do these seem like reasonable starting points? We can then evaluate the SPI > against the original failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Side note: I just came across [5541], which shows a case where a failed > > request leaves persisted side effects. > > > > This can be seen as a coding mistake, of course. However, this is about > > executing one request as a single, atomic unit. This is something that > > probably needs foundational support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri. > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Here's a recap of the Community Sync discussion as I remember it. I'm > > sure > > > my recollection is not complete, so please add your thoughts to this > > thread. > > > > > > * Polaris traditionally relies on entity version number to ensure > > > consistency of changes. > > > > > > This works well for internal RBAC grants, which also have version > numbers > > > cross-referenced at the Persistence layer. > > > > > > I do not think this works in more general cases like validating > locations > > > wrt catalog-level storage configuration. > > > > > > Another difficult case involves location overlaps between concurrent > > table > > > creations. > > > > > > * We talked about how Persistence calls related to RDBMS transactions > (in > > > the JDBC persistence case). > > > > > > There were some concerns raised about whether we need to bind > Persistence > > > API to transactions explicitly. > > > > > > I think it is a valid concern. At the same time, the Persistence API > > > should be clear about consistency guarantees across all backend > > > implementations. I think this needs more attention now that the matter > of > > > milti-entioty changes became prominent in [5035]. > > > > > > I personally believe that RDBMS-based Persistence in Polaris has to use > > > Serializable transaction isolation. Consequently, the core code needs > to > > > provide mechanisms to allow the Persistence implementation to properly > > > connect reads and writes from API requests to JDBC transactions... > > whether > > > we use "transation" as an explicit terms in the SPI or not. > > > > > > Note that currently, the same API request performs reads and writes in > > > _separate_ RDBMS transactions (even separate connections). > > > > > > * We did not discuss Persistence SPI implications in the call, but > > > currently each Persistence call is assumed to be one distinct and > atomic > > > change (please correct me if I'm wrong). > > > > > > Consequently, supporting new use cases involves introducing new SPI > > > methods with many parameters. This can be seen in PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > > Ideally, I think the SPI should be simplified to deal with multi-entity > > > and single-entity changes in a coherent manner to avoid any ambiguity > > about > > > which method the caller should use. > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > >> Heads up: This discussion is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >> Interested parties, please plan to attend, if possible :) > > >> > > >> Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <[email protected]> > > >> wrote: > > >> > > >>> Hi JB, > > >>> > > >>> I've added an agenda item to the next community sync call for this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you mean a real GH discussion or dev email (this > > >>> thread)? Just double checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > > >>> Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, > > >>>> > > >>>> I agree on the need to align on these points, though I'm not > entirely > > >>>> sure a dedicated meeting is necessary. Let's start by using some > time > > >>>> during the next community meeting to discuss it. > > >>>> > > >>>> In the meantime, what if we outline the key points in a GitHub > > >>>> Discussion first? We can then use that as a reference during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > [email protected] > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > > >>>> > > > >>>> > My impression from this thread is that it might be time for a call > > to > > >>>> > discuss all the related issues and try to arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > > > >>>> > I know meetings are not ideal, but in this case it might be > > >>>> beneficial as a > > >>>> > means for achieving a common understanding of the set of problems > > and > > >>>> > priorities related to this thread. A dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my POV. > > >>>> > > > >>>> > Allocating a time slot in the next community meeting might be an > > >>>> option, > > >>>> > although I think we might need the full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S < > > >>>> [email protected]> > > >>>> > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, > following > > >>>> the > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written manager-level consistency contract in > > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > - CAS-aware EntityMutation / GrantMutation records. > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>> backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > >>>> > > - commitChangeSet implementations for the in-memory TreeMap > > >>>> backend and > > >>>> > > JDBC. > > >>>> > > - AtomicOperationMetaStoreManager and > > >>>> TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now build one change set per operation, > > >>>> falling back > > >>>> > > to > > >>>> > > individual operations when the backend does not support mixed > > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to > > use > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI > > primitive > > >>>> can be > > >>>> > > reviewed before the larger operation migrations land. > > >>>> > > > > >>>> > > Could you please take a look? In particular, I would like to > know > > >>>> whether > > >>>> > > the contract captures the consensus so far, or if it needs to > > >>>> address retry > > >>>> > > / external-work coordination before we merge this foundation. > > >>>> > > > > >>>> > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API > > request) > > >>>> > > consists > > >>>> > > > of many distinct phases. The database / persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > > > these phases. STS (as an example) is another. My point about > > >>>> transactions > > >>>> > > > in JDBC Persistence applies to the former (database changes) > > >>>> phase - one > > >>>> > > > retry attempt there should ideally involve exactly one > > >>>> transaction. > > >>>> > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various external systems depending on outcomes from previous > > >>>> request > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree that validation based on database state should be > > >>>> repeated > > >>>> > > > from scratch if we retry the database changes. > > >>>> > > > > > >>>> > > > However, internal RBAC authorization naturally has to happen > > >>>> inside the > > >>>> > > > database update phase (read-check-write). IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency guarantees, which were discussed during > > early > > >>>> > > project > > >>>> > > > months (but never got written down in full clarity, > > >>>> unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > > will be there for other Authorizers too and that can involve > > >>>> external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not sure how quick it can be in practice. > > >>>> There's > > >>>> > > > certainly potential for delays in some (perhaps infrequent) > > cases. > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > > > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <[email protected]> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical operation needs a consistent view > of > > >>>> the whole > > >>>> > > > > backend state. > > >>>> > > > > What worries me is treating one long JDBC transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > > that operation and then retrying the complete request if the > > >>>> commit > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can also write metadata to object storage or > call > > >>>> STS. > > >>>> > > > > Each system can leave us with an outcome whose certainty we > > >>>> cannot > > >>>> > > know. > > >>>> > > > > The database may have committed before the connection was > > lost, > > >>>> an > > >>>> > > > > object-store write may have succeeded before a timeout, or > STS > > >>>> may have > > >>>> > > > > issued credentials before its response was lost. > > >>>> > > > > A database rollback cannot undo any of those effects, and > > >>>> retrying the > > >>>> > > > > whole request may repeat them. > > >>>> > > > > > > >>>> > > > > So I think the contract has to separate the overall > operation > > >>>> from each > > >>>> > > > > attempt to commit it. > > >>>> > > > > Each database or backend commit attempt should stay short. > > >>>> > > > > When we retry, validation and authorization need to use the > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we need clear rules for which external work > > >>>> can be > > >>>> > > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual state could be useful for carrying that > state. > > >>>> > > > > But the JDBC connection and transaction should be an > > >>>> implementation > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> [email protected] > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> > > > > > However, I am not sure it is sufficient for all the use > > cases > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More specifically, the state read by validation code > > is > > >>>> not > > >>>> > > > > > necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > > > > > * Unchanged entities might be considered by validation, > but > > >>>> changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel request. > > >>>> > > > > > * Even for changed entities the state read by validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > the state being altered. Some validation code talks to the > > >>>> MetaStore > > >>>> > > > > > directly, outside of the data produced by the Resolver. > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > > consistency between these reads and subsequent writes. > > >>>> > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>> > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a transaction might be long to accommodate > > calls > > >>>> made by > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > alternative is for the JDBC Persistence impl. to > "manually" > > >>>> track all > > >>>> > > > > reads > > >>>> > > > > > under the same request and redo them in the small > > transaction > > >>>> that > > >>>> > > > > persists > > >>>> > > > > > the writes. This will also add RDBMS overhead and require > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > > Tracking the request-wide transaction can be done only for > > >>>> JDBC > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > > will use other mechanisms to ensure read/write > consistency. > > >>>> > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each request establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > - In NoSQL the Data Context tracks one or more reference > > >>>> hashes > > >>>> > > > > > * All Persistence access in the same request goes through > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > >>>> > > > > > * All Persistence changes are committed once at the end of > > the > > >>>> > > request > > >>>> > > > > > - Not all changes have to be globally atomic. For > example, > > >>>> changes > > >>>> > > in > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect to > > >>>> each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A commit can fail in two main ways: > > >>>> > > > > > - A retriable failure like an optimistic lock error or > Tx > > >>>> > > > > serializability > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error (e.g. entity not found) > > >>>> > > > > > * On a retriable error the whole request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted > > >>>> > > > > > by a client) a few times (configurable timeout). > > >>>> > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > > [email protected] > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > I think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > If the atomicity guarantee lives only in > BasePersistence, > > >>>> then the > > >>>> > > > > > > manager contract can't tell a caller whether the state > it > > >>>> read for > > >>>> > > > > > > validation/authorization/credential-vending is the same > > >>>> state that > > >>>> > > > > > > eventually commits. That's the actual bug class behind > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single operation being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > > being silent about it. Every operation-specific fix > (like > > >>>> #4939, > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this question locally and it keeps > > >>>> recurring. > > >>>> > > So > > >>>> > > > > > > the deliverable should start with a written consistency > > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It would tie the logical change set to a > > DB > > >>>> > > > > > > transaction spanning the REST request. It means: > > >>>> > > > > > > - it holds a durable transaction open across slow > external > > >>>> work > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the transactional manager isn't a portable > > >>>> target. It's > > >>>> > > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > > > > > I think Privthi's approach is right. A backend-agnostic > > >>>> change-set > > >>>> > > > > > > primitive with a documented fallback is the correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat: without carrying the original entitiy (not > > just > > >>>> the new > > >>>> > > > > > > one) the change set can't express optimistic > concurrency. > > >>>> > > > > > > > > >>>> > > > > > > I propose the following multi-steps approach: > > >>>> > > > > > > 1. We write the manager-level consistency contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > > should include the explicit statement that a logical > > change > > >>>> set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB transaction. > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > > > > > > first-class in EntityMutation before merging the SPI > > >>>> > > > > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>> > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi all, > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > > > > > > If it is a property of the lower-level BasePersistence > > >>>> contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the > > general > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > >>>> > > > > > > > tell callers whether the state used for validation, > > >>>> > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential vending is consistent with the change that > > >>>> eventually > > >>>> > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual cases, while leaving the same contract > > >>>> question to > > >>>> > > > > recur > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>> > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical change set as a > > >>>> database > > >>>> > > > > > > transaction > > >>>> > > > > > > > around an entire REST request. > > >>>> > > > > > > > That would tie the contract to the backend and > database, > > >>>> and > > >>>> > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > > > > > > durable attempt open across slow external work. > > >>>> > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > > >>>> > > > > > > > do we first need to revisit the boundary and > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov < > > >>>> > > > > [email protected] > > >>>> > > > > > > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> > > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as defined in javadoc does apply only to > > >>>> > > BasePersistence > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates > > to > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>> > > > > > > > > Note that all non-test Persistence implementations > in > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > > > > > > > > extend those classes (which is probably why I was > > >>>> confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > > > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI > > >>>> > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>> > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this principle, it will create a conceptual > difficulty > > >>>> at call > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about > > >>>> consistency > > >>>> > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > behaviours in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe we need to address that as part of this > > >>>> discussion. > > >>>> > > > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < > > >>>> [email protected] > > >>>> > > > > > >>>> > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface > > >>>> > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't think we've ever state each method to be > > >>>> atomic. We did > > >>>> > > > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > [email protected]> > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled > > more > > >>>> > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > > > I'm not sure that simply moving to > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > > means one atomic change [1]. The "transactional" > > >>>> MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple object > > >>>> parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be a > > >>>> sound > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>> > > > > > > > > > > backend type. More importantly, adding multi-arg > > >>>> change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > > > > > > > > won't > > >>>> > > > > > > > > > > address the problem of reads being consistent > with > > >>>> writes, > > >>>> > > > > > because > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still be independent regarding > > the > > >>>> data > > >>>> > > > stored > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a "change > > set" > > >>>> or > > >>>> > > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept to core Persistence and associate each > > REST > > >>>> API > > >>>> > > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>> > > > > > > > > > > such change set, which will be committed (or > > rolled > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end of > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each change set will naturally be > associated > > >>>> with an > > >>>> > > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > > >>>> > > > > > > > > > > associated with one CAS operation on the > > underlying > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > [1] > > >>>> > > > > > > > > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > > > >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > >>>> > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > > [email protected] > > >>>> > > > > > > > >>>> > > > > > > wrote: > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were > already > > >>>> > > recognized > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At > that > > >>>> time, we > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > >>>> > > > > > > > > > > > most of the issues mentioned here can already > be > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in a > > >>>> transaction > > >>>> > > in > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > > > > > > > > > > > involves reading and creating multiple > objects, > > >>>> is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the > > required > > >>>> > > > > consistency > > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > > >>>> > > > > > > > > > > > specific operations, similar to what > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > > > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as > > >>>> JDBC) to use > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>> > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > > > > with > > >>>> > > > > > > > > > > > transactional semantic mismatches across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > > > > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we would likely avoid using > > JDBC's > > >>>> > > > > > > > > > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > > > > > > > > > > single row updates, which adds additional > > >>>> overhead and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > > without > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>> Bourlatchkov < > > >>>> > > > > > > > > [email protected]> > > >>>> > > > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>> > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > > > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > It looks like people are starting to > encounter > > >>>> > > > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > > They > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > > incremental fixes that can work. However, I > > >>>> believe it > > >>>> > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > > > Polaris community to review and improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > > these > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and validates current catalog state, then > > >>>> commits a > > >>>> > > > > change > > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes during renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > > >>>> > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > > > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > > > > > > > > > > > > * Supporting atomic changes across multiple > > >>>> similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering > of > > >>>> list > > >>>> > > > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > > [4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side retries for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4831] > > >>>> https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>> https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>> https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > > > > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >>> > > >
