Hi EJ, > 1. Which backend assumptions should we design for? [...]
I know I used the term "transaction" a lot during the call, but the real challenge is about establishing an SPI with clear contracts that could be implemented by JDBC and NoSQL persistence using the database features natural to each model. I am fairly confident that a JDBC impl. is possible with Serializable transaction. We should be able to solve all the problems raised in this thread. However, that will make in-memory caching impossible because the database needs to see all the reads (and writes) in a Tx to be able to ensure serializable guarantees. If a cache prevents a read from hitting the database, that read (e.g. storage config) will not be a factor in serializability. This connects to my earlier question about the in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are different approaches possible with JDBC? Probably. What is involved? This depends on the SPI contract, I think. > 2. How do we protect the data used by validation? [...] The point mentioned by Robert previously IIRC, is that we need to ground all runtime decisions to a particular state of the catalog. If at commit time the state of the catalog in the database shifts from what was used during validation, the request has to restart from scratch. Current JDBC Persistence does this only for relationships between entities and grants and between versions of the same entity. Relationships between table locations and storage config are not actualized for the purpose of state consistency tracking. >From my POV we need to begin by formulating this Persistence state in the Persistence SPI terms. Then, think how we can implement that for JDBC. NoSQL Persistence already has a per-catalog state (not granular), so wiring the new SPI to NoSQL should not be too hard, I hope. The state in the SPI does not have to be global to the catalog (as in NoSQL), but it needs to be able to express the idea of which elements were engaged during request processing so serve as input to Persistence impl. for cosistency guarantees. If Persistence caches data in memory (for example), it will have to do so in a way that does not violate consistency expectations. >From my POV we could begin by defining a "Persistence Session" concept. Each request is 1:1 with a session. All reads and writes go through the session. The next step would be the JDBC impl., which probably needs some brainstorming... but I wonder if we could reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <[email protected]> wrote: > Thanks for the recap, Dmitri. I'd like to follow up on the three points I > raised during the sync. We didn't have much time to discuss them, so it > would help to confirm these before choosing an SPI design. > > 1. Which backend assumptions should we design for? > > Within one Polaris server instance, do we assume entities, grants and > secrets all use the same database backend, or must we support mixing > backends across them? > Must every supported persistence implementation provide transactional > guarantees? Does the database itself need to provide transactions, or can > Polaris build that capability on top of simpler database operations? > > I'm happy to work with a homogeneous assumption if that's the intended > scope. Existing NoSQL illustrates the second question: it can publish > changes to multiple objects under one reference using a single conditional > pointer update. > > 2. How do we protect the data used by validation? > > Your storage-configuration and location-overlap examples make the problem > concrete. Consider these races: > > A table creation passes validation against the catalog's allowed locations, > then another request changes that configuration before the table is > committed. > Two table-creation requests each check for overlapping locations, both see > no overlap, and then create tables at conflicting locations. > > Checking only the record being written cannot cover these dependencies. The > first also depends on the catalog configuration. The second depends on the > absence of another overlapping table. > > Could the persistence layer protect these reads together with the writes, > using backend-internal versions/tokens or database isolation? Shared code > would still need to identify the required reads and checks. Could we keep > the mechanism for protecting them inside each backend? > > 3. Business rules are duplicated across backend implementations > > Today, the TreeMap > < > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > and JDBC > < > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > implementations each load a secret, check its principal, apply the > rotation/reset steps and write it back. > > Changing these business rules therefore requires keeping multiple > implementations in sync. A fix applied to one backend can be missed in > another, causing the same Polaris operation to behave differently depending > on the database. > > My proposal is to implement each business workflow once, against a common > set of storage operations with defined consistency and failure semantics. > The shared workflow should not need to know whether those operations are > implemented through JDBC transactions, NoSQL reference CAS or another > mechanism. Each backend would implement the same storage contract, keeping > its execution details internal. A new database backend could then reuse the > existing business workflows. > > Do these seem like reasonable starting points? We can then evaluate the SPI > against the original failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Side note: I just came across [5541], which shows a case where a failed > > request leaves persisted side effects. > > > > This can be seen as a coding mistake, of course. However, this is about > > executing one request as a single, atomic unit. This is something that > > probably needs foundational support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri. > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Here's a recap of the Community Sync discussion as I remember it. I'm > > sure > > > my recollection is not complete, so please add your thoughts to this > > thread. > > > > > > * Polaris traditionally relies on entity version number to ensure > > > consistency of changes. > > > > > > This works well for internal RBAC grants, which also have version > numbers > > > cross-referenced at the Persistence layer. > > > > > > I do not think this works in more general cases like validating > locations > > > wrt catalog-level storage configuration. > > > > > > Another difficult case involves location overlaps between concurrent > > table > > > creations. > > > > > > * We talked about how Persistence calls related to RDBMS transactions > (in > > > the JDBC persistence case). > > > > > > There were some concerns raised about whether we need to bind > Persistence > > > API to transactions explicitly. > > > > > > I think it is a valid concern. At the same time, the Persistence API > > > should be clear about consistency guarantees across all backend > > > implementations. I think this needs more attention now that the matter > of > > > milti-entioty changes became prominent in [5035]. > > > > > > I personally believe that RDBMS-based Persistence in Polaris has to use > > > Serializable transaction isolation. Consequently, the core code needs > to > > > provide mechanisms to allow the Persistence implementation to properly > > > connect reads and writes from API requests to JDBC transactions... > > whether > > > we use "transation" as an explicit terms in the SPI or not. > > > > > > Note that currently, the same API request performs reads and writes in > > > _separate_ RDBMS transactions (even separate connections). > > > > > > * We did not discuss Persistence SPI implications in the call, but > > > currently each Persistence call is assumed to be one distinct and > atomic > > > change (please correct me if I'm wrong). > > > > > > Consequently, supporting new use cases involves introducing new SPI > > > methods with many parameters. This can be seen in PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > > Ideally, I think the SPI should be simplified to deal with multi-entity > > > and single-entity changes in a coherent manner to avoid any ambiguity > > about > > > which method the caller should use. > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > >> Heads up: This discussion is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >> Interested parties, please plan to attend, if possible :) > > >> > > >> Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <[email protected]> > > >> wrote: > > >> > > >>> Hi JB, > > >>> > > >>> I've added an agenda item to the next community sync call for this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you mean a real GH discussion or dev email (this > > >>> thread)? Just double checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > > >>> Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, > > >>>> > > >>>> I agree on the need to align on these points, though I'm not > entirely > > >>>> sure a dedicated meeting is necessary. Let's start by using some > time > > >>>> during the next community meeting to discuss it. > > >>>> > > >>>> In the meantime, what if we outline the key points in a GitHub > > >>>> Discussion first? We can then use that as a reference during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > [email protected] > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > > >>>> > > > >>>> > My impression from this thread is that it might be time for a call > > to > > >>>> > discuss all the related issues and try to arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > > > >>>> > I know meetings are not ideal, but in this case it might be > > >>>> beneficial as a > > >>>> > means for achieving a common understanding of the set of problems > > and > > >>>> > priorities related to this thread. A dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my POV. > > >>>> > > > >>>> > Allocating a time slot in the next community meeting might be an > > >>>> option, > > >>>> > although I think we might need the full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S < > > >>>> [email protected]> > > >>>> > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, > following > > >>>> the > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written manager-level consistency contract in > > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > - CAS-aware EntityMutation / GrantMutation records. > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>> backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > >>>> > > - commitChangeSet implementations for the in-memory TreeMap > > >>>> backend and > > >>>> > > JDBC. > > >>>> > > - AtomicOperationMetaStoreManager and > > >>>> TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now build one change set per operation, > > >>>> falling back > > >>>> > > to > > >>>> > > individual operations when the backend does not support mixed > > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity to > > use > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI > > primitive > > >>>> can be > > >>>> > > reviewed before the larger operation migrations land. > > >>>> > > > > >>>> > > Could you please take a look? In particular, I would like to > know > > >>>> whether > > >>>> > > the contract captures the consensus so far, or if it needs to > > >>>> address retry > > >>>> > > / external-work coordination before we merge this foundation. > > >>>> > > > > >>>> > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API > > request) > > >>>> > > consists > > >>>> > > > of many distinct phases. The database / persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > > > these phases. STS (as an example) is another. My point about > > >>>> transactions > > >>>> > > > in JDBC Persistence applies to the former (database changes) > > >>>> phase - one > > >>>> > > > retry attempt there should ideally involve exactly one > > >>>> transaction. > > >>>> > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various external systems depending on outcomes from previous > > >>>> request > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree that validation based on database state should be > > >>>> repeated > > >>>> > > > from scratch if we retry the database changes. > > >>>> > > > > > >>>> > > > However, internal RBAC authorization naturally has to happen > > >>>> inside the > > >>>> > > > database update phase (read-check-write). IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency guarantees, which were discussed during > > early > > >>>> > > project > > >>>> > > > months (but never got written down in full clarity, > > >>>> unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > > will be there for other Authorizers too and that can involve > > >>>> external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not sure how quick it can be in practice. > > >>>> There's > > >>>> > > > certainly potential for delays in some (perhaps infrequent) > > cases. > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > > > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <[email protected]> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical operation needs a consistent view > of > > >>>> the whole > > >>>> > > > > backend state. > > >>>> > > > > What worries me is treating one long JDBC transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > > that operation and then retrying the complete request if the > > >>>> commit > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can also write metadata to object storage or > call > > >>>> STS. > > >>>> > > > > Each system can leave us with an outcome whose certainty we > > >>>> cannot > > >>>> > > know. > > >>>> > > > > The database may have committed before the connection was > > lost, > > >>>> an > > >>>> > > > > object-store write may have succeeded before a timeout, or > STS > > >>>> may have > > >>>> > > > > issued credentials before its response was lost. > > >>>> > > > > A database rollback cannot undo any of those effects, and > > >>>> retrying the > > >>>> > > > > whole request may repeat them. > > >>>> > > > > > > >>>> > > > > So I think the contract has to separate the overall > operation > > >>>> from each > > >>>> > > > > attempt to commit it. > > >>>> > > > > Each database or backend commit attempt should stay short. > > >>>> > > > > When we retry, validation and authorization need to use the > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we need clear rules for which external work > > >>>> can be > > >>>> > > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual state could be useful for carrying that > state. > > >>>> > > > > But the JDBC connection and transaction should be an > > >>>> implementation > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> [email protected] > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> > > > > > However, I am not sure it is sufficient for all the use > > cases > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More specifically, the state read by validation code > > is > > >>>> not > > >>>> > > > > > necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > > > > > * Unchanged entities might be considered by validation, > but > > >>>> changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel request. > > >>>> > > > > > * Even for changed entities the state read by validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > the state being altered. Some validation code talks to the > > >>>> MetaStore > > >>>> > > > > > directly, outside of the data produced by the Resolver. > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > > consistency between these reads and subsequent writes. > > >>>> > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>> > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a transaction might be long to accommodate > > calls > > >>>> made by > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > alternative is for the JDBC Persistence impl. to > "manually" > > >>>> track all > > >>>> > > > > reads > > >>>> > > > > > under the same request and redo them in the small > > transaction > > >>>> that > > >>>> > > > > persists > > >>>> > > > > > the writes. This will also add RDBMS overhead and require > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > > Tracking the request-wide transaction can be done only for > > >>>> JDBC > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > > will use other mechanisms to ensure read/write > consistency. > > >>>> > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each request establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > - In NoSQL the Data Context tracks one or more reference > > >>>> hashes > > >>>> > > > > > * All Persistence access in the same request goes through > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > >>>> > > > > > * All Persistence changes are committed once at the end of > > the > > >>>> > > request > > >>>> > > > > > - Not all changes have to be globally atomic. For > example, > > >>>> changes > > >>>> > > in > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect to > > >>>> each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A commit can fail in two main ways: > > >>>> > > > > > - A retriable failure like an optimistic lock error or > Tx > > >>>> > > > > serializability > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error (e.g. entity not found) > > >>>> > > > > > * On a retriable error the whole request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted > > >>>> > > > > > by a client) a few times (configurable timeout). > > >>>> > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > > [email protected] > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > I think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > If the atomicity guarantee lives only in > BasePersistence, > > >>>> then the > > >>>> > > > > > > manager contract can't tell a caller whether the state > it > > >>>> read for > > >>>> > > > > > > validation/authorization/credential-vending is the same > > >>>> state that > > >>>> > > > > > > eventually commits. That's the actual bug class behind > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single operation being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > > being silent about it. Every operation-specific fix > (like > > >>>> #4939, > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this question locally and it keeps > > >>>> recurring. > > >>>> > > So > > >>>> > > > > > > the deliverable should start with a written consistency > > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It would tie the logical change set to a > > DB > > >>>> > > > > > > transaction spanning the REST request. It means: > > >>>> > > > > > > - it holds a durable transaction open across slow > external > > >>>> work > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the transactional manager isn't a portable > > >>>> target. It's > > >>>> > > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > > > > > I think Privthi's approach is right. A backend-agnostic > > >>>> change-set > > >>>> > > > > > > primitive with a documented fallback is the correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat: without carrying the original entitiy (not > > just > > >>>> the new > > >>>> > > > > > > one) the change set can't express optimistic > concurrency. > > >>>> > > > > > > > > >>>> > > > > > > I propose the following multi-steps approach: > > >>>> > > > > > > 1. We write the manager-level consistency contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > > should include the explicit statement that a logical > > change > > >>>> set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB transaction. > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > > > > > > first-class in EntityMutation before merging the SPI > > >>>> > > > > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>> > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> [email protected]> > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi all, > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > > > > > > If it is a property of the lower-level BasePersistence > > >>>> contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the > > general > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > >>>> > > > > > > > tell callers whether the state used for validation, > > >>>> > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential vending is consistent with the change that > > >>>> eventually > > >>>> > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual cases, while leaving the same contract > > >>>> question to > > >>>> > > > > recur > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>> > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical change set as a > > >>>> database > > >>>> > > > > > > transaction > > >>>> > > > > > > > around an entire REST request. > > >>>> > > > > > > > That would tie the contract to the backend and > database, > > >>>> and > > >>>> > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > > > > > > durable attempt open across slow external work. > > >>>> > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > > >>>> > > > > > > > do we first need to revisit the boundary and > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov < > > >>>> > > > > [email protected] > > >>>> > > > > > > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> > > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as defined in javadoc does apply only to > > >>>> > > BasePersistence > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates > > to > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>> > > > > > > > > Note that all non-test Persistence implementations > in > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > > > > > > > > extend those classes (which is probably why I was > > >>>> confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > > > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI > > >>>> > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>> > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this principle, it will create a conceptual > difficulty > > >>>> at call > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about > > >>>> consistency > > >>>> > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > behaviours in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe we need to address that as part of this > > >>>> discussion. > > >>>> > > > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < > > >>>> [email protected] > > >>>> > > > > > >>>> > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface > > >>>> > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't think we've ever state each method to be > > >>>> atomic. We did > > >>>> > > > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > [email protected]> > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled > > more > > >>>> > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > > > I'm not sure that simply moving to > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > > means one atomic change [1]. The "transactional" > > >>>> MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple object > > >>>> parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be a > > >>>> sound > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>> > > > > > > > > > > backend type. More importantly, adding multi-arg > > >>>> change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > > > > > > > > won't > > >>>> > > > > > > > > > > address the problem of reads being consistent > with > > >>>> writes, > > >>>> > > > > > because > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still be independent regarding > > the > > >>>> data > > >>>> > > > stored > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a "change > > set" > > >>>> or > > >>>> > > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept to core Persistence and associate each > > REST > > >>>> API > > >>>> > > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>> > > > > > > > > > > such change set, which will be committed (or > > rolled > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end of > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each change set will naturally be > associated > > >>>> with an > > >>>> > > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > > >>>> > > > > > > > > > > associated with one CAS operation on the > > underlying > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > [1] > > >>>> > > > > > > > > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > > > >>>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > >>>> > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > > [email protected] > > >>>> > > > > > > > >>>> > > > > > > wrote: > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were > already > > >>>> > > recognized > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At > that > > >>>> time, we > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > >>>> > > > > > > > > > > > most of the issues mentioned here can already > be > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in a > > >>>> transaction > > >>>> > > in > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > > > > > > > > > > > involves reading and creating multiple > objects, > > >>>> is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the > > required > > >>>> > > > > consistency > > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > > >>>> > > > > > > > > > > > specific operations, similar to what > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > > > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as > > >>>> JDBC) to use > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>> > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > > > > with > > >>>> > > > > > > > > > > > transactional semantic mismatches across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > > > > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we would likely avoid using > > JDBC's > > >>>> > > > > > > > > > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > > > > > > > > > > single row updates, which adds additional > > >>>> overhead and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > > without > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>> Bourlatchkov < > > >>>> > > > > > > > > [email protected]> > > >>>> > > > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>> > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > > > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > It looks like people are starting to > encounter > > >>>> > > > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > > They > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > > incremental fixes that can work. However, I > > >>>> believe it > > >>>> > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > > > Polaris community to review and improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > > these > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and validates current catalog state, then > > >>>> commits a > > >>>> > > > > change > > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes during renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > > >>>> > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > > > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > > > > > > > > > > > > * Supporting atomic changes across multiple > > >>>> similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering > of > > >>>> list > > >>>> > > > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > > [4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side retries for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4831] > > >>>> https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>> https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>> https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > > > > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > >>> > > >
