Heads up: This discussion is on the Community Sync call agenda for Sept 17.
Interested parties, please plan to attend, if possible :) Cheers, Dmitri. On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi JB, > > I've added an agenda item to the next community sync call for this (Sept > 17, 2026). > > Re: key points: did you mean a real GH discussion or dev email (this > thread)? Just double checking :) I'm fine with either approach. > > Cheers, > Dmitri. > > On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Dmitri, >> >> I agree on the need to align on these points, though I'm not entirely >> sure a dedicated meeting is necessary. Let's start by using some time >> during the next community meeting to discuss it. >> >> In the meantime, what if we outline the key points in a GitHub >> Discussion first? We can then use that as a reference during the >> meeting. >> >> Regards, >> JB >> >> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <[email protected]> >> wrote: >> > >> > Hi All, >> > >> > My impression from this thread is that it might be time for a call to >> > discuss all the related issues and try to arrive at a shared >> implementation >> > plan. >> > >> > I know meetings are not ideal, but in this case it might be beneficial >> as a >> > means for achieving a common understanding of the set of problems and >> > priorities related to this thread. A dedicated metrics meeting worked >> well >> > from my POV. >> > >> > Allocating a time slot in the next community meeting might be an option, >> > although I think we might need the full hour in this case. >> > >> > WDYT? >> > >> > Thanks, >> > Dmitri. >> > >> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <[email protected]> >> > wrote: >> > >> > > Hi all, >> > > >> > > I rewrote https://github.com/apache/polaris/pull/5035, following the >> > > discussion to focus only on the SPI foundation. >> > > >> > > What is in the PR now: >> > > >> > > - A written manager-level consistency contract in >> > > site/content/in-dev/unreleased/persistence-consistency.md. >> > > - CAS-aware EntityMutation / GrantMutation records. >> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with >> backends >> > > opting in via supportsAtomicMixedCommit(). >> > > - commitChangeSet implementations for the in-memory TreeMap >> backend and >> > > JDBC. >> > > - AtomicOperationMetaStoreManager and >> TransactionalMetaStoreManagerImpl >> > > grant/revoke paths now build one change set per operation, falling >> back >> > > to >> > > individual operations when the backend does not support mixed >> commits. >> > > - Tests for the change-set API and TreeMap atomic commits. >> > > >> > > I intentionally left out: >> > > >> > > - Entity deletes in MetaStoreChangeSet. >> > > - Refactoring createCatalog, dropEntity, and renameEntity to use >> change >> > > sets. >> > > - NoSQL support for commitChangeSet. >> > > >> > > I am treating this as Phase 1 so the contract and the SPI primitive >> can be >> > > reviewed before the larger operation migrations land. >> > > >> > > Could you please take a look? In particular, I would like to know >> whether >> > > the contract captures the consensus so far, or if it needs to address >> retry >> > > / external-work coordination before we merge this foundation. >> > > >> > > Thanks, >> > > Prithvi S >> > > >> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <[email protected] >> > >> > > wrote: >> > > >> > > > Hi Robert, >> > > > >> > > > I agree that the client-visible operation (e.g. REST API request) >> > > consists >> > > > of many distinct phases. The database / persistence changes are >> just one >> > > of >> > > > these phases. STS (as an example) is another. My point about >> transactions >> > > > in JDBC Persistence applies to the former (database changes) phase >> - one >> > > > retry attempt there should ideally involve exactly one transaction. >> > > > >> > > > We certainly need logic inside Polaris Servers to to coordinate >> requests >> > > to >> > > > various external systems depending on outcomes from previous request >> > > > processing phases. >> > > > >> > > > I also agree that validation based on database state should be >> repeated >> > > > from scratch if we retry the database changes. >> > > > >> > > > However, internal RBAC authorization naturally has to happen inside >> the >> > > > database update phase (read-check-write). IIRC, that is part of the >> > > > implicit consistency guarantees, which were discussed during early >> > > project >> > > > months (but never got written down in full clarity, unfortunately). >> If >> > > the >> > > > Authorization check is inside that phase for internal RBAC, I >> suppose it >> > > > will be there for other Authorizers too and that can involve >> external >> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization >> to be >> > > > efficient, but I'm not sure how quick it can be in practice. There's >> > > > certainly potential for delays in some (perhaps infrequent) cases. >> > > > >> > > > Cheers, >> > > > Dmitri. >> > > > >> > > > >> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <[email protected]> >> wrote: >> > > > >> > > > > Hi Dmitri, >> > > > > >> > > > > I agree that one logical operation needs a consistent view of the >> whole >> > > > > backend state. >> > > > > What worries me is treating one long JDBC transaction as the >> boundary >> > > of >> > > > > that operation and then retrying the complete request if the >> commit >> > > > fails. >> > > > > >> > > > > The request can also write metadata to object storage or call STS. >> > > > > Each system can leave us with an outcome whose certainty we cannot >> > > know. >> > > > > The database may have committed before the connection was lost, an >> > > > > object-store write may have succeeded before a timeout, or STS >> may have >> > > > > issued credentials before its response was lost. >> > > > > A database rollback cannot undo any of those effects, and >> retrying the >> > > > > whole request may repeat them. >> > > > > >> > > > > So I think the contract has to separate the overall operation >> from each >> > > > > attempt to commit it. >> > > > > Each database or backend commit attempt should stay short. >> > > > > When we retry, validation and authorization need to use the state >> for >> > > > that >> > > > > new attempt, and we need clear rules for which external work can >> be >> > > > reused, >> > > > > repeated, cleaned up, or reconciled after an uncertain outcome. >> > > > > >> > > > > Some contextual state could be useful for carrying that state. >> > > > > But the JDBC connection and transaction should be an >> implementation >> > > > detail >> > > > > inside it, not the boundary of the complete operation. >> > > > > >> > > > > Cheers, >> > > > > Robert >> > > > > >> > > > > >> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < >> [email protected] >> > > > >> > > > > wrote: >> > > > > >> > > > > > Hi All, >> > > > > > >> > > > > > The idea of a backend-agnostic change-set primitive sounds good. >> > > > > > >> > > > > > However, I am not sure it is sufficient for all the use cases we >> > > > touched >> > > > > > here. More specifically, the state read by validation code is >> not >> > > > > > necessarily reflected in the change set: >> > > > > > >> > > > > > * Unchanged entities might be considered by validation, but >> changed >> > > in >> > > > a >> > > > > > parallel request. >> > > > > > * Even for changed entities the state read by validation may >> differ >> > > > from >> > > > > > the state being altered. Some validation code talks to the >> MetaStore >> > > > > > directly, outside of the data produced by the Resolver. >> > > > > > >> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to >> ensure >> > > > > > consistency between these reads and subsequent writes. >> > > > > > >> > > > > > As far as JDBC goes, running an overarching Transaction across >> all >> > > > reads >> > > > > > and writes in the same request, at the SERIALIZABLE isolation >> level >> > > in >> > > > > the >> > > > > > backing RDBMS could solve the problem, I think. >> > > > > > >> > > > > > Indeed, such a transaction might be long to accommodate calls >> made by >> > > > > > Polaris to external storage, etc. However, is that a problem? I >> think >> > > > the >> > > > > > alternative is for the JDBC Persistence impl. to "manually" >> track all >> > > > > reads >> > > > > > under the same request and redo them in the small transaction >> that >> > > > > persists >> > > > > > the writes. This will also add RDBMS overhead and require >> complex >> > > code >> > > > in >> > > > > > Polaris to handle the data properly. >> > > > > > >> > > > > > Tracking the request-wide transaction can be done only for JDBC >> > > without >> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I >> think. >> > > NoSQL >> > > > > > will use other mechanisms to ensure read/write consistency. >> > > > > > >> > > > > > Connecting to Robert's email (a parallel branch in this >> discussion), >> > > > I'd >> > > > > > like to propose this approach: >> > > > > > >> > > > > > * Each request establishes a "Data Context" >> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection + >> Tx >> > > > > > - In NoSQL the Data Context tracks one or more reference >> hashes >> > > > > > * All Persistence access in the same request goes through the >> same >> > > Data >> > > > > > Context >> > > > > > * All Persistence changes are committed once at the end of the >> > > request >> > > > > > - Not all changes have to be globally atomic. For example, >> changes >> > > in >> > > > > > Catalogs A and B do not have to be atomic with respect to each >> other. >> > > > We >> > > > > > can go deeper into this later. This is relevant to NoSQL. >> > > > > > - Transactional backends like JDBC can, of course, choose to >> make >> > > all >> > > > > > changes globally atomic. >> > > > > > * A commit can fail in two main ways: >> > > > > > - A retriable failure like an optimistic lock error or Tx >> > > > > serializability >> > > > > > error >> > > > > > - A non-triable logical error (e.g. entity not found) >> > > > > > * On a retriable error the whole request is re-attempted (as if >> > > > > resubmitted >> > > > > > by a client) a few times (configurable timeout). >> > > > > > >> > > > > > WDYT? >> > > > > > >> > > > > > Cheers, >> > > > > > Dmitri. >> > > > > > >> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < >> > > [email protected] >> > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Hi all >> > > > > > > >> > > > > > > I think Robert has a good point. >> > > > > > > >> > > > > > > If the atomicity guarantee lives only in BasePersistence, >> then the >> > > > > > > manager contract can't tell a caller whether the state it >> read for >> > > > > > > validation/authorization/credential-vending is the same state >> that >> > > > > > > eventually commits. That's the actual bug class behind the >> JDBC >> > > > > > > symptoms (not any single operation being non-atomic, but the >> > > contract >> > > > > > > being silent about it. Every operation-specific fix (like >> #4939, >> > > > #5035 >> > > > > > > or #5095) re-answers this question locally and it keeps >> recurring. >> > > So >> > > > > > > the deliverable should start with a written consistency >> contract at >> > > > > > > the manager level. >> > > > > > > >> > > > > > > I'm not sure migrating everyone to >> > > TransactionalMetaStoreManagerImpl >> > > > > > > is a good idea. It would tie the logical change set to a DB >> > > > > > > transaction spanning the REST request. It means: >> > > > > > > - it holds a durable transaction open across slow external >> work >> > > > > > > (credential vending, OPA, Ranger, ...) >> > > > > > > - it doesn't map to NoSQL >> > > > > > > - It wraps single-row updates in runWiithinTransaction, which >> is a >> > > > > > overhead >> > > > > > > >> > > > > > > So, I think the transactional manager isn't a portable >> target. It's >> > > > > > > "only" one backend's strategy. >> > > > > > > >> > > > > > > I think Privthi's approach is right. A backend-agnostic >> change-set >> > > > > > > primitive with a documented fallback is the correct shape. >> There is >> > > > > > > one caveat: without carrying the original entitiy (not just >> the new >> > > > > > > one) the change set can't express optimistic concurrency. >> > > > > > > >> > > > > > > I propose the following multi-steps approach: >> > > > > > > 1. We write the manager-level consistency contract as Robert >> asked. >> > > > It >> > > > > > > should include the explicit statement that a logical change >> set is >> > > > not >> > > > > > > a request-scope DB transaction. >> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern) >> > > baseline >> > > > > > > first-class in EntityMutation before merging the SPI >> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity. >> > > > > > > >> > > > > > > Thoughts? >> > > > > > > >> > > > > > > Regards >> > > > > > > JB >> > > > > > > >> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <[email protected]> >> > > wrote: >> > > > > > > > >> > > > > > > > Hi all, >> > > > > > > > >> > > > > > > > Yufei's clarification about where the atomicity guarantee is >> > > > defined >> > > > > > > seems >> > > > > > > > important. >> > > > > > > > If it is a property of the lower-level BasePersistence >> contract >> > > > > rather >> > > > > > > than >> > > > > > > > the general PolarisMetaStoreManager contract, the general >> > > contract >> > > > > does >> > > > > > > not >> > > > > > > > tell callers whether the state used for validation, >> > > authorization, >> > > > or >> > > > > > > > credential vending is consistent with the change that >> eventually >> > > > > > commits. >> > > > > > > > >> > > > > > > > The current PRs suggest that operation-specific multi-object >> > > > methods >> > > > > > can >> > > > > > > > fix individual cases, while leaving the same contract >> question to >> > > > > recur >> > > > > > > for >> > > > > > > > each new case. >> > > > > > > > >> > > > > > > > I would also avoid defining a logical change set as a >> database >> > > > > > > transaction >> > > > > > > > around an entire REST request. >> > > > > > > > That would tie the contract to the backend and database, and >> > > could >> > > > > keep >> > > > > > > the >> > > > > > > > durable attempt open across slow external work. >> > > > > > > > >> > > > > > > > So is the choice really between the two current manager >> > > > > > implementations, >> > > > > > > or >> > > > > > > > do we first need to revisit the boundary and guarantees >> exposed >> > > to >> > > > > > their >> > > > > > > > callers? >> > > > > > > > >> > > > > > > > Cheers, >> > > > > > > > Robert >> > > > > > > > >> > > > > > > > >> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov < >> > > > > [email protected] >> > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi Yufei, >> > > > > > > > > >> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic >> change >> > > > per >> > > > > > > method" >> > > > > > > > > contract as defined in javadoc does apply only to >> > > BasePersistence >> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to >> > > > > > > BasePersistence). >> > > > > > > > > >> > > > > > > > > Note that all non-test Persistence implementations in the >> > > Polaris >> > > > > > > codebase >> > > > > > > > > extend those classes (which is probably why I was confused >> > > about >> > > > > > > atomicity >> > > > > > > > > expectations). >> > > > > > > > > >> > > > > > > > > However, this creates a gap in the Persistence SPI >> > > specification. >> > > > > If >> > > > > > > > > other PolarisMetaStoreManager implementations do not have >> to >> > > > comply >> > > > > > > with >> > > > > > > > > this principle, it will create a conceptual difficulty at >> call >> > > > > sites. >> > > > > > > How >> > > > > > > > > can PolarisMetaStoreManager callers reason about >> consistency >> > > and >> > > > > > > durability >> > > > > > > > > behaviours in general? >> > > > > > > > > >> > > > > > > > > I believe we need to address that as part of this >> discussion. >> > > > > > > > > >> > > > > > > > > Cheers, >> > > > > > > > > Dmitri. >> > > > > > > > > >> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < >> [email protected] >> > > > >> > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > > The MetaStore SPI is currently defined with the idea >> that >> > > one >> > > > > > > method >> > > > > > > > > call >> > > > > > > > > > means one atomic change. >> > > > > > > > > > >> > > > > > > > > > If the "MetaStore SPI" refers to the interface >> > > > > > > PolarisMetaStoreManager, I >> > > > > > > > > > don't think we've ever state each method to be atomic. >> We did >> > > > > > clarify >> > > > > > > > > > atomicity[1] in the interface BasePersistence though. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > 1. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Yufei >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov < >> > > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Hi Yufei, >> > > > > > > > > > > >> > > > > > > > > > > I agree that JDBC transactions must be handled more >> > > > explicitly. >> > > > > > > > > However, >> > > > > > > > > > > I'm not sure that simply moving to >> > > > > > > TransactionalMetaStoreManagerImpl is >> > > > > > > > > > > sufficient. >> > > > > > > > > > > >> > > > > > > > > > > The MetaStore SPI is currently defined with the idea >> that >> > > one >> > > > > > > method >> > > > > > > > > call >> > > > > > > > > > > means one atomic change [1]. The "transactional" >> MetaStore >> > > > > impl. >> > > > > > is >> > > > > > > > > but a >> > > > > > > > > > > sub-case of that. It cannot alter the high-level >> contract. >> > > > > > > > > > > >> > > > > > > > > > > We could add SPI methods having multiple object >> parameters >> > > to >> > > > > > > represent >> > > > > > > > > > > grouped changes, but I am not sure it will be a sound >> > > design. >> > > > > > This >> > > > > > > will >> > > > > > > > > > > bloat the interface surfaces and require extra impl. >> effort >> > > > for >> > > > > > > each >> > > > > > > > > > > backend type. More importantly, adding multi-arg >> change >> > > > methods >> > > > > > > still >> > > > > > > > > > won't >> > > > > > > > > > > address the problem of reads being consistent with >> writes, >> > > > > > because >> > > > > > > each >> > > > > > > > > > > method call will still be independent regarding the >> data >> > > > stored >> > > > > > in >> > > > > > > the >> > > > > > > > > > > database. >> > > > > > > > > > > >> > > > > > > > > > > I tend to think we need to introduce a "change set" or >> > > > "atomic >> > > > > > > batch" >> > > > > > > > > > > concept to core Persistence and associate each REST >> API >> > > > request >> > > > > > > with >> > > > > > > > > one >> > > > > > > > > > > such change set, which will be committed (or rolled >> back) >> > > at >> > > > > the >> > > > > > > end of >> > > > > > > > > > the >> > > > > > > > > > > request. I believe Ayush mentioned a similar concept >> in PR >> > > > 4939 >> > > > > > > [2]. In >> > > > > > > > > > > JDBC each change set will naturally be associated >> with an >> > > > RDBMS >> > > > > > > > > > > transaction. In NoSQL persistence, each atomic change >> set >> > > > will >> > > > > be >> > > > > > > > > > > associated with one CAS operation on the underlying >> > > database. >> > > > > > > > > > > >> > > > > > > > > > > [1] >> > > > > > > >> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r >> > > > > > > > > > > >> > > > > > > > > > > [2] >> > > > > > > >> https://github.com/apache/polaris/pull/4939#discussion_r3575719158 >> > > > > > > > > > > >> > > > > > > > > > > Cheers, >> > > > > > > > > > > Dmitri. >> > > > > > > > > > > >> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < >> > > > [email protected] >> > > > > > >> > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Thanks for raising this, Dmitri. >> > > > > > > > > > > > >> > > > > > > > > > > > These are valid concerns, and they were already >> > > recognized >> > > > > when >> > > > > > > we >> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that >> time, we >> > > > > chose >> > > > > > > to use >> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to >> the >> > > > > > > simplicity. I >> > > > > > > > > > > think >> > > > > > > > > > > > most of the issues mentioned here can already be >> > > addressed >> > > > by >> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. >> > > > > > > > > > > > >> > > > > > > > > > > > For example, rename is already wrapped in a >> transaction >> > > in >> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly, >> catalog >> > > > > > > creation, >> > > > > > > > > > which >> > > > > > > > > > > > involves reading and creating multiple objects, is >> also >> > > > > > executed >> > > > > > > > > > within a >> > > > > > > > > > > > transaction [2]. >> > > > > > > > > > > > >> > > > > > > > > > > > I see two possible directions: >> > > > > > > > > > > > >> > > > > > > > > > > > 1. >> > > > > > > > > > > > >> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager together >> with >> > > the >> > > > > > > > > persistence >> > > > > > > > > > > > backends (such as JDBC) to provide the required >> > > > > consistency >> > > > > > > > > > guarantees >> > > > > > > > > > > > for >> > > > > > > > > > > > specific operations, similar to what >> > > > > > > > > > TransactionalMetaStoreManagerImpl >> > > > > > > > > > > > does. >> > > > > > > > > > > > 2. >> > > > > > > > > > > > >> > > > > > > > > > > > Migrate the persistence backends (such as JDBC) >> to use >> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We >> may >> > > have >> > > > to >> > > > > > > deal >> > > > > > > > > with >> > > > > > > > > > > > transactional semantic mismatches across >> different >> > > > > > persistence >> > > > > > > > > > > backends. >> > > > > > > > > > > > For example, we would likely avoid using JDBC's >> > > > > > > > > > `runWithinTransaction` >> > > > > > > > > > > > for >> > > > > > > > > > > > single row updates, which adds additional >> overhead and >> > > > > > > complexity >> > > > > > > > > > > > without >> > > > > > > > > > > > benefits. >> > > > > > > > > > > > >> > > > > > > > > > > > References: >> > > > > > > > > > > > >> > > > > > > > > > > > 1. >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 >> > > > > > > > > > > > 2. >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 >> > > > > > > > > > > > >> > > > > > > > > > > > Yufei >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov >> < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi all, >> > > > > > > > > > > > > >> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple of >> > > > > > interesting >> > > > > > > PRs: >> > > > > > > > > > > > > [4939], [5035]. >> > > > > > > > > > > > > >> > > > > > > > > > > > > It looks like people are starting to encounter >> > > > consistency >> > > > > > > issues >> > > > > > > > > in >> > > > > > > > > > > > > JDBC persistence. >> > > > > > > > > > > > > >> > > > > > > > > > > > > The PRs provide valuable insight into the >> underlying >> > > > > issues. >> > > > > > > They >> > > > > > > > > > offer >> > > > > > > > > > > > > incremental fixes that can work. However, I >> believe it >> > > is >> > > > > > time >> > > > > > > for >> > > > > > > > > > the >> > > > > > > > > > > > > Polaris community to review and improve this area >> of >> > > the >> > > > > > > codebase >> > > > > > > > > > > > > holistically. >> > > > > > > > > > > > > >> > > > > > > > > > > > > By this, I mean finding a solution that can be >> applied >> > > to >> > > > > all >> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and >> > > > addresses >> > > > > > > these >> > > > > > > > > > > > > aspects: >> > > > > > > > > > > > > >> > > > > > > > > > > > > * Supporting concurrent and consistent changes >> where >> > > the >> > > > > > > service >> > > > > > > > > > reads >> > > > > > > > > > > > > and validates current catalog state, then >> commits a >> > > > > change >> > > > > > > (e.g. >> > > > > > > > > > > > > name clashes during renames). >> > > > > > > > > > > > > * Supporting consistent but independent changes >> to RBAC >> > > > > > grants >> > > > > > > and >> > > > > > > > > > > > > MetaStore entities. This independence is needed >> to >> > > > > support >> > > > > > > > > > > > > external authorizers like OPA and Ranger. >> > > > > > > > > > > > > * Supporting atomic changes across multiple >> similar >> > > > > entities. >> > > > > > > > > > > > > * Supporting authorization-based filtering of list >> > > > > operations >> > > > > > > (cf. >> > > > > > > > > > > > > [4831]). >> > > > > > > > > > > > > * Supporting credential-vending decisions that are >> > > rooted >> > > > > in >> > > > > > > the >> > > > > > > > > > > > > exact state of the catalog. >> > > > > > > > > > > > > * Supporting server-side retries for transient >> > > > persistence >> > > > > > > failures >> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). >> > > > > > > > > > > > > >> > > > > > > > > > > > > Please share your comments and ideas. >> > > > > > > > > > > > > >> > > > > > > > > > > > > [4831] >> https://github.com/apache/polaris/pull/4831 >> > > > > > > > > > > > > >> > > > > > > > > > > > > [4939] >> https://github.com/apache/polaris/pull/4939 >> > > > > > > > > > > > > >> > > > > > > > > > > > > [5035] >> https://github.com/apache/polaris/pull/5035 >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > Dmitri >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
