Hi Dennis I agree with you. I think "multi object conditional commits" rather than a request-scoped transaction is aligned with what Robert and I argued: commit attempts stay short, it maps to both JDBC and NoSQL. An explicit read set checked at commit time stays compatible with in-memory caching (I don't think a serializable only approach does not).
I believe it is also a small step forward from what we already have. The multi-table commit path already does a multi-entity CAS through updateEntitiesPropertiesIfNotChanged. We should extend it. I have some concerns about using PolarisResolutionManifest as is: 1. As Dmitri mentioned, the location overlap check does not go through the manifest at all. So the commit "contract" needs predicate checks that the backend should re-evaluate at commit time. 2. The manifest is not an "open book" today. getPassthroughResolvedPath creates a single user Resolver on every call and does not record the version it returned. I don't see how to store the version easily today. 3. In your example the catalog is not pinned, but validating a table location against the allowed locations depends on the catalog storage configuration, so for a create it would have to be. I would prefer recording versions on read, or pinning by default with an explicit opt-out. 4. To simplify, I would define the read set as a small type in the Persistence SPI, which the manifest produces, rather than making the manifest itself the SPI. I would propose to define the read set in Persistence SPI terms, independent from the resolver. and also include predicate (location overlap, name conflict) in the contract. I think it's pretty close to what Dmitri is proposing (the "Persistence Session "iidea). Regards JB On Thu, Oct 1, 2026 at 2:41 AM Dennis Huo <[email protected]> wrote: > > Agree we should formulate in terms of Persistence SPIs first. > > IIRC we talked about this in one of the community syncs but I forgot to > bring it here -- the PolarisResolutionManifest in concept already > represents *exactly* the set of "which elements were engaged during request > processing". > > Adherence may have drifted, but at least early on, we made sure all the > IcebergCatalog logic did *not* get to do new direct reads from the > persistence layer, but instead needs to "read" from the *authorized* set of > entities contained in a PolarisResolutionManifest (or declared as a > "passthroughPath" == allow reading fresh from DB but had to pre-register it > for authz purposes). > > It's important to keep this unified anyways because it really serves two > things: > > 1. Declares the set of entities whose state you expect to resolve/snapshot > up-front in the operation-handling, that the IcebergCatalog logic then > operates on later; also explicitly declare which ones require fetching > *fresh* versions mid-operation (addPassthroughPath) > 2. Enforces that RBAC authz is performed against that pre-declared set of > entities -- if later processign were allowed to access arbitrary other > entities to make handler decisions, then how did we "prove" that the > authorization engine actually allowed the request to access those arbitrary > other entities? Even if it's just for internal-processing purposes, the > very fact that something got accessed means there's a route to accidental > leakage of unauthorized data. > > So in a way, solid/provable authz and solid/provable consistency go > hand-in-hand. > > I think PolarisResolutionManifest in its current form is probably a bit > opaque and doesn't 100% convey the semantics we want quite yet, but it > seems like the right SPI starting point to extend or at least evolve from. > Basically, aside from addPath and addPassthroughPath, there could be a > notion of annotating a given path as being one of the "dependent states" to > carry over into the later commit. Conceptually: > > // Prepare operation processing > resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime = > false); // For this operation let's pretend we don't care if the root > catalog changes before we commit the change to the tables > resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime = > false); // For this operation let's pretend we don't care if the parent > namespace changes before we commit the change to the tables > resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true); > // Assume multi-table commit - table1 will have changes and also must not > have already changed at commit time > resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true); > // Same for table2 > > // .... Do all the Iceberg processing work > > // When we commit, we use the resolutionManifest as the ledger of things we > used that must still be at their unchanged version at time of commit > metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated), > resolutionManifest.getSetOfUnchangedEntityRequirements()); > > Overall this fits the optimistic-concurrency theme of Iceberg and Polaris > in general, and would *intentionally* not try to express a full-fledged > "Transaction" session in which you do a bunch of reads/writes/reads/writes > on a single entity as if you're operating in an uncommitted block on a > RDBMS, instead opting for the middle-ground of "multi-object conditional > commits". > > On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi EJ, > > > > > 1. Which backend assumptions should we design for? [...] > > > > I know I used the term "transaction" a lot during the call, but the real > > challenge is about establishing an SPI with clear contracts that could be > > implemented by JDBC and NoSQL persistence using the database features > > natural to each model. > > > > I am fairly confident that a JDBC impl. is possible with Serializable > > transaction. We should be able to solve all the problems raised in this > > thread. However, that will make in-memory caching impossible because the > > database needs to see all the reads (and writes) in a Tx to be able to > > ensure serializable guarantees. If a cache prevents a read from hitting the > > database, that read (e.g. storage config) will not be a factor in > > serializability. > > > > This connects to my earlier question about the in-memory cache and JDBC > > (title: Usefulness of InMemoryEntityCache) > > > > Are different approaches possible with JDBC? Probably. What is involved? > > This depends on the SPI contract, I think. > > > > > 2. How do we protect the data used by validation? [...] > > > > The point mentioned by Robert previously IIRC, is that we need to ground > > all runtime decisions to a particular state of the catalog. > > > > If at commit time the state of the catalog in the database shifts from what > > was used during validation, the request has to restart from scratch. > > > > Current JDBC Persistence does this only for relationships between entities > > and grants and between versions of the same entity. > > > > Relationships between table locations and storage config are not actualized > > for the purpose of state consistency tracking. > > > > From my POV we need to begin by formulating this Persistence state in the > > Persistence SPI terms. Then, think how we can implement that for JDBC. > > > > NoSQL Persistence already has a per-catalog state (not granular), so wiring > > the new SPI to NoSQL should not be too hard, I hope. > > > > The state in the SPI does not have to be global to the catalog (as in > > NoSQL), but it needs to be able to express the idea of which elements were > > engaged during request processing so serve as input to Persistence impl. > > for cosistency guarantees. If Persistence caches data in memory (for > > example), it will have to do so in a way that does not violate consistency > > expectations. > > > > From my POV we could begin by defining a "Persistence Session" concept. > > Each request is 1:1 with a session. All reads and writes go through the > > session. The next step would be the JDBC impl., which probably needs some > > brainstorming... but I wonder if we could reach a consensus at a high level > > first. > > > > WDYT? > > > > Thanks, > > Dmitri. > > > > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <[email protected]> > > wrote: > > > > > Thanks for the recap, Dmitri. I'd like to follow up on the three points I > > > raised during the sync. We didn't have much time to discuss them, so it > > > would help to confirm these before choosing an SPI design. > > > > > > 1. Which backend assumptions should we design for? > > > > > > Within one Polaris server instance, do we assume entities, grants and > > > secrets all use the same database backend, or must we support mixing > > > backends across them? > > > Must every supported persistence implementation provide transactional > > > guarantees? Does the database itself need to provide transactions, or can > > > Polaris build that capability on top of simpler database operations? > > > > > > I'm happy to work with a homogeneous assumption if that's the intended > > > scope. Existing NoSQL illustrates the second question: it can publish > > > changes to multiple objects under one reference using a single > > conditional > > > pointer update. > > > > > > 2. How do we protect the data used by validation? > > > > > > Your storage-configuration and location-overlap examples make the problem > > > concrete. Consider these races: > > > > > > A table creation passes validation against the catalog's allowed > > locations, > > > then another request changes that configuration before the table is > > > committed. > > > Two table-creation requests each check for overlapping locations, both > > see > > > no overlap, and then create tables at conflicting locations. > > > > > > Checking only the record being written cannot cover these dependencies. > > The > > > first also depends on the catalog configuration. The second depends on > > the > > > absence of another overlapping table. > > > > > > Could the persistence layer protect these reads together with the writes, > > > using backend-internal versions/tokens or database isolation? Shared code > > > would still need to identify the required reads and checks. Could we keep > > > the mechanism for protecting them inside each backend? > > > > > > 3. Business rules are duplicated across backend implementations > > > > > > Today, the TreeMap > > > < > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > > > > > and JDBC > > > < > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > > > > > implementations each load a secret, check its principal, apply the > > > rotation/reset steps and write it back. > > > > > > Changing these business rules therefore requires keeping multiple > > > implementations in sync. A fix applied to one backend can be missed in > > > another, causing the same Polaris operation to behave differently > > depending > > > on the database. > > > > > > My proposal is to implement each business workflow once, against a common > > > set of storage operations with defined consistency and failure semantics. > > > The shared workflow should not need to know whether those operations are > > > implemented through JDBC transactions, NoSQL reference CAS or another > > > mechanism. Each backend would implement the same storage contract, > > keeping > > > its execution details internal. A new database backend could then reuse > > the > > > existing business workflows. > > > > > > Do these seem like reasonable starting points? We can then evaluate the > > SPI > > > against the original failure cases on both JDBC and existing NoSQL. > > > > > > -ej > > > > > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Side note: I just came across [5541], which shows a case where a failed > > > > request leaves persisted side effects. > > > > > > > > This can be seen as a coding mistake, of course. However, this is about > > > > executing one request as a single, atomic unit. This is something that > > > > probably needs foundational support and coding patterns in Polaris > > code. > > > > > > > > Cheers, > > > > Dmitri. > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov <[email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Here's a recap of the Community Sync discussion as I remember it. I'm > > > > sure > > > > > my recollection is not complete, so please add your thoughts to this > > > > thread. > > > > > > > > > > * Polaris traditionally relies on entity version number to ensure > > > > > consistency of changes. > > > > > > > > > > This works well for internal RBAC grants, which also have version > > > numbers > > > > > cross-referenced at the Persistence layer. > > > > > > > > > > I do not think this works in more general cases like validating > > > locations > > > > > wrt catalog-level storage configuration. > > > > > > > > > > Another difficult case involves location overlaps between concurrent > > > > table > > > > > creations. > > > > > > > > > > * We talked about how Persistence calls related to RDBMS transactions > > > (in > > > > > the JDBC persistence case). > > > > > > > > > > There were some concerns raised about whether we need to bind > > > Persistence > > > > > API to transactions explicitly. > > > > > > > > > > I think it is a valid concern. At the same time, the Persistence API > > > > > should be clear about consistency guarantees across all backend > > > > > implementations. I think this needs more attention now that the > > matter > > > of > > > > > milti-entioty changes became prominent in [5035]. > > > > > > > > > > I personally believe that RDBMS-based Persistence in Polaris has to > > use > > > > > Serializable transaction isolation. Consequently, the core code needs > > > to > > > > > provide mechanisms to allow the Persistence implementation to > > properly > > > > > connect reads and writes from API requests to JDBC transactions... > > > > whether > > > > > we use "transation" as an explicit terms in the SPI or not. > > > > > > > > > > Note that currently, the same API request performs reads and writes > > in > > > > > _separate_ RDBMS transactions (even separate connections). > > > > > > > > > > * We did not discuss Persistence SPI implications in the call, but > > > > > currently each Persistence call is assumed to be one distinct and > > > atomic > > > > > change (please correct me if I'm wrong). > > > > > > > > > > Consequently, supporting new use cases involves introducing new SPI > > > > > methods with many parameters. This can be seen in PR [5035]. This > > tends > > > > to > > > > > overcomplicate the SPI. > > > > > > > > > > Ideally, I think the SPI should be simplified to deal with > > multi-entity > > > > > and single-entity changes in a coherent manner to avoid any ambiguity > > > > about > > > > > which method the caller should use. > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > >> Heads up: This discussion is on the Community Sync call agenda for > > > Sept > > > > >> 17. > > > > >> > > > > >> Interested parties, please plan to attend, if possible :) > > > > >> > > > > >> Cheers, > > > > >> Dmitri. > > > > >> > > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov < > > [email protected]> > > > > >> wrote: > > > > >> > > > > >>> Hi JB, > > > > >>> > > > > >>> I've added an agenda item to the next community sync call for this > > > > (Sept > > > > >>> 17, 2026). > > > > >>> > > > > >>> Re: key points: did you mean a real GH discussion or dev email > > (this > > > > >>> thread)? Just double checking :) I'm fine with either approach. > > > > >>> > > > > >>> Cheers, > > > > >>> Dmitri. > > > > >>> > > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < > > > [email protected]> > > > > >>> wrote: > > > > >>> > > > > >>>> Hi Dmitri, > > > > >>>> > > > > >>>> I agree on the need to align on these points, though I'm not > > > entirely > > > > >>>> sure a dedicated meeting is necessary. Let's start by using some > > > time > > > > >>>> during the next community meeting to discuss it. > > > > >>>> > > > > >>>> In the meantime, what if we outline the key points in a GitHub > > > > >>>> Discussion first? We can then use that as a reference during the > > > > >>>> meeting. > > > > >>>> > > > > >>>> Regards, > > > > >>>> JB > > > > >>>> > > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > >>>> wrote: > > > > >>>> > > > > > >>>> > Hi All, > > > > >>>> > > > > > >>>> > My impression from this thread is that it might be time for a > > call > > > > to > > > > >>>> > discuss all the related issues and try to arrive at a shared > > > > >>>> implementation > > > > >>>> > plan. > > > > >>>> > > > > > >>>> > I know meetings are not ideal, but in this case it might be > > > > >>>> beneficial as a > > > > >>>> > means for achieving a common understanding of the set of > > problems > > > > and > > > > >>>> > priorities related to this thread. A dedicated metrics meeting > > > > worked > > > > >>>> well > > > > >>>> > from my POV. > > > > >>>> > > > > > >>>> > Allocating a time slot in the next community meeting might be an > > > > >>>> option, > > > > >>>> > although I think we might need the full hour in this case. > > > > >>>> > > > > > >>>> > WDYT? > > > > >>>> > > > > > >>>> > Thanks, > > > > >>>> > Dmitri. > > > > >>>> > > > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S < > > > > >>>> [email protected]> > > > > >>>> > wrote: > > > > >>>> > > > > > >>>> > > Hi all, > > > > >>>> > > > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, > > > following > > > > >>>> the > > > > >>>> > > discussion to focus only on the SPI foundation. > > > > >>>> > > > > > > >>>> > > What is in the PR now: > > > > >>>> > > > > > > >>>> > > - A written manager-level consistency contract in > > > > >>>> > > site/content/in-dev/unreleased/persistence-consistency.md. > > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records. > > > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, > > > with > > > > >>>> backends > > > > >>>> > > opting in via supportsAtomicMixedCommit(). > > > > >>>> > > - commitChangeSet implementations for the in-memory TreeMap > > > > >>>> backend and > > > > >>>> > > JDBC. > > > > >>>> > > - AtomicOperationMetaStoreManager and > > > > >>>> TransactionalMetaStoreManagerImpl > > > > >>>> > > grant/revoke paths now build one change set per operation, > > > > >>>> falling back > > > > >>>> > > to > > > > >>>> > > individual operations when the backend does not support > > mixed > > > > >>>> commits. > > > > >>>> > > - Tests for the change-set API and TreeMap atomic commits. > > > > >>>> > > > > > > >>>> > > I intentionally left out: > > > > >>>> > > > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity > > to > > > > use > > > > >>>> change > > > > >>>> > > sets. > > > > >>>> > > - NoSQL support for commitChangeSet. > > > > >>>> > > > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI > > > > primitive > > > > >>>> can be > > > > >>>> > > reviewed before the larger operation migrations land. > > > > >>>> > > > > > > >>>> > > Could you please take a look? In particular, I would like to > > > know > > > > >>>> whether > > > > >>>> > > the contract captures the consensus so far, or if it needs to > > > > >>>> address retry > > > > >>>> > > / external-work coordination before we merge this foundation. > > > > >>>> > > > > > > >>>> > > Thanks, > > > > >>>> > > Prithvi S > > > > >>>> > > > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < > > > > >>>> [email protected]> > > > > >>>> > > wrote: > > > > >>>> > > > > > > >>>> > > > Hi Robert, > > > > >>>> > > > > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API > > > > request) > > > > >>>> > > consists > > > > >>>> > > > of many distinct phases. The database / persistence changes > > > are > > > > >>>> just one > > > > >>>> > > of > > > > >>>> > > > these phases. STS (as an example) is another. My point about > > > > >>>> transactions > > > > >>>> > > > in JDBC Persistence applies to the former (database changes) > > > > >>>> phase - one > > > > >>>> > > > retry attempt there should ideally involve exactly one > > > > >>>> transaction. > > > > >>>> > > > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > > > coordinate > > > > >>>> requests > > > > >>>> > > to > > > > >>>> > > > various external systems depending on outcomes from previous > > > > >>>> request > > > > >>>> > > > processing phases. > > > > >>>> > > > > > > > >>>> > > > I also agree that validation based on database state should > > be > > > > >>>> repeated > > > > >>>> > > > from scratch if we retry the database changes. > > > > >>>> > > > > > > > >>>> > > > However, internal RBAC authorization naturally has to happen > > > > >>>> inside the > > > > >>>> > > > database update phase (read-check-write). IIRC, that is part > > > of > > > > >>>> the > > > > >>>> > > > implicit consistency guarantees, which were discussed during > > > > early > > > > >>>> > > project > > > > >>>> > > > months (but never got written down in full clarity, > > > > >>>> unfortunately). If > > > > >>>> > > the > > > > >>>> > > > Authorization check is inside that phase for internal RBAC, > > I > > > > >>>> suppose it > > > > >>>> > > > will be there for other Authorizers too and that can involve > > > > >>>> external > > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect > > > > >>>> authorization to be > > > > >>>> > > > efficient, but I'm not sure how quick it can be in practice. > > > > >>>> There's > > > > >>>> > > > certainly potential for delays in some (perhaps infrequent) > > > > cases. > > > > >>>> > > > > > > > >>>> > > > Cheers, > > > > >>>> > > > Dmitri. > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp < > > [email protected]> > > > > >>>> wrote: > > > > >>>> > > > > > > > >>>> > > > > Hi Dmitri, > > > > >>>> > > > > > > > > >>>> > > > > I agree that one logical operation needs a consistent view > > > of > > > > >>>> the whole > > > > >>>> > > > > backend state. > > > > >>>> > > > > What worries me is treating one long JDBC transaction as > > the > > > > >>>> boundary > > > > >>>> > > of > > > > >>>> > > > > that operation and then retrying the complete request if > > the > > > > >>>> commit > > > > >>>> > > > fails. > > > > >>>> > > > > > > > > >>>> > > > > The request can also write metadata to object storage or > > > call > > > > >>>> STS. > > > > >>>> > > > > Each system can leave us with an outcome whose certainty > > we > > > > >>>> cannot > > > > >>>> > > know. > > > > >>>> > > > > The database may have committed before the connection was > > > > lost, > > > > >>>> an > > > > >>>> > > > > object-store write may have succeeded before a timeout, or > > > STS > > > > >>>> may have > > > > >>>> > > > > issued credentials before its response was lost. > > > > >>>> > > > > A database rollback cannot undo any of those effects, and > > > > >>>> retrying the > > > > >>>> > > > > whole request may repeat them. > > > > >>>> > > > > > > > > >>>> > > > > So I think the contract has to separate the overall > > > operation > > > > >>>> from each > > > > >>>> > > > > attempt to commit it. > > > > >>>> > > > > Each database or backend commit attempt should stay short. > > > > >>>> > > > > When we retry, validation and authorization need to use > > the > > > > >>>> state for > > > > >>>> > > > that > > > > >>>> > > > > new attempt, and we need clear rules for which external > > work > > > > >>>> can be > > > > >>>> > > > reused, > > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain > > > > outcome. > > > > >>>> > > > > > > > > >>>> > > > > Some contextual state could be useful for carrying that > > > state. > > > > >>>> > > > > But the JDBC connection and transaction should be an > > > > >>>> implementation > > > > >>>> > > > detail > > > > >>>> > > > > inside it, not the boundary of the complete operation. > > > > >>>> > > > > > > > > >>>> > > > > Cheers, > > > > >>>> > > > > Robert > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > > > >>>> [email protected] > > > > >>>> > > > > > > > >>>> > > > > wrote: > > > > >>>> > > > > > > > > >>>> > > > > > Hi All, > > > > >>>> > > > > > > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive > > sounds > > > > >>>> good. > > > > >>>> > > > > > > > > > >>>> > > > > > However, I am not sure it is sufficient for all the use > > > > cases > > > > >>>> we > > > > >>>> > > > touched > > > > >>>> > > > > > here. More specifically, the state read by validation > > code > > > > is > > > > >>>> not > > > > >>>> > > > > > necessarily reflected in the change set: > > > > >>>> > > > > > > > > > >>>> > > > > > * Unchanged entities might be considered by validation, > > > but > > > > >>>> changed > > > > >>>> > > in > > > > >>>> > > > a > > > > >>>> > > > > > parallel request. > > > > >>>> > > > > > * Even for changed entities the state read by validation > > > may > > > > >>>> differ > > > > >>>> > > > from > > > > >>>> > > > > > the state being altered. Some validation code talks to > > the > > > > >>>> MetaStore > > > > >>>> > > > > > directly, outside of the data produced by the Resolver. > > > > >>>> > > > > > > > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms > > > (ATM) > > > > >>>> to ensure > > > > >>>> > > > > > consistency between these reads and subsequent writes. > > > > >>>> > > > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction > > > > >>>> across all > > > > >>>> > > > reads > > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE > > > > isolation > > > > >>>> level > > > > >>>> > > in > > > > >>>> > > > > the > > > > >>>> > > > > > backing RDBMS could solve the problem, I think. > > > > >>>> > > > > > > > > > >>>> > > > > > Indeed, such a transaction might be long to accommodate > > > > calls > > > > >>>> made by > > > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > > > problem? > > > > >>>> I think > > > > >>>> > > > the > > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to > > > "manually" > > > > >>>> track all > > > > >>>> > > > > reads > > > > >>>> > > > > > under the same request and redo them in the small > > > > transaction > > > > >>>> that > > > > >>>> > > > > persists > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and > > require > > > > >>>> complex > > > > >>>> > > code > > > > >>>> > > > in > > > > >>>> > > > > > Polaris to handle the data properly. > > > > >>>> > > > > > > > > > >>>> > > > > > Tracking the request-wide transaction can be done only > > for > > > > >>>> JDBC > > > > >>>> > > without > > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL > > Persistence, I > > > > >>>> think. > > > > >>>> > > NoSQL > > > > >>>> > > > > > will use other mechanisms to ensure read/write > > > consistency. > > > > >>>> > > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in this > > > > >>>> discussion), > > > > >>>> > > > I'd > > > > >>>> > > > > > like to propose this approach: > > > > >>>> > > > > > > > > > >>>> > > > > > * Each request establishes a "Data Context" > > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC > > > > Connection > > > > >>>> + Tx > > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more > > reference > > > > >>>> hashes > > > > >>>> > > > > > * All Persistence access in the same request goes > > through > > > > the > > > > >>>> same > > > > >>>> > > Data > > > > >>>> > > > > > Context > > > > >>>> > > > > > * All Persistence changes are committed once at the end > > of > > > > the > > > > >>>> > > request > > > > >>>> > > > > > - Not all changes have to be globally atomic. For > > > example, > > > > >>>> changes > > > > >>>> > > in > > > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect > > to > > > > >>>> each other. > > > > >>>> > > > We > > > > >>>> > > > > > can go deeper into this later. This is relevant to > > NoSQL. > > > > >>>> > > > > > - Transactional backends like JDBC can, of course, > > > choose > > > > >>>> to make > > > > >>>> > > all > > > > >>>> > > > > > changes globally atomic. > > > > >>>> > > > > > * A commit can fail in two main ways: > > > > >>>> > > > > > - A retriable failure like an optimistic lock error or > > > Tx > > > > >>>> > > > > serializability > > > > >>>> > > > > > error > > > > >>>> > > > > > - A non-triable logical error (e.g. entity not found) > > > > >>>> > > > > > * On a retriable error the whole request is re-attempted > > > (as > > > > >>>> if > > > > >>>> > > > > resubmitted > > > > >>>> > > > > > by a client) a few times (configurable timeout). > > > > >>>> > > > > > > > > > >>>> > > > > > WDYT? > > > > >>>> > > > > > > > > > >>>> > > > > > Cheers, > > > > >>>> > > > > > Dmitri. > > > > >>>> > > > > > > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > > > >>>> > > [email protected] > > > > >>>> > > > > > > > > >>>> > > > > > wrote: > > > > >>>> > > > > > > > > > >>>> > > > > > > Hi all > > > > >>>> > > > > > > > > > > >>>> > > > > > > I think Robert has a good point. > > > > >>>> > > > > > > > > > > >>>> > > > > > > If the atomicity guarantee lives only in > > > BasePersistence, > > > > >>>> then the > > > > >>>> > > > > > > manager contract can't tell a caller whether the state > > > it > > > > >>>> read for > > > > >>>> > > > > > > validation/authorization/credential-vending is the > > same > > > > >>>> state that > > > > >>>> > > > > > > eventually commits. That's the actual bug class behind > > > the > > > > >>>> JDBC > > > > >>>> > > > > > > symptoms (not any single operation being non-atomic, > > but > > > > the > > > > >>>> > > contract > > > > >>>> > > > > > > being silent about it. Every operation-specific fix > > > (like > > > > >>>> #4939, > > > > >>>> > > > #5035 > > > > >>>> > > > > > > or #5095) re-answers this question locally and it > > keeps > > > > >>>> recurring. > > > > >>>> > > So > > > > >>>> > > > > > > the deliverable should start with a written > > consistency > > > > >>>> contract at > > > > >>>> > > > > > > the manager level. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > > > >>>> > > TransactionalMetaStoreManagerImpl > > > > >>>> > > > > > > is a good idea. It would tie the logical change set > > to a > > > > DB > > > > >>>> > > > > > > transaction spanning the REST request. It means: > > > > >>>> > > > > > > - it holds a durable transaction open across slow > > > external > > > > >>>> work > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > > > >>>> > > > > > > - it doesn't map to NoSQL > > > > >>>> > > > > > > - It wraps single-row updates in > > runWiithinTransaction, > > > > >>>> which is a > > > > >>>> > > > > > overhead > > > > >>>> > > > > > > > > > > >>>> > > > > > > So, I think the transactional manager isn't a portable > > > > >>>> target. It's > > > > >>>> > > > > > > "only" one backend's strategy. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I think Privthi's approach is right. A > > backend-agnostic > > > > >>>> change-set > > > > >>>> > > > > > > primitive with a documented fallback is the correct > > > shape. > > > > >>>> There is > > > > >>>> > > > > > > one caveat: without carrying the original entitiy (not > > > > just > > > > >>>> the new > > > > >>>> > > > > > > one) the change set can't express optimistic > > > concurrency. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps approach: > > > > >>>> > > > > > > 1. We write the manager-level consistency contract as > > > > >>>> Robert asked. > > > > >>>> > > > It > > > > >>>> > > > > > > should include the explicit statement that a logical > > > > change > > > > >>>> set is > > > > >>>> > > > not > > > > >>>> > > > > > > a request-scope DB transaction. > > > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency > > > > pattern) > > > > >>>> > > baseline > > > > >>>> > > > > > > first-class in EntityMutation before merging the SPI > > > > >>>> > > > > > > 3. Then we refactor > > > createCatalog/dropEntity/renameEntity. > > > > >>>> > > > > > > > > > > >>>> > > > > > > Thoughts? > > > > >>>> > > > > > > > > > > >>>> > > > > > > Regards > > > > >>>> > > > > > > JB > > > > >>>> > > > > > > > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > > > >>>> [email protected]> > > > > >>>> > > wrote: > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Hi all, > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity > > > > guarantee > > > > >>>> is > > > > >>>> > > > defined > > > > >>>> > > > > > > seems > > > > >>>> > > > > > > > important. > > > > >>>> > > > > > > > If it is a property of the lower-level > > BasePersistence > > > > >>>> contract > > > > >>>> > > > > rather > > > > >>>> > > > > > > than > > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the > > > > general > > > > >>>> > > contract > > > > >>>> > > > > does > > > > >>>> > > > > > > not > > > > >>>> > > > > > > > tell callers whether the state used for validation, > > > > >>>> > > authorization, > > > > >>>> > > > or > > > > >>>> > > > > > > > credential vending is consistent with the change > > that > > > > >>>> eventually > > > > >>>> > > > > > commits. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > The current PRs suggest that operation-specific > > > > >>>> multi-object > > > > >>>> > > > methods > > > > >>>> > > > > > can > > > > >>>> > > > > > > > fix individual cases, while leaving the same > > contract > > > > >>>> question to > > > > >>>> > > > > recur > > > > >>>> > > > > > > for > > > > >>>> > > > > > > > each new case. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical change set as > > a > > > > >>>> database > > > > >>>> > > > > > > transaction > > > > >>>> > > > > > > > around an entire REST request. > > > > >>>> > > > > > > > That would tie the contract to the backend and > > > database, > > > > >>>> and > > > > >>>> > > could > > > > >>>> > > > > keep > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > durable attempt open across slow external work. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > > > manager > > > > >>>> > > > > > implementations, > > > > >>>> > > > > > > or > > > > >>>> > > > > > > > do we first need to revisit the boundary and > > > guarantees > > > > >>>> exposed > > > > >>>> > > to > > > > >>>> > > > > > their > > > > >>>> > > > > > > > callers? > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Cheers, > > > > >>>> > > > > > > > Robert > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri > > Bourlatchkov < > > > > >>>> > > > > [email protected] > > > > >>>> > > > > > > > > > > >>>> > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > Hi Yufei, > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one > > > > atomic > > > > >>>> change > > > > >>>> > > > per > > > > >>>> > > > > > > method" > > > > >>>> > > > > > > > > contract as defined in javadoc does apply only to > > > > >>>> > > BasePersistence > > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which > > delegates > > > > to > > > > >>>> > > > > > > BasePersistence). > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Note that all non-test Persistence implementations > > > in > > > > >>>> the > > > > >>>> > > Polaris > > > > >>>> > > > > > > codebase > > > > >>>> > > > > > > > > extend those classes (which is probably why I was > > > > >>>> confused > > > > >>>> > > about > > > > >>>> > > > > > > atomicity > > > > >>>> > > > > > > > > expectations). > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > However, this creates a gap in the Persistence SPI > > > > >>>> > > specification. > > > > >>>> > > > > If > > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do > > not > > > > >>>> have to > > > > >>>> > > > comply > > > > >>>> > > > > > > with > > > > >>>> > > > > > > > > this principle, it will create a conceptual > > > difficulty > > > > >>>> at call > > > > >>>> > > > > sites. > > > > >>>> > > > > > > How > > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about > > > > >>>> consistency > > > > >>>> > > and > > > > >>>> > > > > > > durability > > > > >>>> > > > > > > > > behaviours in general? > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > I believe we need to address that as part of this > > > > >>>> discussion. > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Cheers, > > > > >>>> > > > > > > > > Dmitri. > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < > > > > >>>> [email protected] > > > > >>>> > > > > > > > >>>> > > > > > wrote: > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with > > the > > > > >>>> idea that > > > > >>>> > > one > > > > >>>> > > > > > > method > > > > >>>> > > > > > > > > call > > > > >>>> > > > > > > > > > means one atomic change. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface > > > > >>>> > > > > > > PolarisMetaStoreManager, I > > > > >>>> > > > > > > > > > don't think we've ever state each method to be > > > > >>>> atomic. We did > > > > >>>> > > > > > clarify > > > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence > > > > though. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri > > > > Bourlatchkov < > > > > >>>> > > > > > > [email protected]> > > > > >>>> > > > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be handled > > > > more > > > > >>>> > > > explicitly. > > > > >>>> > > > > > > > > However, > > > > >>>> > > > > > > > > > > I'm not sure that simply moving to > > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is > > > > >>>> > > > > > > > > > > sufficient. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with > > the > > > > >>>> idea that > > > > >>>> > > one > > > > >>>> > > > > > > method > > > > >>>> > > > > > > > > call > > > > >>>> > > > > > > > > > > means one atomic change [1]. The > > "transactional" > > > > >>>> MetaStore > > > > >>>> > > > > impl. > > > > >>>> > > > > > is > > > > >>>> > > > > > > > > but a > > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the > > high-level > > > > >>>> contract. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple > > object > > > > >>>> parameters > > > > >>>> > > to > > > > >>>> > > > > > > represent > > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will be > > a > > > > >>>> sound > > > > >>>> > > design. > > > > >>>> > > > > > This > > > > >>>> > > > > > > will > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra > > > > >>>> impl. effort > > > > >>>> > > > for > > > > >>>> > > > > > > each > > > > >>>> > > > > > > > > > > backend type. More importantly, adding > > multi-arg > > > > >>>> change > > > > >>>> > > > methods > > > > >>>> > > > > > > still > > > > >>>> > > > > > > > > > won't > > > > >>>> > > > > > > > > > > address the problem of reads being consistent > > > with > > > > >>>> writes, > > > > >>>> > > > > > because > > > > >>>> > > > > > > each > > > > >>>> > > > > > > > > > > method call will still be independent > > regarding > > > > the > > > > >>>> data > > > > >>>> > > > stored > > > > >>>> > > > > > in > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > > > > database. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a "change > > > > set" > > > > >>>> or > > > > >>>> > > > "atomic > > > > >>>> > > > > > > batch" > > > > >>>> > > > > > > > > > > concept to core Persistence and associate each > > > > REST > > > > >>>> API > > > > >>>> > > > request > > > > >>>> > > > > > > with > > > > >>>> > > > > > > > > one > > > > >>>> > > > > > > > > > > such change set, which will be committed (or > > > > rolled > > > > >>>> back) > > > > >>>> > > at > > > > >>>> > > > > the > > > > >>>> > > > > > > end of > > > > >>>> > > > > > > > > > the > > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar > > > > >>>> concept in PR > > > > >>>> > > > 4939 > > > > >>>> > > > > > > [2]. In > > > > >>>> > > > > > > > > > > JDBC each change set will naturally be > > > associated > > > > >>>> with an > > > > >>>> > > > RDBMS > > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic > > > > >>>> change set > > > > >>>> > > > will > > > > >>>> > > > > be > > > > >>>> > > > > > > > > > > associated with one CAS operation on the > > > > underlying > > > > >>>> > > database. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [1] > > > > >>>> > > > > > > > > > > >>>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > > > >>>> > > > > > > > > > > >>>> > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > > > >>>> > > > > > > > > > > Dmitri. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > > > >>>> > > > [email protected] > > > > >>>> > > > > > > > > > >>>> > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were > > > already > > > > >>>> > > recognized > > > > >>>> > > > > when > > > > >>>> > > > > > > we > > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At > > > that > > > > >>>> time, we > > > > >>>> > > > > chose > > > > >>>> > > > > > > to use > > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC > > > due > > > > >>>> to the > > > > >>>> > > > > > > simplicity. I > > > > >>>> > > > > > > > > > > think > > > > >>>> > > > > > > > > > > > most of the issues mentioned here can > > already > > > be > > > > >>>> > > addressed > > > > >>>> > > > by > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in a > > > > >>>> transaction > > > > >>>> > > in > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > > > Similarly, > > > > >>>> catalog > > > > >>>> > > > > > > creation, > > > > >>>> > > > > > > > > > which > > > > >>>> > > > > > > > > > > > involves reading and creating multiple > > > objects, > > > > >>>> is also > > > > >>>> > > > > > executed > > > > >>>> > > > > > > > > > within a > > > > >>>> > > > > > > > > > > > transaction [2]. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager > > > > >>>> together with > > > > >>>> > > the > > > > >>>> > > > > > > > > persistence > > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the > > > > required > > > > >>>> > > > > consistency > > > > >>>> > > > > > > > > > guarantees > > > > >>>> > > > > > > > > > > > for > > > > >>>> > > > > > > > > > > > specific operations, similar to what > > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl > > > > >>>> > > > > > > > > > > > does. > > > > >>>> > > > > > > > > > > > 2. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such as > > > > >>>> JDBC) to use > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > directly. > > > > We > > > > >>>> may > > > > >>>> > > have > > > > >>>> > > > to > > > > >>>> > > > > > > deal > > > > >>>> > > > > > > > > with > > > > >>>> > > > > > > > > > > > transactional semantic mismatches across > > > > >>>> different > > > > >>>> > > > > > persistence > > > > >>>> > > > > > > > > > > backends. > > > > >>>> > > > > > > > > > > > For example, we would likely avoid using > > > > JDBC's > > > > >>>> > > > > > > > > > `runWithinTransaction` > > > > >>>> > > > > > > > > > > > for > > > > >>>> > > > > > > > > > > > single row updates, which adds additional > > > > >>>> overhead and > > > > >>>> > > > > > > complexity > > > > >>>> > > > > > > > > > > > without > > > > >>>> > > > > > > > > > > > benefits. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > > > >>>> > > > > > > > > > > > 2. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > > > >>>> Bourlatchkov < > > > > >>>> > > > > > > > > [email protected]> > > > > >>>> > > > > > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi all, > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a > > > > couple > > > > >>>> of > > > > >>>> > > > > > interesting > > > > >>>> > > > > > > PRs: > > > > >>>> > > > > > > > > > > > > [4939], [5035]. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > It looks like people are starting to > > > encounter > > > > >>>> > > > consistency > > > > >>>> > > > > > > issues > > > > >>>> > > > > > > > > in > > > > >>>> > > > > > > > > > > > > JDBC persistence. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into the > > > > >>>> underlying > > > > >>>> > > > > issues. > > > > >>>> > > > > > > They > > > > >>>> > > > > > > > > > offer > > > > >>>> > > > > > > > > > > > > incremental fixes that can work. However, > > I > > > > >>>> believe it > > > > >>>> > > is > > > > >>>> > > > > > time > > > > >>>> > > > > > > for > > > > >>>> > > > > > > > > > the > > > > >>>> > > > > > > > > > > > > Polaris community to review and improve > > this > > > > >>>> area of > > > > >>>> > > the > > > > >>>> > > > > > > codebase > > > > >>>> > > > > > > > > > > > > holistically. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that > > can > > > be > > > > >>>> applied > > > > >>>> > > to > > > > >>>> > > > > all > > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, > > > NoSQL) > > > > >>>> and > > > > >>>> > > > addresses > > > > >>>> > > > > > > these > > > > >>>> > > > > > > > > > > > > aspects: > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > > > changes > > > > >>>> where > > > > >>>> > > the > > > > >>>> > > > > > > service > > > > >>>> > > > > > > > > > reads > > > > >>>> > > > > > > > > > > > > and validates current catalog state, > > then > > > > >>>> commits a > > > > >>>> > > > > change > > > > >>>> > > > > > > (e.g. > > > > >>>> > > > > > > > > > > > > name clashes during renames). > > > > >>>> > > > > > > > > > > > > * Supporting consistent but independent > > > > changes > > > > >>>> to RBAC > > > > >>>> > > > > > grants > > > > >>>> > > > > > > and > > > > >>>> > > > > > > > > > > > > MetaStore entities. This independence is > > > > >>>> needed to > > > > >>>> > > > > support > > > > >>>> > > > > > > > > > > > > external authorizers like OPA and > > Ranger. > > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across > > multiple > > > > >>>> similar > > > > >>>> > > > > entities. > > > > >>>> > > > > > > > > > > > > * Supporting authorization-based filtering > > > of > > > > >>>> list > > > > >>>> > > > > operations > > > > >>>> > > > > > > (cf. > > > > >>>> > > > > > > > > > > > > [4831]). > > > > >>>> > > > > > > > > > > > > * Supporting credential-vending decisions > > > that > > > > >>>> are > > > > >>>> > > rooted > > > > >>>> > > > > in > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > > > > > > exact state of the catalog. > > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for > > > transient > > > > >>>> > > > persistence > > > > >>>> > > > > > > failures > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability > > failures). > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4831] > > > > >>>> https://github.com/apache/polaris/pull/4831 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > > > >>>> https://github.com/apache/polaris/pull/4939 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > > > >>>> https://github.com/apache/polaris/pull/5035 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > > > >>>> > > > > > > > > > > > > Dmitri > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > >>> > > > > > > > > >
