Hi Dennis, This is not a reply to your email, but a quick note regarding PolarisResolutionManifest.
>From my POV, the most fundamental divergence from the idea it represents is the "optimized" location check, which goes directly to persistence and does not work with the resolver or manifest on the request side. That is, it takes one entity on input and finds conflicts with other entities via low-level persistence operations. Cheers, Dmitri. On Wed, Sep 30, 2026 at 8:42 PM Dennis Huo <[email protected]> wrote: > Agree we should formulate in terms of Persistence SPIs first. > > IIRC we talked about this in one of the community syncs but I forgot to > bring it here -- the PolarisResolutionManifest in concept already > represents *exactly* the set of "which elements were engaged during request > processing". > > Adherence may have drifted, but at least early on, we made sure all the > IcebergCatalog logic did *not* get to do new direct reads from the > persistence layer, but instead needs to "read" from the *authorized* set of > entities contained in a PolarisResolutionManifest (or declared as a > "passthroughPath" == allow reading fresh from DB but had to pre-register it > for authz purposes). > > It's important to keep this unified anyways because it really serves two > things: > > 1. Declares the set of entities whose state you expect to resolve/snapshot > up-front in the operation-handling, that the IcebergCatalog logic then > operates on later; also explicitly declare which ones require fetching > *fresh* versions mid-operation (addPassthroughPath) > 2. Enforces that RBAC authz is performed against that pre-declared set of > entities -- if later processign were allowed to access arbitrary other > entities to make handler decisions, then how did we "prove" that the > authorization engine actually allowed the request to access those arbitrary > other entities? Even if it's just for internal-processing purposes, the > very fact that something got accessed means there's a route to accidental > leakage of unauthorized data. > > So in a way, solid/provable authz and solid/provable consistency go > hand-in-hand. > > I think PolarisResolutionManifest in its current form is probably a bit > opaque and doesn't 100% convey the semantics we want quite yet, but it > seems like the right SPI starting point to extend or at least evolve from. > Basically, aside from addPath and addPassthroughPath, there could be a > notion of annotating a given path as being one of the "dependent states" to > carry over into the later commit. Conceptually: > > // Prepare operation processing > resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime = > false); // For this operation let's pretend we don't care if the root > catalog changes before we commit the change to the tables > resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime = > false); // For this operation let's pretend we don't care if the parent > namespace changes before we commit the change to the tables > resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true); > // Assume multi-table commit - table1 will have changes and also must not > have already changed at commit time > resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true); > // Same for table2 > > // .... Do all the Iceberg processing work > > // When we commit, we use the resolutionManifest as the ledger of things we > used that must still be at their unchanged version at time of commit > metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated), > resolutionManifest.getSetOfUnchangedEntityRequirements()); > > Overall this fits the optimistic-concurrency theme of Iceberg and Polaris > in general, and would *intentionally* not try to express a full-fledged > "Transaction" session in which you do a bunch of reads/writes/reads/writes > on a single entity as if you're operating in an uncommitted block on a > RDBMS, instead opting for the middle-ground of "multi-object conditional > commits". > > On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi EJ, > > > > > 1. Which backend assumptions should we design for? [...] > > > > I know I used the term "transaction" a lot during the call, but the real > > challenge is about establishing an SPI with clear contracts that could be > > implemented by JDBC and NoSQL persistence using the database features > > natural to each model. > > > > I am fairly confident that a JDBC impl. is possible with Serializable > > transaction. We should be able to solve all the problems raised in this > > thread. However, that will make in-memory caching impossible because the > > database needs to see all the reads (and writes) in a Tx to be able to > > ensure serializable guarantees. If a cache prevents a read from hitting > the > > database, that read (e.g. storage config) will not be a factor in > > serializability. > > > > This connects to my earlier question about the in-memory cache and JDBC > > (title: Usefulness of InMemoryEntityCache) > > > > Are different approaches possible with JDBC? Probably. What is involved? > > This depends on the SPI contract, I think. > > > > > 2. How do we protect the data used by validation? [...] > > > > The point mentioned by Robert previously IIRC, is that we need to ground > > all runtime decisions to a particular state of the catalog. > > > > If at commit time the state of the catalog in the database shifts from > what > > was used during validation, the request has to restart from scratch. > > > > Current JDBC Persistence does this only for relationships between > entities > > and grants and between versions of the same entity. > > > > Relationships between table locations and storage config are not > actualized > > for the purpose of state consistency tracking. > > > > From my POV we need to begin by formulating this Persistence state in the > > Persistence SPI terms. Then, think how we can implement that for JDBC. > > > > NoSQL Persistence already has a per-catalog state (not granular), so > wiring > > the new SPI to NoSQL should not be too hard, I hope. > > > > The state in the SPI does not have to be global to the catalog (as in > > NoSQL), but it needs to be able to express the idea of which elements > were > > engaged during request processing so serve as input to Persistence impl. > > for cosistency guarantees. If Persistence caches data in memory (for > > example), it will have to do so in a way that does not violate > consistency > > expectations. > > > > From my POV we could begin by defining a "Persistence Session" concept. > > Each request is 1:1 with a session. All reads and writes go through the > > session. The next step would be the JDBC impl., which probably needs some > > brainstorming... but I wonder if we could reach a consensus at a high > level > > first. > > > > WDYT? > > > > Thanks, > > Dmitri. > > > > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang <[email protected]> > > wrote: > > > > > Thanks for the recap, Dmitri. I'd like to follow up on the three > points I > > > raised during the sync. We didn't have much time to discuss them, so it > > > would help to confirm these before choosing an SPI design. > > > > > > 1. Which backend assumptions should we design for? > > > > > > Within one Polaris server instance, do we assume entities, grants and > > > secrets all use the same database backend, or must we support mixing > > > backends across them? > > > Must every supported persistence implementation provide transactional > > > guarantees? Does the database itself need to provide transactions, or > can > > > Polaris build that capability on top of simpler database operations? > > > > > > I'm happy to work with a homogeneous assumption if that's the intended > > > scope. Existing NoSQL illustrates the second question: it can publish > > > changes to multiple objects under one reference using a single > > conditional > > > pointer update. > > > > > > 2. How do we protect the data used by validation? > > > > > > Your storage-configuration and location-overlap examples make the > problem > > > concrete. Consider these races: > > > > > > A table creation passes validation against the catalog's allowed > > locations, > > > then another request changes that configuration before the table is > > > committed. > > > Two table-creation requests each check for overlapping locations, both > > see > > > no overlap, and then create tables at conflicting locations. > > > > > > Checking only the record being written cannot cover these dependencies. > > The > > > first also depends on the catalog configuration. The second depends on > > the > > > absence of another overlapping table. > > > > > > Could the persistence layer protect these reads together with the > writes, > > > using backend-internal versions/tokens or database isolation? Shared > code > > > would still need to identify the required reads and checks. Could we > keep > > > the mechanism for protecting them inside each backend? > > > > > > 3. Business rules are duplicated across backend implementations > > > > > > Today, the TreeMap > > > < > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > > > > > and JDBC > > > < > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > > > > > implementations each load a secret, check its principal, apply the > > > rotation/reset steps and write it back. > > > > > > Changing these business rules therefore requires keeping multiple > > > implementations in sync. A fix applied to one backend can be missed in > > > another, causing the same Polaris operation to behave differently > > depending > > > on the database. > > > > > > My proposal is to implement each business workflow once, against a > common > > > set of storage operations with defined consistency and failure > semantics. > > > The shared workflow should not need to know whether those operations > are > > > implemented through JDBC transactions, NoSQL reference CAS or another > > > mechanism. Each backend would implement the same storage contract, > > keeping > > > its execution details internal. A new database backend could then reuse > > the > > > existing business workflows. > > > > > > Do these seem like reasonable starting points? We can then evaluate the > > SPI > > > against the original failure cases on both JDBC and existing NoSQL. > > > > > > -ej > > > > > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > Side note: I just came across [5541], which shows a case where a > failed > > > > request leaves persisted side effects. > > > > > > > > This can be seen as a coding mistake, of course. However, this is > about > > > > executing one request as a single, atomic unit. This is something > that > > > > probably needs foundational support and coding patterns in Polaris > > code. > > > > > > > > Cheers, > > > > Dmitri. > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Here's a recap of the Community Sync discussion as I remember it. > I'm > > > > sure > > > > > my recollection is not complete, so please add your thoughts to > this > > > > thread. > > > > > > > > > > * Polaris traditionally relies on entity version number to ensure > > > > > consistency of changes. > > > > > > > > > > This works well for internal RBAC grants, which also have version > > > numbers > > > > > cross-referenced at the Persistence layer. > > > > > > > > > > I do not think this works in more general cases like validating > > > locations > > > > > wrt catalog-level storage configuration. > > > > > > > > > > Another difficult case involves location overlaps between > concurrent > > > > table > > > > > creations. > > > > > > > > > > * We talked about how Persistence calls related to RDBMS > transactions > > > (in > > > > > the JDBC persistence case). > > > > > > > > > > There were some concerns raised about whether we need to bind > > > Persistence > > > > > API to transactions explicitly. > > > > > > > > > > I think it is a valid concern. At the same time, the Persistence > API > > > > > should be clear about consistency guarantees across all backend > > > > > implementations. I think this needs more attention now that the > > matter > > > of > > > > > milti-entioty changes became prominent in [5035]. > > > > > > > > > > I personally believe that RDBMS-based Persistence in Polaris has to > > use > > > > > Serializable transaction isolation. Consequently, the core code > needs > > > to > > > > > provide mechanisms to allow the Persistence implementation to > > properly > > > > > connect reads and writes from API requests to JDBC transactions... > > > > whether > > > > > we use "transation" as an explicit terms in the SPI or not. > > > > > > > > > > Note that currently, the same API request performs reads and writes > > in > > > > > _separate_ RDBMS transactions (even separate connections). > > > > > > > > > > * We did not discuss Persistence SPI implications in the call, but > > > > > currently each Persistence call is assumed to be one distinct and > > > atomic > > > > > change (please correct me if I'm wrong). > > > > > > > > > > Consequently, supporting new use cases involves introducing new SPI > > > > > methods with many parameters. This can be seen in PR [5035]. This > > tends > > > > to > > > > > overcomplicate the SPI. > > > > > > > > > > Ideally, I think the SPI should be simplified to deal with > > multi-entity > > > > > and single-entity changes in a coherent manner to avoid any > ambiguity > > > > about > > > > > which method the caller should use. > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > >> Heads up: This discussion is on the Community Sync call agenda for > > > Sept > > > > >> 17. > > > > >> > > > > >> Interested parties, please plan to attend, if possible :) > > > > >> > > > > >> Cheers, > > > > >> Dmitri. > > > > >> > > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov < > > [email protected]> > > > > >> wrote: > > > > >> > > > > >>> Hi JB, > > > > >>> > > > > >>> I've added an agenda item to the next community sync call for > this > > > > (Sept > > > > >>> 17, 2026). > > > > >>> > > > > >>> Re: key points: did you mean a real GH discussion or dev email > > (this > > > > >>> thread)? Just double checking :) I'm fine with either approach. > > > > >>> > > > > >>> Cheers, > > > > >>> Dmitri. > > > > >>> > > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < > > > [email protected]> > > > > >>> wrote: > > > > >>> > > > > >>>> Hi Dmitri, > > > > >>>> > > > > >>>> I agree on the need to align on these points, though I'm not > > > entirely > > > > >>>> sure a dedicated meeting is necessary. Let's start by using some > > > time > > > > >>>> during the next community meeting to discuss it. > > > > >>>> > > > > >>>> In the meantime, what if we outline the key points in a GitHub > > > > >>>> Discussion first? We can then use that as a reference during the > > > > >>>> meeting. > > > > >>>> > > > > >>>> Regards, > > > > >>>> JB > > > > >>>> > > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > >>>> wrote: > > > > >>>> > > > > > >>>> > Hi All, > > > > >>>> > > > > > >>>> > My impression from this thread is that it might be time for a > > call > > > > to > > > > >>>> > discuss all the related issues and try to arrive at a shared > > > > >>>> implementation > > > > >>>> > plan. > > > > >>>> > > > > > >>>> > I know meetings are not ideal, but in this case it might be > > > > >>>> beneficial as a > > > > >>>> > means for achieving a common understanding of the set of > > problems > > > > and > > > > >>>> > priorities related to this thread. A dedicated metrics meeting > > > > worked > > > > >>>> well > > > > >>>> > from my POV. > > > > >>>> > > > > > >>>> > Allocating a time slot in the next community meeting might be > an > > > > >>>> option, > > > > >>>> > although I think we might need the full hour in this case. > > > > >>>> > > > > > >>>> > WDYT? > > > > >>>> > > > > > >>>> > Thanks, > > > > >>>> > Dmitri. > > > > >>>> > > > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S < > > > > >>>> [email protected]> > > > > >>>> > wrote: > > > > >>>> > > > > > >>>> > > Hi all, > > > > >>>> > > > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, > > > following > > > > >>>> the > > > > >>>> > > discussion to focus only on the SPI foundation. > > > > >>>> > > > > > > >>>> > > What is in the PR now: > > > > >>>> > > > > > > >>>> > > - A written manager-level consistency contract in > > > > >>>> > > > site/content/in-dev/unreleased/persistence-consistency.md. > > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records. > > > > >>>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, > > > with > > > > >>>> backends > > > > >>>> > > opting in via supportsAtomicMixedCommit(). > > > > >>>> > > - commitChangeSet implementations for the in-memory > TreeMap > > > > >>>> backend and > > > > >>>> > > JDBC. > > > > >>>> > > - AtomicOperationMetaStoreManager and > > > > >>>> TransactionalMetaStoreManagerImpl > > > > >>>> > > grant/revoke paths now build one change set per > operation, > > > > >>>> falling back > > > > >>>> > > to > > > > >>>> > > individual operations when the backend does not support > > mixed > > > > >>>> commits. > > > > >>>> > > - Tests for the change-set API and TreeMap atomic > commits. > > > > >>>> > > > > > > >>>> > > I intentionally left out: > > > > >>>> > > > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > > > >>>> > > - Refactoring createCatalog, dropEntity, and renameEntity > > to > > > > use > > > > >>>> change > > > > >>>> > > sets. > > > > >>>> > > - NoSQL support for commitChangeSet. > > > > >>>> > > > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI > > > > primitive > > > > >>>> can be > > > > >>>> > > reviewed before the larger operation migrations land. > > > > >>>> > > > > > > >>>> > > Could you please take a look? In particular, I would like to > > > know > > > > >>>> whether > > > > >>>> > > the contract captures the consensus so far, or if it needs > to > > > > >>>> address retry > > > > >>>> > > / external-work coordination before we merge this > foundation. > > > > >>>> > > > > > > >>>> > > Thanks, > > > > >>>> > > Prithvi S > > > > >>>> > > > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < > > > > >>>> [email protected]> > > > > >>>> > > wrote: > > > > >>>> > > > > > > >>>> > > > Hi Robert, > > > > >>>> > > > > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API > > > > request) > > > > >>>> > > consists > > > > >>>> > > > of many distinct phases. The database / persistence > changes > > > are > > > > >>>> just one > > > > >>>> > > of > > > > >>>> > > > these phases. STS (as an example) is another. My point > about > > > > >>>> transactions > > > > >>>> > > > in JDBC Persistence applies to the former (database > changes) > > > > >>>> phase - one > > > > >>>> > > > retry attempt there should ideally involve exactly one > > > > >>>> transaction. > > > > >>>> > > > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > > > coordinate > > > > >>>> requests > > > > >>>> > > to > > > > >>>> > > > various external systems depending on outcomes from > previous > > > > >>>> request > > > > >>>> > > > processing phases. > > > > >>>> > > > > > > > >>>> > > > I also agree that validation based on database state > should > > be > > > > >>>> repeated > > > > >>>> > > > from scratch if we retry the database changes. > > > > >>>> > > > > > > > >>>> > > > However, internal RBAC authorization naturally has to > happen > > > > >>>> inside the > > > > >>>> > > > database update phase (read-check-write). IIRC, that is > part > > > of > > > > >>>> the > > > > >>>> > > > implicit consistency guarantees, which were discussed > during > > > > early > > > > >>>> > > project > > > > >>>> > > > months (but never got written down in full clarity, > > > > >>>> unfortunately). If > > > > >>>> > > the > > > > >>>> > > > Authorization check is inside that phase for internal > RBAC, > > I > > > > >>>> suppose it > > > > >>>> > > > will be there for other Authorizers too and that can > involve > > > > >>>> external > > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect > > > > >>>> authorization to be > > > > >>>> > > > efficient, but I'm not sure how quick it can be in > practice. > > > > >>>> There's > > > > >>>> > > > certainly potential for delays in some (perhaps > infrequent) > > > > cases. > > > > >>>> > > > > > > > >>>> > > > Cheers, > > > > >>>> > > > Dmitri. > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp < > > [email protected]> > > > > >>>> wrote: > > > > >>>> > > > > > > > >>>> > > > > Hi Dmitri, > > > > >>>> > > > > > > > > >>>> > > > > I agree that one logical operation needs a consistent > view > > > of > > > > >>>> the whole > > > > >>>> > > > > backend state. > > > > >>>> > > > > What worries me is treating one long JDBC transaction as > > the > > > > >>>> boundary > > > > >>>> > > of > > > > >>>> > > > > that operation and then retrying the complete request if > > the > > > > >>>> commit > > > > >>>> > > > fails. > > > > >>>> > > > > > > > > >>>> > > > > The request can also write metadata to object storage or > > > call > > > > >>>> STS. > > > > >>>> > > > > Each system can leave us with an outcome whose certainty > > we > > > > >>>> cannot > > > > >>>> > > know. > > > > >>>> > > > > The database may have committed before the connection > was > > > > lost, > > > > >>>> an > > > > >>>> > > > > object-store write may have succeeded before a timeout, > or > > > STS > > > > >>>> may have > > > > >>>> > > > > issued credentials before its response was lost. > > > > >>>> > > > > A database rollback cannot undo any of those effects, > and > > > > >>>> retrying the > > > > >>>> > > > > whole request may repeat them. > > > > >>>> > > > > > > > > >>>> > > > > So I think the contract has to separate the overall > > > operation > > > > >>>> from each > > > > >>>> > > > > attempt to commit it. > > > > >>>> > > > > Each database or backend commit attempt should stay > short. > > > > >>>> > > > > When we retry, validation and authorization need to use > > the > > > > >>>> state for > > > > >>>> > > > that > > > > >>>> > > > > new attempt, and we need clear rules for which external > > work > > > > >>>> can be > > > > >>>> > > > reused, > > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain > > > > outcome. > > > > >>>> > > > > > > > > >>>> > > > > Some contextual state could be useful for carrying that > > > state. > > > > >>>> > > > > But the JDBC connection and transaction should be an > > > > >>>> implementation > > > > >>>> > > > detail > > > > >>>> > > > > inside it, not the boundary of the complete operation. > > > > >>>> > > > > > > > > >>>> > > > > Cheers, > > > > >>>> > > > > Robert > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > > > >>>> [email protected] > > > > >>>> > > > > > > > >>>> > > > > wrote: > > > > >>>> > > > > > > > > >>>> > > > > > Hi All, > > > > >>>> > > > > > > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive > > sounds > > > > >>>> good. > > > > >>>> > > > > > > > > > >>>> > > > > > However, I am not sure it is sufficient for all the > use > > > > cases > > > > >>>> we > > > > >>>> > > > touched > > > > >>>> > > > > > here. More specifically, the state read by validation > > code > > > > is > > > > >>>> not > > > > >>>> > > > > > necessarily reflected in the change set: > > > > >>>> > > > > > > > > > >>>> > > > > > * Unchanged entities might be considered by > validation, > > > but > > > > >>>> changed > > > > >>>> > > in > > > > >>>> > > > a > > > > >>>> > > > > > parallel request. > > > > >>>> > > > > > * Even for changed entities the state read by > validation > > > may > > > > >>>> differ > > > > >>>> > > > from > > > > >>>> > > > > > the state being altered. Some validation code talks to > > the > > > > >>>> MetaStore > > > > >>>> > > > > > directly, outside of the data produced by the > Resolver. > > > > >>>> > > > > > > > > > >>>> > > > > > I do not think Polaris offers any explicit mechanisms > > > (ATM) > > > > >>>> to ensure > > > > >>>> > > > > > consistency between these reads and subsequent writes. > > > > >>>> > > > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching > Transaction > > > > >>>> across all > > > > >>>> > > > reads > > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE > > > > isolation > > > > >>>> level > > > > >>>> > > in > > > > >>>> > > > > the > > > > >>>> > > > > > backing RDBMS could solve the problem, I think. > > > > >>>> > > > > > > > > > >>>> > > > > > Indeed, such a transaction might be long to > accommodate > > > > calls > > > > >>>> made by > > > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > > > problem? > > > > >>>> I think > > > > >>>> > > > the > > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to > > > "manually" > > > > >>>> track all > > > > >>>> > > > > reads > > > > >>>> > > > > > under the same request and redo them in the small > > > > transaction > > > > >>>> that > > > > >>>> > > > > persists > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and > > require > > > > >>>> complex > > > > >>>> > > code > > > > >>>> > > > in > > > > >>>> > > > > > Polaris to handle the data properly. > > > > >>>> > > > > > > > > > >>>> > > > > > Tracking the request-wide transaction can be done only > > for > > > > >>>> JDBC > > > > >>>> > > without > > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL > > Persistence, I > > > > >>>> think. > > > > >>>> > > NoSQL > > > > >>>> > > > > > will use other mechanisms to ensure read/write > > > consistency. > > > > >>>> > > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in > this > > > > >>>> discussion), > > > > >>>> > > > I'd > > > > >>>> > > > > > like to propose this approach: > > > > >>>> > > > > > > > > > >>>> > > > > > * Each request establishes a "Data Context" > > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC > > > > Connection > > > > >>>> + Tx > > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more > > reference > > > > >>>> hashes > > > > >>>> > > > > > * All Persistence access in the same request goes > > through > > > > the > > > > >>>> same > > > > >>>> > > Data > > > > >>>> > > > > > Context > > > > >>>> > > > > > * All Persistence changes are committed once at the > end > > of > > > > the > > > > >>>> > > request > > > > >>>> > > > > > - Not all changes have to be globally atomic. For > > > example, > > > > >>>> changes > > > > >>>> > > in > > > > >>>> > > > > > Catalogs A and B do not have to be atomic with respect > > to > > > > >>>> each other. > > > > >>>> > > > We > > > > >>>> > > > > > can go deeper into this later. This is relevant to > > NoSQL. > > > > >>>> > > > > > - Transactional backends like JDBC can, of course, > > > choose > > > > >>>> to make > > > > >>>> > > all > > > > >>>> > > > > > changes globally atomic. > > > > >>>> > > > > > * A commit can fail in two main ways: > > > > >>>> > > > > > - A retriable failure like an optimistic lock error > or > > > Tx > > > > >>>> > > > > serializability > > > > >>>> > > > > > error > > > > >>>> > > > > > - A non-triable logical error (e.g. entity not > found) > > > > >>>> > > > > > * On a retriable error the whole request is > re-attempted > > > (as > > > > >>>> if > > > > >>>> > > > > resubmitted > > > > >>>> > > > > > by a client) a few times (configurable timeout). > > > > >>>> > > > > > > > > > >>>> > > > > > WDYT? > > > > >>>> > > > > > > > > > >>>> > > > > > Cheers, > > > > >>>> > > > > > Dmitri. > > > > >>>> > > > > > > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré > < > > > > >>>> > > [email protected] > > > > >>>> > > > > > > > > >>>> > > > > > wrote: > > > > >>>> > > > > > > > > > >>>> > > > > > > Hi all > > > > >>>> > > > > > > > > > > >>>> > > > > > > I think Robert has a good point. > > > > >>>> > > > > > > > > > > >>>> > > > > > > If the atomicity guarantee lives only in > > > BasePersistence, > > > > >>>> then the > > > > >>>> > > > > > > manager contract can't tell a caller whether the > state > > > it > > > > >>>> read for > > > > >>>> > > > > > > validation/authorization/credential-vending is the > > same > > > > >>>> state that > > > > >>>> > > > > > > eventually commits. That's the actual bug class > behind > > > the > > > > >>>> JDBC > > > > >>>> > > > > > > symptoms (not any single operation being non-atomic, > > but > > > > the > > > > >>>> > > contract > > > > >>>> > > > > > > being silent about it. Every operation-specific fix > > > (like > > > > >>>> #4939, > > > > >>>> > > > #5035 > > > > >>>> > > > > > > or #5095) re-answers this question locally and it > > keeps > > > > >>>> recurring. > > > > >>>> > > So > > > > >>>> > > > > > > the deliverable should start with a written > > consistency > > > > >>>> contract at > > > > >>>> > > > > > > the manager level. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > > > >>>> > > TransactionalMetaStoreManagerImpl > > > > >>>> > > > > > > is a good idea. It would tie the logical change set > > to a > > > > DB > > > > >>>> > > > > > > transaction spanning the REST request. It means: > > > > >>>> > > > > > > - it holds a durable transaction open across slow > > > external > > > > >>>> work > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > > > >>>> > > > > > > - it doesn't map to NoSQL > > > > >>>> > > > > > > - It wraps single-row updates in > > runWiithinTransaction, > > > > >>>> which is a > > > > >>>> > > > > > overhead > > > > >>>> > > > > > > > > > > >>>> > > > > > > So, I think the transactional manager isn't a > portable > > > > >>>> target. It's > > > > >>>> > > > > > > "only" one backend's strategy. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I think Privthi's approach is right. A > > backend-agnostic > > > > >>>> change-set > > > > >>>> > > > > > > primitive with a documented fallback is the correct > > > shape. > > > > >>>> There is > > > > >>>> > > > > > > one caveat: without carrying the original entitiy > (not > > > > just > > > > >>>> the new > > > > >>>> > > > > > > one) the change set can't express optimistic > > > concurrency. > > > > >>>> > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps approach: > > > > >>>> > > > > > > 1. We write the manager-level consistency contract > as > > > > >>>> Robert asked. > > > > >>>> > > > It > > > > >>>> > > > > > > should include the explicit statement that a logical > > > > change > > > > >>>> set is > > > > >>>> > > > not > > > > >>>> > > > > > > a request-scope DB transaction. > > > > >>>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency > > > > pattern) > > > > >>>> > > baseline > > > > >>>> > > > > > > first-class in EntityMutation before merging the SPI > > > > >>>> > > > > > > 3. Then we refactor > > > createCatalog/dropEntity/renameEntity. > > > > >>>> > > > > > > > > > > >>>> > > > > > > Thoughts? > > > > >>>> > > > > > > > > > > >>>> > > > > > > Regards > > > > >>>> > > > > > > JB > > > > >>>> > > > > > > > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > > > >>>> [email protected]> > > > > >>>> > > wrote: > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Hi all, > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity > > > > guarantee > > > > >>>> is > > > > >>>> > > > defined > > > > >>>> > > > > > > seems > > > > >>>> > > > > > > > important. > > > > >>>> > > > > > > > If it is a property of the lower-level > > BasePersistence > > > > >>>> contract > > > > >>>> > > > > rather > > > > >>>> > > > > > > than > > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, the > > > > general > > > > >>>> > > contract > > > > >>>> > > > > does > > > > >>>> > > > > > > not > > > > >>>> > > > > > > > tell callers whether the state used for > validation, > > > > >>>> > > authorization, > > > > >>>> > > > or > > > > >>>> > > > > > > > credential vending is consistent with the change > > that > > > > >>>> eventually > > > > >>>> > > > > > commits. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > The current PRs suggest that operation-specific > > > > >>>> multi-object > > > > >>>> > > > methods > > > > >>>> > > > > > can > > > > >>>> > > > > > > > fix individual cases, while leaving the same > > contract > > > > >>>> question to > > > > >>>> > > > > recur > > > > >>>> > > > > > > for > > > > >>>> > > > > > > > each new case. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical change set > as > > a > > > > >>>> database > > > > >>>> > > > > > > transaction > > > > >>>> > > > > > > > around an entire REST request. > > > > >>>> > > > > > > > That would tie the contract to the backend and > > > database, > > > > >>>> and > > > > >>>> > > could > > > > >>>> > > > > keep > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > durable attempt open across slow external work. > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > > > manager > > > > >>>> > > > > > implementations, > > > > >>>> > > > > > > or > > > > >>>> > > > > > > > do we first need to revisit the boundary and > > > guarantees > > > > >>>> exposed > > > > >>>> > > to > > > > >>>> > > > > > their > > > > >>>> > > > > > > > callers? > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > Cheers, > > > > >>>> > > > > > > > Robert > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri > > Bourlatchkov < > > > > >>>> > > > > [email protected] > > > > >>>> > > > > > > > > > > >>>> > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > Hi Yufei, > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The "one > > > > atomic > > > > >>>> change > > > > >>>> > > > per > > > > >>>> > > > > > > method" > > > > >>>> > > > > > > > > contract as defined in javadoc does apply only > to > > > > >>>> > > BasePersistence > > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which > > delegates > > > > to > > > > >>>> > > > > > > BasePersistence). > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Note that all non-test Persistence > implementations > > > in > > > > >>>> the > > > > >>>> > > Polaris > > > > >>>> > > > > > > codebase > > > > >>>> > > > > > > > > extend those classes (which is probably why I > was > > > > >>>> confused > > > > >>>> > > about > > > > >>>> > > > > > > atomicity > > > > >>>> > > > > > > > > expectations). > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > However, this creates a gap in the Persistence > SPI > > > > >>>> > > specification. > > > > >>>> > > > > If > > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations do > > not > > > > >>>> have to > > > > >>>> > > > comply > > > > >>>> > > > > > > with > > > > >>>> > > > > > > > > this principle, it will create a conceptual > > > difficulty > > > > >>>> at call > > > > >>>> > > > > sites. > > > > >>>> > > > > > > How > > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason about > > > > >>>> consistency > > > > >>>> > > and > > > > >>>> > > > > > > durability > > > > >>>> > > > > > > > > behaviours in general? > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > I believe we need to address that as part of > this > > > > >>>> discussion. > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Cheers, > > > > >>>> > > > > > > > > Dmitri. > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < > > > > >>>> [email protected] > > > > >>>> > > > > > > > >>>> > > > > > wrote: > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with > > the > > > > >>>> idea that > > > > >>>> > > one > > > > >>>> > > > > > > method > > > > >>>> > > > > > > > > call > > > > >>>> > > > > > > > > > means one atomic change. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the interface > > > > >>>> > > > > > > PolarisMetaStoreManager, I > > > > >>>> > > > > > > > > > don't think we've ever state each method to be > > > > >>>> atomic. We did > > > > >>>> > > > > > clarify > > > > >>>> > > > > > > > > > atomicity[1] in the interface BasePersistence > > > > though. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri > > > > Bourlatchkov < > > > > >>>> > > > > > > [email protected]> > > > > >>>> > > > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be > handled > > > > more > > > > >>>> > > > explicitly. > > > > >>>> > > > > > > > > However, > > > > >>>> > > > > > > > > > > I'm not sure that simply moving to > > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is > > > > >>>> > > > > > > > > > > sufficient. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined with > > the > > > > >>>> idea that > > > > >>>> > > one > > > > >>>> > > > > > > method > > > > >>>> > > > > > > > > call > > > > >>>> > > > > > > > > > > means one atomic change [1]. The > > "transactional" > > > > >>>> MetaStore > > > > >>>> > > > > impl. > > > > >>>> > > > > > is > > > > >>>> > > > > > > > > but a > > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the > > high-level > > > > >>>> contract. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple > > object > > > > >>>> parameters > > > > >>>> > > to > > > > >>>> > > > > > > represent > > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it will > be > > a > > > > >>>> sound > > > > >>>> > > design. > > > > >>>> > > > > > This > > > > >>>> > > > > > > will > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require > extra > > > > >>>> impl. effort > > > > >>>> > > > for > > > > >>>> > > > > > > each > > > > >>>> > > > > > > > > > > backend type. More importantly, adding > > multi-arg > > > > >>>> change > > > > >>>> > > > methods > > > > >>>> > > > > > > still > > > > >>>> > > > > > > > > > won't > > > > >>>> > > > > > > > > > > address the problem of reads being > consistent > > > with > > > > >>>> writes, > > > > >>>> > > > > > because > > > > >>>> > > > > > > each > > > > >>>> > > > > > > > > > > method call will still be independent > > regarding > > > > the > > > > >>>> data > > > > >>>> > > > stored > > > > >>>> > > > > > in > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > > > > database. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a > "change > > > > set" > > > > >>>> or > > > > >>>> > > > "atomic > > > > >>>> > > > > > > batch" > > > > >>>> > > > > > > > > > > concept to core Persistence and associate > each > > > > REST > > > > >>>> API > > > > >>>> > > > request > > > > >>>> > > > > > > with > > > > >>>> > > > > > > > > one > > > > >>>> > > > > > > > > > > such change set, which will be committed (or > > > > rolled > > > > >>>> back) > > > > >>>> > > at > > > > >>>> > > > > the > > > > >>>> > > > > > > end of > > > > >>>> > > > > > > > > > the > > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a similar > > > > >>>> concept in PR > > > > >>>> > > > 4939 > > > > >>>> > > > > > > [2]. In > > > > >>>> > > > > > > > > > > JDBC each change set will naturally be > > > associated > > > > >>>> with an > > > > >>>> > > > RDBMS > > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each > atomic > > > > >>>> change set > > > > >>>> > > > will > > > > >>>> > > > > be > > > > >>>> > > > > > > > > > > associated with one CAS operation on the > > > > underlying > > > > >>>> > > database. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [1] > > > > >>>> > > > > > > > > > > >>>> > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > > > >>>> > > > > > > > > > > >>>> > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > > > >>>> > > > > > > > > > > Dmitri. > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > > > >>>> > > > [email protected] > > > > >>>> > > > > > > > > > >>>> > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were > > > already > > > > >>>> > > recognized > > > > >>>> > > > > when > > > > >>>> > > > > > > we > > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At > > > that > > > > >>>> time, we > > > > >>>> > > > > chose > > > > >>>> > > > > > > to use > > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the > JDBC > > > due > > > > >>>> to the > > > > >>>> > > > > > > simplicity. I > > > > >>>> > > > > > > > > > > think > > > > >>>> > > > > > > > > > > > most of the issues mentioned here can > > already > > > be > > > > >>>> > > addressed > > > > >>>> > > > by > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped in > a > > > > >>>> transaction > > > > >>>> > > in > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > > > Similarly, > > > > >>>> catalog > > > > >>>> > > > > > > creation, > > > > >>>> > > > > > > > > > which > > > > >>>> > > > > > > > > > > > involves reading and creating multiple > > > objects, > > > > >>>> is also > > > > >>>> > > > > > executed > > > > >>>> > > > > > > > > > within a > > > > >>>> > > > > > > > > > > > transaction [2]. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager > > > > >>>> together with > > > > >>>> > > the > > > > >>>> > > > > > > > > persistence > > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide the > > > > required > > > > >>>> > > > > consistency > > > > >>>> > > > > > > > > > guarantees > > > > >>>> > > > > > > > > > > > for > > > > >>>> > > > > > > > > > > > specific operations, similar to what > > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl > > > > >>>> > > > > > > > > > > > does. > > > > >>>> > > > > > > > > > > > 2. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such > as > > > > >>>> JDBC) to use > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > directly. > > > > We > > > > >>>> may > > > > >>>> > > have > > > > >>>> > > > to > > > > >>>> > > > > > > deal > > > > >>>> > > > > > > > > with > > > > >>>> > > > > > > > > > > > transactional semantic mismatches > across > > > > >>>> different > > > > >>>> > > > > > persistence > > > > >>>> > > > > > > > > > > backends. > > > > >>>> > > > > > > > > > > > For example, we would likely avoid > using > > > > JDBC's > > > > >>>> > > > > > > > > > `runWithinTransaction` > > > > >>>> > > > > > > > > > > > for > > > > >>>> > > > > > > > > > > > single row updates, which adds > additional > > > > >>>> overhead and > > > > >>>> > > > > > > complexity > > > > >>>> > > > > > > > > > > > without > > > > >>>> > > > > > > > > > > > benefits. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > > > >>>> > > > > > > > > > > > 2. > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > > > >>>> Bourlatchkov < > > > > >>>> > > > > > > > > [email protected]> > > > > >>>> > > > > > > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi all, > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a > > > > couple > > > > >>>> of > > > > >>>> > > > > > interesting > > > > >>>> > > > > > > PRs: > > > > >>>> > > > > > > > > > > > > [4939], [5035]. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > It looks like people are starting to > > > encounter > > > > >>>> > > > consistency > > > > >>>> > > > > > > issues > > > > >>>> > > > > > > > > in > > > > >>>> > > > > > > > > > > > > JDBC persistence. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into > the > > > > >>>> underlying > > > > >>>> > > > > issues. > > > > >>>> > > > > > > They > > > > >>>> > > > > > > > > > offer > > > > >>>> > > > > > > > > > > > > incremental fixes that can work. > However, > > I > > > > >>>> believe it > > > > >>>> > > is > > > > >>>> > > > > > time > > > > >>>> > > > > > > for > > > > >>>> > > > > > > > > > the > > > > >>>> > > > > > > > > > > > > Polaris community to review and improve > > this > > > > >>>> area of > > > > >>>> > > the > > > > >>>> > > > > > > codebase > > > > >>>> > > > > > > > > > > > > holistically. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution that > > can > > > be > > > > >>>> applied > > > > >>>> > > to > > > > >>>> > > > > all > > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, > > > NoSQL) > > > > >>>> and > > > > >>>> > > > addresses > > > > >>>> > > > > > > these > > > > >>>> > > > > > > > > > > > > aspects: > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > > > changes > > > > >>>> where > > > > >>>> > > the > > > > >>>> > > > > > > service > > > > >>>> > > > > > > > > > reads > > > > >>>> > > > > > > > > > > > > and validates current catalog state, > > then > > > > >>>> commits a > > > > >>>> > > > > change > > > > >>>> > > > > > > (e.g. > > > > >>>> > > > > > > > > > > > > name clashes during renames). > > > > >>>> > > > > > > > > > > > > * Supporting consistent but independent > > > > changes > > > > >>>> to RBAC > > > > >>>> > > > > > grants > > > > >>>> > > > > > > and > > > > >>>> > > > > > > > > > > > > MetaStore entities. This independence > is > > > > >>>> needed to > > > > >>>> > > > > support > > > > >>>> > > > > > > > > > > > > external authorizers like OPA and > > Ranger. > > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across > > multiple > > > > >>>> similar > > > > >>>> > > > > entities. > > > > >>>> > > > > > > > > > > > > * Supporting authorization-based > filtering > > > of > > > > >>>> list > > > > >>>> > > > > operations > > > > >>>> > > > > > > (cf. > > > > >>>> > > > > > > > > > > > > [4831]). > > > > >>>> > > > > > > > > > > > > * Supporting credential-vending > decisions > > > that > > > > >>>> are > > > > >>>> > > rooted > > > > >>>> > > > > in > > > > >>>> > > > > > > the > > > > >>>> > > > > > > > > > > > > exact state of the catalog. > > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for > > > transient > > > > >>>> > > > persistence > > > > >>>> > > > > > > failures > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability > > failures). > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas. > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4831] > > > > >>>> https://github.com/apache/polaris/pull/4831 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > > > >>>> https://github.com/apache/polaris/pull/4939 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > > > >>>> https://github.com/apache/polaris/pull/5035 > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > > > >>>> > > > > > > > > > > > > Dmitri > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > >>> > > > > > > > > > >
