I don't think using AtomicOperationMetaStoreManager for DynamoDB will introduce significant complexity regarding DynamoDB transactions. Additionally, I'm not sure if DynamoDB users need the global versioning feature, given the performance and cleanup overhead it requires.
Maybe Yuewei can clarify whether global versioning is necessary for his use case. Yufei On Tue, Sep 29, 2026 at 9:59 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi all > > I agree with Robert's assessment here: persistence/nosql is the most > natural and architecturally sound home for DynamoDB. > > A few thoughts from my side: > 1. BasePersistence and AtomicOperationMetaStoreManager implicitly > assume relational multi-entity transaction semantics (as implemented > by JDBC via serializable transactions). Forcing DynamoDB into that > shape means orchestrating TransactWriteItems to handle multi-entity > updates (writeEntities), which introduces significant complexity > around DynamoDB's transaction item/size limits, retry strategies, and > ambiguous outcome handling. > 2. In contrast, the persistence/nosql framework was built specifically > around the primitive that KV/NoSQL stores natively provide: single-row > conditional updates (CAS) on named pointers, with immutable objects > written beforehand. This eliminates distributed rollback issues, > naturally fits DynamoDB's consistency model, and stays well within > DynamoDB's 400 KB item limit; > 3. While Dennis makes a fair point about wanting comparative evidence > and understanding the concurrency trade-offs between per-entity CAS > and named-pointer commits, I don't think we should block delivering an > AWS-native DynamoDB backend on that research. We can always run > benchmarks and comparative experiments in parallel or as part of the > broader persistence discussion (we used the same approach when we > moved from OpenJPA to JDBC). > > Given that Robert already has a working implementation on his fork, I > propose we use that as the baseline and work together (Yuewei, Robert, > and anyone interested) to slice it into clean, reviewable, minimal PRs > validated against the existing > persistence/nosql/persistence/correctness test suite. > > Thoughts? > > Regards > JB > > On Tue, Sep 29, 2026 at 4:31 PM Robert Stupp <[email protected]> wrote: > > > > Hi Dennis, > > > > I agree that the borderline cases would make useful apples-to-apples > > validation scenarios. > > > > I do not think we should make implementing a DynamoDB `BasePersistence` > > adapter under `AtomicOperationMetaStoreManager` a prerequisite for adding > > DynamoDB, though. > > > > The current atomic-manager API is not limited to single-entity updates. > > For example, it calls `writeEntities()` with a list for multi-entity > create > > and update operations. > > JDBC implements that list as one database transaction (using SERIALIZABLE > > isolation). > > A DynamoDB implementation on this path would therefore need to define and > > validate much more than the single-entity CAS case: > > > > - which multi-entity operations use `TransactWriteItems`, and their > > atomicity domain and limits; > > - how conflicts, retries, and ambiguous post-send outcomes are > represented; > > - how grants, cleanup intent, and related state remain consistent; and > > - which cases are intentionally unsupported rather than silently partial. > > > > That is worthwhile design work, but it is the broader > persistence-contract > > discussion we already have on dev@. > > It would effectively require us to build and validate a second DynamoDB > > persistence model before delivering the stated optional-backend use case. > > > > The existing NoSQL persistence framework is a more natural fit for > DynamoDB. > > Its conditional named-reference update is the publication point for a > > validated catalog state, while immutable objects can be written before > that > > point. > > DynamoDB's conditional writes map directly to that model, including its > > consistency and conflict behavior. > > > > That path still needs the normal upstream work: > > Reviewable adapter and runtime-integration slice(s), the existing > > correctness/conformance coverage, and DynamoDB-specific tests for > > conditional-write and unknown-outcome behavior. > > > > I am very much in favor of benchmarks and comparative evidence. > > We can define those scenarios and run them against both implementations > > where useful. > > I would keep that as validation work, rather than making the optional > > DynamoDB backend wait for a competing implementation to be designed and > > built first. > > > > Cheers, > > Robert > > > > On Tue, Sep 29, 2026 at 12:53 AM Dennis Huo <[email protected]> wrote: > > > > > From what I remember of where the persistence discussion left off, we > > > wanted some of the borderline examples to actually be used for > > > apples-to-apples comparison between the NoSQL vs AtomicMetaStoreManager > > > approaches. > > > > > > Especially nowadays when prototyping can be done more quickly, I'd > prefer > > > to take this opportunity to use DynamoDB to help quantify the tradeoffs > > > between the two, and following through with at least a first draft of > > > Yuewei's proposed design. > > > > > > The design's assessment of the single-entity CAS basic-case + > > > TrasactWriteItems special-case makes sense to me as aligning with the > core > > > design of the AtomicMetaStoreManager. I guess we might have to look at > > > whether anything has drifted further from that model in newer features > > > though. > > > > > > On Mon, Sep 28, 2026 at 10:59 AM Yuewei Zhou <[email protected] > > > > > wrote: > > > > > > > Thanks Robert - and it's great you already have a DynamoDB backend > > > > implementation in your fork. > > > > > > > > I'd like to stay involved in bringing DynamoDB support to Polaris. Is > > > there > > > > anything I can pick up to help move it forward? Happy to dig > > > > into whatever's useful. > > > > > > > > Thanks, > > > > Yuewei > > > > > > > > On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > Just to chime in here: I don't think we need to turn this into a > choice > > > > > between two persistence models. > > > > > > > > > > I would prefer DynamoDB to be added as another backend in the > existing > > > > > NoSQL persistence framework, rather than as a new BasePersistence > > > adapter > > > > > below AtomicOperationMetaStoreManager. > > > > > > > > > > The NoSQL reference update CAS is intentional. > > > > > That is the point where a validated catalog change becomes visible > as > > > one > > > > > consistent state. > > > > > It gives us a way to handle changes involving related entities and > > > > > cross-entity validation consistently. > > > > > > > > > > There are broader metastore questions behind this, but they are > already > > > > > being discussed in the "[DISCUSS] Consistent multi-object changes > in > > > > > Polaris persistence" thread [1]. > > > > > > > > > > This includes how we obtain current state, what belongs to one > logical > > > > > operation versus one backend commit attempt, and how conflicts, > > > retries, > > > > > and unknown outcomes are handled. > > > > > Those questions apply to JDBC and NoSQL alike, so I don't think we > need > > > > to > > > > > resolve them again specifically for DynamoDB. > > > > > > > > > > FWIW, I already have a DynamoDB implementation in my public fork > [2]. > > > > > > > > > > It may be useful as a starting point for the DynamoDB adapter and > its > > > > > runtime integration. > > > > > We could then split the work into smaller reviewable pieces and > use the > > > > > existing correctness tests. > > > > > > > > > > Cheers, > > > > > Robert > > > > > > > > > > [1] > https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > > > > > [2] > > > > > > > > > > > > > > > > > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Following the discussion on the GitHub issue, and as suggested > there, > > > > I'd > > > > > > like to bring this proposal to the list for wider visibility and > > > > > feedback. > > > > > > > > > > > > Summary: a new, opt-in persistence.type=dynamodb backend that > reuses > > > > the > > > > > > existing AtomicOperationMetaStoreManager (per-entity > compare-and-set) > > > > > over > > > > > > a new DynamoDbBasePersistence adapter - the same shape > > > relational-jdbc > > > > > > uses, so the manager, catalog API, and entity model are > unchanged. > > > The > > > > > > motivation is a serverless, AWS-native metastore, so AWS > deployments > > > > > don't > > > > > > have to run a stateful Postgres or MongoDB purely to hold catalog > > > > > metadata. > > > > > > > > > > > > On the issue we aligned that this fits Polaris's usual pattern > for > > > > > optional > > > > > > backends - a self-contained, pluggable module, selected by > config, > > > with > > > > > no > > > > > > default change, no migration, and no new dependency or > enforcement of > > > > any > > > > > > kind on users or downstream projects that don't enable it - > which is > > > a > > > > > > reasonable basis to start building on and kick off a series of > PRs. > > > I'd > > > > > of > > > > > > course still welcome input on the design as it takes shape. > > > > > > > > > > > > One trade-off I'd especially like thoughts on: this > per-entity-CAS > > > > > approach > > > > > > versus layering DynamoDB under the existing NoSQL framework > (single > > > > > > named-pointer / HEAD commit) - roughly, independent-write > concurrency > > > > on > > > > > > one side vs. atomic multi-entity commits and cross-entity > validation > > > on > > > > > the > > > > > > other. > > > > > > > > > > > > GitHub issue: https://github.com/apache/polaris/issues/5615 > > > > > > Design doc: > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > > > > > > > > > Thanks, > > > > > > ZephyrYWZhou > > > > > > > > > > > > > > > > > > >
