>From what I remember of where the persistence discussion left off, we wanted some of the borderline examples to actually be used for apples-to-apples comparison between the NoSQL vs AtomicMetaStoreManager approaches.
Especially nowadays when prototyping can be done more quickly, I'd prefer to take this opportunity to use DynamoDB to help quantify the tradeoffs between the two, and following through with at least a first draft of Yuewei's proposed design. The design's assessment of the single-entity CAS basic-case + TrasactWriteItems special-case makes sense to me as aligning with the core design of the AtomicMetaStoreManager. I guess we might have to look at whether anything has drifted further from that model in newer features though. On Mon, Sep 28, 2026 at 10:59 AM Yuewei Zhou <[email protected]> wrote: > Thanks Robert - and it's great you already have a DynamoDB backend > implementation in your fork. > > I'd like to stay involved in bringing DynamoDB support to Polaris. Is there > anything I can pick up to help move it forward? Happy to dig > into whatever's useful. > > Thanks, > Yuewei > > On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected]> wrote: > > > Hi all, > > > > Just to chime in here: I don't think we need to turn this into a choice > > between two persistence models. > > > > I would prefer DynamoDB to be added as another backend in the existing > > NoSQL persistence framework, rather than as a new BasePersistence adapter > > below AtomicOperationMetaStoreManager. > > > > The NoSQL reference update CAS is intentional. > > That is the point where a validated catalog change becomes visible as one > > consistent state. > > It gives us a way to handle changes involving related entities and > > cross-entity validation consistently. > > > > There are broader metastore questions behind this, but they are already > > being discussed in the "[DISCUSS] Consistent multi-object changes in > > Polaris persistence" thread [1]. > > > > This includes how we obtain current state, what belongs to one logical > > operation versus one backend commit attempt, and how conflicts, retries, > > and unknown outcomes are handled. > > Those questions apply to JDBC and NoSQL alike, so I don't think we need > to > > resolve them again specifically for DynamoDB. > > > > FWIW, I already have a DynamoDB implementation in my public fork [2]. > > > > It may be useful as a starting point for the DynamoDB adapter and its > > runtime integration. > > We could then split the work into smaller reviewable pieces and use the > > existing correctness tests. > > > > Cheers, > > Robert > > > > [1] https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > > [2] > > > > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Following the discussion on the GitHub issue, and as suggested there, > I'd > > > like to bring this proposal to the list for wider visibility and > > feedback. > > > > > > Summary: a new, opt-in persistence.type=dynamodb backend that reuses > the > > > existing AtomicOperationMetaStoreManager (per-entity compare-and-set) > > over > > > a new DynamoDbBasePersistence adapter - the same shape relational-jdbc > > > uses, so the manager, catalog API, and entity model are unchanged. The > > > motivation is a serverless, AWS-native metastore, so AWS deployments > > don't > > > have to run a stateful Postgres or MongoDB purely to hold catalog > > metadata. > > > > > > On the issue we aligned that this fits Polaris's usual pattern for > > optional > > > backends - a self-contained, pluggable module, selected by config, with > > no > > > default change, no migration, and no new dependency or enforcement of > any > > > kind on users or downstream projects that don't enable it - which is a > > > reasonable basis to start building on and kick off a series of PRs. I'd > > of > > > course still welcome input on the design as it takes shape. > > > > > > One trade-off I'd especially like thoughts on: this per-entity-CAS > > approach > > > versus layering DynamoDB under the existing NoSQL framework (single > > > named-pointer / HEAD commit) - roughly, independent-write concurrency > on > > > one side vs. atomic multi-entity commits and cross-entity validation on > > the > > > other. > > > > > > GitHub issue: https://github.com/apache/polaris/issues/5615 > > > Design doc: > > > > > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > > > Thanks, > > > ZephyrYWZhou > > > > > >
