Thanks Robert - and it's great you already have a DynamoDB backend implementation in your fork.
I'd like to stay involved in bringing DynamoDB support to Polaris. Is there anything I can pick up to help move it forward? Happy to dig into whatever's useful. Thanks, Yuewei On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected]> wrote: > Hi all, > > Just to chime in here: I don't think we need to turn this into a choice > between two persistence models. > > I would prefer DynamoDB to be added as another backend in the existing > NoSQL persistence framework, rather than as a new BasePersistence adapter > below AtomicOperationMetaStoreManager. > > The NoSQL reference update CAS is intentional. > That is the point where a validated catalog change becomes visible as one > consistent state. > It gives us a way to handle changes involving related entities and > cross-entity validation consistently. > > There are broader metastore questions behind this, but they are already > being discussed in the "[DISCUSS] Consistent multi-object changes in > Polaris persistence" thread [1]. > > This includes how we obtain current state, what belongs to one logical > operation versus one backend commit attempt, and how conflicts, retries, > and unknown outcomes are handled. > Those questions apply to JDBC and NoSQL alike, so I don't think we need to > resolve them again specifically for DynamoDB. > > FWIW, I already have a DynamoDB implementation in my public fork [2]. > > It may be useful as a starting point for the DynamoDB adapter and its > runtime integration. > We could then split the work into smaller reviewable pieces and use the > existing correctness tests. > > Cheers, > Robert > > [1] https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > [2] > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou <[email protected]> > wrote: > > > Hi all, > > > > Following the discussion on the GitHub issue, and as suggested there, I'd > > like to bring this proposal to the list for wider visibility and > feedback. > > > > Summary: a new, opt-in persistence.type=dynamodb backend that reuses the > > existing AtomicOperationMetaStoreManager (per-entity compare-and-set) > over > > a new DynamoDbBasePersistence adapter - the same shape relational-jdbc > > uses, so the manager, catalog API, and entity model are unchanged. The > > motivation is a serverless, AWS-native metastore, so AWS deployments > don't > > have to run a stateful Postgres or MongoDB purely to hold catalog > metadata. > > > > On the issue we aligned that this fits Polaris's usual pattern for > optional > > backends - a self-contained, pluggable module, selected by config, with > no > > default change, no migration, and no new dependency or enforcement of any > > kind on users or downstream projects that don't enable it - which is a > > reasonable basis to start building on and kick off a series of PRs. I'd > of > > course still welcome input on the design as it takes shape. > > > > One trade-off I'd especially like thoughts on: this per-entity-CAS > approach > > versus layering DynamoDB under the existing NoSQL framework (single > > named-pointer / HEAD commit) - roughly, independent-write concurrency on > > one side vs. atomic multi-entity commits and cross-entity validation on > the > > other. > > > > GitHub issue: https://github.com/apache/polaris/issues/5615 > > Design doc: > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > Thanks, > > ZephyrYWZhou > > >
