Hi all, Just to chime in here: I don't think we need to turn this into a choice between two persistence models.
I would prefer DynamoDB to be added as another backend in the existing NoSQL persistence framework, rather than as a new BasePersistence adapter below AtomicOperationMetaStoreManager. The NoSQL reference update CAS is intentional. That is the point where a validated catalog change becomes visible as one consistent state. It gives us a way to handle changes involving related entities and cross-entity validation consistently. There are broader metastore questions behind this, but they are already being discussed in the "[DISCUSS] Consistent multi-object changes in Polaris persistence" thread [1]. This includes how we obtain current state, what belongs to one logical operation versus one backend commit attempt, and how conflicts, retries, and unknown outcomes are handled. Those questions apply to JDBC and NoSQL alike, so I don't think we need to resolve them again specifically for DynamoDB. FWIW, I already have a DynamoDB implementation in my public fork [2]. It may be useful as a starting point for the DynamoDB adapter and its runtime integration. We could then split the work into smaller reviewable pieces and use the existing correctness tests. Cheers, Robert [1] https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy [2] https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou <[email protected]> wrote: > Hi all, > > Following the discussion on the GitHub issue, and as suggested there, I'd > like to bring this proposal to the list for wider visibility and feedback. > > Summary: a new, opt-in persistence.type=dynamodb backend that reuses the > existing AtomicOperationMetaStoreManager (per-entity compare-and-set) over > a new DynamoDbBasePersistence adapter - the same shape relational-jdbc > uses, so the manager, catalog API, and entity model are unchanged. The > motivation is a serverless, AWS-native metastore, so AWS deployments don't > have to run a stateful Postgres or MongoDB purely to hold catalog metadata. > > On the issue we aligned that this fits Polaris's usual pattern for optional > backends - a self-contained, pluggable module, selected by config, with no > default change, no migration, and no new dependency or enforcement of any > kind on users or downstream projects that don't enable it - which is a > reasonable basis to start building on and kick off a series of PRs. I'd of > course still welcome input on the design as it takes shape. > > One trade-off I'd especially like thoughts on: this per-entity-CAS approach > versus layering DynamoDB under the existing NoSQL framework (single > named-pointer / HEAD commit) - roughly, independent-write concurrency on > one side vs. atomic multi-entity commits and cross-entity validation on the > other. > > GitHub issue: https://github.com/apache/polaris/issues/5615 > Design doc: > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > Thanks, > ZephyrYWZhou >
