Hi Dennis, I agree that the borderline cases would make useful apples-to-apples validation scenarios.
I do not think we should make implementing a DynamoDB `BasePersistence` adapter under `AtomicOperationMetaStoreManager` a prerequisite for adding DynamoDB, though. The current atomic-manager API is not limited to single-entity updates. For example, it calls `writeEntities()` with a list for multi-entity create and update operations. JDBC implements that list as one database transaction (using SERIALIZABLE isolation). A DynamoDB implementation on this path would therefore need to define and validate much more than the single-entity CAS case: - which multi-entity operations use `TransactWriteItems`, and their atomicity domain and limits; - how conflicts, retries, and ambiguous post-send outcomes are represented; - how grants, cleanup intent, and related state remain consistent; and - which cases are intentionally unsupported rather than silently partial. That is worthwhile design work, but it is the broader persistence-contract discussion we already have on dev@. It would effectively require us to build and validate a second DynamoDB persistence model before delivering the stated optional-backend use case. The existing NoSQL persistence framework is a more natural fit for DynamoDB. Its conditional named-reference update is the publication point for a validated catalog state, while immutable objects can be written before that point. DynamoDB's conditional writes map directly to that model, including its consistency and conflict behavior. That path still needs the normal upstream work: Reviewable adapter and runtime-integration slice(s), the existing correctness/conformance coverage, and DynamoDB-specific tests for conditional-write and unknown-outcome behavior. I am very much in favor of benchmarks and comparative evidence. We can define those scenarios and run them against both implementations where useful. I would keep that as validation work, rather than making the optional DynamoDB backend wait for a competing implementation to be designed and built first. Cheers, Robert On Tue, Sep 29, 2026 at 12:53 AM Dennis Huo <[email protected]> wrote: > From what I remember of where the persistence discussion left off, we > wanted some of the borderline examples to actually be used for > apples-to-apples comparison between the NoSQL vs AtomicMetaStoreManager > approaches. > > Especially nowadays when prototyping can be done more quickly, I'd prefer > to take this opportunity to use DynamoDB to help quantify the tradeoffs > between the two, and following through with at least a first draft of > Yuewei's proposed design. > > The design's assessment of the single-entity CAS basic-case + > TrasactWriteItems special-case makes sense to me as aligning with the core > design of the AtomicMetaStoreManager. I guess we might have to look at > whether anything has drifted further from that model in newer features > though. > > On Mon, Sep 28, 2026 at 10:59 AM Yuewei Zhou <[email protected]> > wrote: > > > Thanks Robert - and it's great you already have a DynamoDB backend > > implementation in your fork. > > > > I'd like to stay involved in bringing DynamoDB support to Polaris. Is > there > > anything I can pick up to help move it forward? Happy to dig > > into whatever's useful. > > > > Thanks, > > Yuewei > > > > On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected]> wrote: > > > > > Hi all, > > > > > > Just to chime in here: I don't think we need to turn this into a choice > > > between two persistence models. > > > > > > I would prefer DynamoDB to be added as another backend in the existing > > > NoSQL persistence framework, rather than as a new BasePersistence > adapter > > > below AtomicOperationMetaStoreManager. > > > > > > The NoSQL reference update CAS is intentional. > > > That is the point where a validated catalog change becomes visible as > one > > > consistent state. > > > It gives us a way to handle changes involving related entities and > > > cross-entity validation consistently. > > > > > > There are broader metastore questions behind this, but they are already > > > being discussed in the "[DISCUSS] Consistent multi-object changes in > > > Polaris persistence" thread [1]. > > > > > > This includes how we obtain current state, what belongs to one logical > > > operation versus one backend commit attempt, and how conflicts, > retries, > > > and unknown outcomes are handled. > > > Those questions apply to JDBC and NoSQL alike, so I don't think we need > > to > > > resolve them again specifically for DynamoDB. > > > > > > FWIW, I already have a DynamoDB implementation in my public fork [2]. > > > > > > It may be useful as a starting point for the DynamoDB adapter and its > > > runtime integration. > > > We could then split the work into smaller reviewable pieces and use the > > > existing correctness tests. > > > > > > Cheers, > > > Robert > > > > > > [1] https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > > > [2] > > > > > > > > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > > > > > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou <[email protected] > > > > > wrote: > > > > > > > Hi all, > > > > > > > > Following the discussion on the GitHub issue, and as suggested there, > > I'd > > > > like to bring this proposal to the list for wider visibility and > > > feedback. > > > > > > > > Summary: a new, opt-in persistence.type=dynamodb backend that reuses > > the > > > > existing AtomicOperationMetaStoreManager (per-entity compare-and-set) > > > over > > > > a new DynamoDbBasePersistence adapter - the same shape > relational-jdbc > > > > uses, so the manager, catalog API, and entity model are unchanged. > The > > > > motivation is a serverless, AWS-native metastore, so AWS deployments > > > don't > > > > have to run a stateful Postgres or MongoDB purely to hold catalog > > > metadata. > > > > > > > > On the issue we aligned that this fits Polaris's usual pattern for > > > optional > > > > backends - a self-contained, pluggable module, selected by config, > with > > > no > > > > default change, no migration, and no new dependency or enforcement of > > any > > > > kind on users or downstream projects that don't enable it - which is > a > > > > reasonable basis to start building on and kick off a series of PRs. > I'd > > > of > > > > course still welcome input on the design as it takes shape. > > > > > > > > One trade-off I'd especially like thoughts on: this per-entity-CAS > > > approach > > > > versus layering DynamoDB under the existing NoSQL framework (single > > > > named-pointer / HEAD commit) - roughly, independent-write concurrency > > on > > > > one side vs. atomic multi-entity commits and cross-entity validation > on > > > the > > > > other. > > > > > > > > GitHub issue: https://github.com/apache/polaris/issues/5615 > > > > Design doc: > > > > > > > > > > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > > > > > Thanks, > > > > ZephyrYWZhou > > > > > > > > > >
