Hi all, Thanks, this seems like a good way forward.
I agree that comparative scenarios and an AtomicOperationMetaStoreManager prototype could provide useful evidence for longer-term support decisions. I also agree that this work relates to the broader multi-object persistence discussion. I would keep that comparison separate from moving the NoSQL DynamoDB backend forward, though. Just to avoid a terminology mix-up: the named reference is the conditional publication point for a validated catalog state. Objects can be written immutably before the reference update, and the successful CAS makes the related state visible together. That is a natural fit for DynamoDB's conditional-write model. It gives us one explicit consistency and conflict boundary without making the DynamoDB backend reproduce the Atomic manager's current semantics and open TODOs. I have opened: - A draft PR[5667] with the complete NoSQL DynamoDB implementation (3 commits) - PR[5668] containing the first reviewable NoSQL DynamoDB slice The draft is only a reference for the complete integration, not a request to review the entire change at once. I suggest that we use the existing NoSQL correctness suite, plus DynamoDB-specific coverage for conditional-write failures and unknown outcomes, as the baseline validation. I am happy for an Atomic DynamoDB implementation to proceed experimentally in parallel if there is interest. The comparison should cover the concrete scenarios discussed here, including concurrent container/child changes, rename and parent-path validation, grants, multi-table commits, failure/unknown-outcome behavior, and the correctness-sensitive GSI lookups such as `hasChildren` and overlap checks. I would not make either implementation a prerequisite for the other, or commit users to two permanent DynamoDB configuration paths before that evidence exists. We can use the comparison to inform a later support decision while getting the existing NoSQL implementation reviewed in small, useful pieces now. Cheers, Robert [5667] https://github.com/apache/polaris/pull/5667 [5668] https://github.com/apache/polaris/pull/5668 On Thu, Oct 1, 2026 at 6:33 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi guys > > Thanks Dennis, I agree with the "iterative" approach (it was my call > as well). I also have no problem to work on both implementations, it's > certainly valuable even if we end up keeping only one (assuming we > have the bandwidth to work on both :)). > > I have a few comments: > 1. My bad about TransactWriteItems. Yufei and Yuewei are right: > writeEntities is only used in two places in > AtomicOperationMetaStoreManager and it is bounded, so it is not where > the complexity is. Yuewei's design already uses TransactWriteItems for > every create and rename, so it's the regular path, not a special case. > 2. Actually, my concern about the Atomic path is not specific to > DynamoDB. It's the cross-entity cases that > AtomicOperationMetaStoreManager documents as TODO today: dropping a > container while a child is created concurrently, dropping an entity > without automatically create the cleanup task, rename/drop without > validating the parent path, and grant records versus > grantRecordsVersion. JDBC has the same gaps, so DynamoDB would not be > worse than what we do today. Definitely an improvement to consider > (separate from this thread). > 3. I have a question for Yuewei: you state that the GSI is used for > discovery only and never for correctness, but I don't see hasChildren > and hasOverlappingSiblings covered. Can you clarify? > 4. I like Dennis' idea to use PolarisResolutionManifest as the > dependency set validated with the commit. It maps well to > ConditionCheck in TransactWriteItems. > > So, in order to move forward, I propose: > - to work on both implementations (if we have volunteers :)): Yuewei > on the Atomic adapter as he described, and the NoSQL backend from > Robert. Each can be reviewed with PRs. > - we flag both as experimental/beta until we decide, and we pick > configuration names to avoid confusing users (jdbc.dynamodb and > nosql.dynamodb for instance); > - we agree all together about the comparison scenarios between the > implementations (for instance drop namespace vs concurrent create > table, concurrent commits to different tables in the same catalog, > multi-table commit with N tables, and write cost per operation) > - we revisit on the dev mailing list once we have the comparison > > If we don't have volunteer on NoSQL, let's more forward with the > current Yuewei's approach. > > Regards > JB > > On Thu, Oct 1, 2026 at 2:21 AM Dennis Huo <[email protected]> wrote: > > > > I generally agree with not wanting to strictly *block* any forward > progress > > on waiting for the alternative impl, especially since a good comparative > > assessment here will need a reasonable first-pass impl for both anyways, > so > > I have nothing against dusting off the existing prototype as Robert and > JB > > suggest. > > > > As with any other launch-and-iterate cases, this just means putting more > > discernment later in the lifecycle when it comes time to decide on > > long-term supportability details, beta/experimental tags, and > > evolving/removing vestigial code later on if it becomes obsolete. > > > > Overall, I think the work we do here for both impls will prove valuable > > regardless of how it ends up long-term (even if we end up deleting one or > > the other, I'd still count the journey as a win if we learn insights from > > it). > > > > I also forgot to call out earlier my appreciation for Yuewei's well > > organized and well-grounded design document; many thanks for putting it > > together! Given the concrete mappings, explicit design choices, etc., we > > can more easily extract insights from where any of our base assumptions > > didn't hold if it hits any limitations, or alternatively it can prove > that > > the model translates to the real system cleanly. > > > > I also think this prototyping goes hand-in-hand with the "Consistent > > multi-object changes" discussion rather than being extraneous. IIRC we > had > > some discussion in a community meeting about whether the > > PolarisResolutionManifest in itself could already be precisely the > > representation of the "set of dependencies" from the "input state" that > > need to be validated atomically with the final commit, but I forgot to > > follow up on that one in the dev mailing list. > > > > I'll post some thoughts there soon, but in a nutshell, I think we > probably > > *don't* want to express a full-fledged "transaction" semantic where a > > session sees its own uncommitted copy of an entity through a series of > > read + write + read + write operations. Instead, the > > PolarisResolutionManifest already carries the entity dependencies of the > > current operation, including their entityIds and entityVersions. Now, we > > probably don't want to turn *all* single-entity updates into a > "conditional > > batch update"-equivalent, but we could express the cases where a set of > > commits are dependent on the state of another set of dependency states > well > > in that form, and it seems to align well with DynamoDB's > TransactWriteItems. > > > > > > > > On Wed, Sep 30, 2026 at 4:33 PM Yuewei Zhou <[email protected]> > > wrote: > > > > > Hi all, thanks for all the insights provided so far. > > > > > > Short answer to Yufei's question first: no - global versioning isn't > > > necessary for the DynamoDB use case I'm targeting, and honestly I agree > > > with Yufei that the Atomic path doesn't add significant complexity > here. > > > > > > Polaris's correctness is already per-entity such as entityVersion CAS > on > > > update, conditional-on-non-existence on create, and a conditional > write for > > > name uniqueness. The one genuinely multi-entity atomic case, the > Iceberg > > > multi-table commit (writeEntities), maps directly onto DynamoDB's > > > TransactWriteItems. None of this diverges meaningfully from DynamoDB's > > > native characteristics or normal usage. Conditional writes and > > > TransactWriteItems are exactly the primitives the service is built > around. > > > > > > On the other hand, a global named-pointer/HEAD version would buy > > > whole-catalog snapshot consistency we don't need, at the cost of write > > > contention on the single reference plus cleanup of immutable objects > which > > > is exactly what we want to avoid for a serverless, > high-write-concurrency > > > AWS-native metastore. > > > > > > Thanks, > > > Yuewei > > > > > > > > > On Wed, Sep 30, 2026 at 1:46 PM Yufei Gu <[email protected]> wrote: > > > > > > > I don't think using AtomicOperationMetaStoreManager for DynamoDB will > > > > introduce significant complexity regarding DynamoDB transactions. > > > > Additionally, I'm not sure if DynamoDB users need the global > versioning > > > > feature, given the performance and cleanup overhead it requires. > > > > > > > > Maybe Yuewei can clarify whether global versioning is necessary for > his > > > use > > > > case. > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Sep 29, 2026 at 9:59 PM Jean-Baptiste Onofré < > [email protected]> > > > > wrote: > > > > > > > > > Hi all > > > > > > > > > > I agree with Robert's assessment here: persistence/nosql is the > most > > > > > natural and architecturally sound home for DynamoDB. > > > > > > > > > > A few thoughts from my side: > > > > > 1. BasePersistence and AtomicOperationMetaStoreManager implicitly > > > > > assume relational multi-entity transaction semantics (as > implemented > > > > > by JDBC via serializable transactions). Forcing DynamoDB into that > > > > > shape means orchestrating TransactWriteItems to handle multi-entity > > > > > updates (writeEntities), which introduces significant complexity > > > > > around DynamoDB's transaction item/size limits, retry strategies, > and > > > > > ambiguous outcome handling. > > > > > 2. In contrast, the persistence/nosql framework was built > specifically > > > > > around the primitive that KV/NoSQL stores natively provide: > single-row > > > > > conditional updates (CAS) on named pointers, with immutable objects > > > > > written beforehand. This eliminates distributed rollback issues, > > > > > naturally fits DynamoDB's consistency model, and stays well within > > > > > DynamoDB's 400 KB item limit; > > > > > 3. While Dennis makes a fair point about wanting comparative > evidence > > > > > and understanding the concurrency trade-offs between per-entity CAS > > > > > and named-pointer commits, I don't think we should block > delivering an > > > > > AWS-native DynamoDB backend on that research. We can always run > > > > > benchmarks and comparative experiments in parallel or as part of > the > > > > > broader persistence discussion (we used the same approach when we > > > > > moved from OpenJPA to JDBC). > > > > > > > > > > Given that Robert already has a working implementation on his > fork, I > > > > > propose we use that as the baseline and work together (Yuewei, > Robert, > > > > > and anyone interested) to slice it into clean, reviewable, minimal > PRs > > > > > validated against the existing > > > > > persistence/nosql/persistence/correctness test suite. > > > > > > > > > > Thoughts? > > > > > > > > > > Regards > > > > > JB > > > > > > > > > > On Tue, Sep 29, 2026 at 4:31 PM Robert Stupp <[email protected]> > wrote: > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > I agree that the borderline cases would make useful > apples-to-apples > > > > > > validation scenarios. > > > > > > > > > > > > I do not think we should make implementing a DynamoDB > > > `BasePersistence` > > > > > > adapter under `AtomicOperationMetaStoreManager` a prerequisite > for > > > > adding > > > > > > DynamoDB, though. > > > > > > > > > > > > The current atomic-manager API is not limited to single-entity > > > updates. > > > > > > For example, it calls `writeEntities()` with a list for > multi-entity > > > > > create > > > > > > and update operations. > > > > > > JDBC implements that list as one database transaction (using > > > > SERIALIZABLE > > > > > > isolation). > > > > > > A DynamoDB implementation on this path would therefore need to > define > > > > and > > > > > > validate much more than the single-entity CAS case: > > > > > > > > > > > > - which multi-entity operations use `TransactWriteItems`, and > their > > > > > > atomicity domain and limits; > > > > > > - how conflicts, retries, and ambiguous post-send outcomes are > > > > > represented; > > > > > > - how grants, cleanup intent, and related state remain > consistent; > > > and > > > > > > - which cases are intentionally unsupported rather than silently > > > > partial. > > > > > > > > > > > > That is worthwhile design work, but it is the broader > > > > > persistence-contract > > > > > > discussion we already have on dev@. > > > > > > It would effectively require us to build and validate a second > > > DynamoDB > > > > > > persistence model before delivering the stated optional-backend > use > > > > case. > > > > > > > > > > > > The existing NoSQL persistence framework is a more natural fit > for > > > > > DynamoDB. > > > > > > Its conditional named-reference update is the publication point > for a > > > > > > validated catalog state, while immutable objects can be written > > > before > > > > > that > > > > > > point. > > > > > > DynamoDB's conditional writes map directly to that model, > including > > > its > > > > > > consistency and conflict behavior. > > > > > > > > > > > > That path still needs the normal upstream work: > > > > > > Reviewable adapter and runtime-integration slice(s), the existing > > > > > > correctness/conformance coverage, and DynamoDB-specific tests for > > > > > > conditional-write and unknown-outcome behavior. > > > > > > > > > > > > I am very much in favor of benchmarks and comparative evidence. > > > > > > We can define those scenarios and run them against both > > > implementations > > > > > > where useful. > > > > > > I would keep that as validation work, rather than making the > optional > > > > > > DynamoDB backend wait for a competing implementation to be > designed > > > and > > > > > > built first. > > > > > > > > > > > > Cheers, > > > > > > Robert > > > > > > > > > > > > On Tue, Sep 29, 2026 at 12:53 AM Dennis Huo <[email protected]> > wrote: > > > > > > > > > > > > > From what I remember of where the persistence discussion left > off, > > > we > > > > > > > wanted some of the borderline examples to actually be used for > > > > > > > apples-to-apples comparison between the NoSQL vs > > > > AtomicMetaStoreManager > > > > > > > approaches. > > > > > > > > > > > > > > Especially nowadays when prototyping can be done more quickly, > I'd > > > > > prefer > > > > > > > to take this opportunity to use DynamoDB to help quantify the > > > > tradeoffs > > > > > > > between the two, and following through with at least a first > draft > > > of > > > > > > > Yuewei's proposed design. > > > > > > > > > > > > > > The design's assessment of the single-entity CAS basic-case + > > > > > > > TrasactWriteItems special-case makes sense to me as aligning > with > > > the > > > > > core > > > > > > > design of the AtomicMetaStoreManager. I guess we might have to > look > > > > at > > > > > > > whether anything has drifted further from that model in newer > > > > features > > > > > > > though. > > > > > > > > > > > > > > On Mon, Sep 28, 2026 at 10:59 AM Yuewei Zhou < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks Robert - and it's great you already have a DynamoDB > > > backend > > > > > > > > implementation in your fork. > > > > > > > > > > > > > > > > I'd like to stay involved in bringing DynamoDB support to > > > Polaris. > > > > Is > > > > > > > there > > > > > > > > anything I can pick up to help move it forward? Happy to dig > > > > > > > > into whatever's useful. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yuewei > > > > > > > > > > > > > > > > On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > Just to chime in here: I don't think we need to turn this > into > > > a > > > > > choice > > > > > > > > > between two persistence models. > > > > > > > > > > > > > > > > > > I would prefer DynamoDB to be added as another backend in > the > > > > > existing > > > > > > > > > NoSQL persistence framework, rather than as a new > > > BasePersistence > > > > > > > adapter > > > > > > > > > below AtomicOperationMetaStoreManager. > > > > > > > > > > > > > > > > > > The NoSQL reference update CAS is intentional. > > > > > > > > > That is the point where a validated catalog change becomes > > > > visible > > > > > as > > > > > > > one > > > > > > > > > consistent state. > > > > > > > > > It gives us a way to handle changes involving related > entities > > > > and > > > > > > > > > cross-entity validation consistently. > > > > > > > > > > > > > > > > > > There are broader metastore questions behind this, but > they are > > > > > already > > > > > > > > > being discussed in the "[DISCUSS] Consistent multi-object > > > changes > > > > > in > > > > > > > > > Polaris persistence" thread [1]. > > > > > > > > > > > > > > > > > > This includes how we obtain current state, what belongs to > one > > > > > logical > > > > > > > > > operation versus one backend commit attempt, and how > conflicts, > > > > > > > retries, > > > > > > > > > and unknown outcomes are handled. > > > > > > > > > Those questions apply to JDBC and NoSQL alike, so I don't > think > > > > we > > > > > need > > > > > > > > to > > > > > > > > > resolve them again specifically for DynamoDB. > > > > > > > > > > > > > > > > > > FWIW, I already have a DynamoDB implementation in my public > > > fork > > > > > [2]. > > > > > > > > > > > > > > > > > > It may be useful as a starting point for the DynamoDB > adapter > > > and > > > > > its > > > > > > > > > runtime integration. > > > > > > > > > We could then split the work into smaller reviewable > pieces and > > > > > use the > > > > > > > > > existing correctness tests. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > [1] > > > > > https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou < > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > Following the discussion on the GitHub issue, and as > > > suggested > > > > > there, > > > > > > > > I'd > > > > > > > > > > like to bring this proposal to the list for wider > visibility > > > > and > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > Summary: a new, opt-in persistence.type=dynamodb backend > that > > > > > reuses > > > > > > > > the > > > > > > > > > > existing AtomicOperationMetaStoreManager (per-entity > > > > > compare-and-set) > > > > > > > > > over > > > > > > > > > > a new DynamoDbBasePersistence adapter - the same shape > > > > > > > relational-jdbc > > > > > > > > > > uses, so the manager, catalog API, and entity model are > > > > > unchanged. > > > > > > > The > > > > > > > > > > motivation is a serverless, AWS-native metastore, so AWS > > > > > deployments > > > > > > > > > don't > > > > > > > > > > have to run a stateful Postgres or MongoDB purely to hold > > > > catalog > > > > > > > > > metadata. > > > > > > > > > > > > > > > > > > > > On the issue we aligned that this fits Polaris's usual > > > pattern > > > > > for > > > > > > > > > optional > > > > > > > > > > backends - a self-contained, pluggable module, selected > by > > > > > config, > > > > > > > with > > > > > > > > > no > > > > > > > > > > default change, no migration, and no new dependency or > > > > > enforcement of > > > > > > > > any > > > > > > > > > > kind on users or downstream projects that don't enable > it - > > > > > which is > > > > > > > a > > > > > > > > > > reasonable basis to start building on and kick off a > series > > > of > > > > > PRs. > > > > > > > I'd > > > > > > > > > of > > > > > > > > > > course still welcome input on the design as it takes > shape. > > > > > > > > > > > > > > > > > > > > One trade-off I'd especially like thoughts on: this > > > > > per-entity-CAS > > > > > > > > > approach > > > > > > > > > > versus layering DynamoDB under the existing NoSQL > framework > > > > > (single > > > > > > > > > > named-pointer / HEAD commit) - roughly, independent-write > > > > > concurrency > > > > > > > > on > > > > > > > > > > one side vs. atomic multi-entity commits and cross-entity > > > > > validation > > > > > > > on > > > > > > > > > the > > > > > > > > > > other. > > > > > > > > > > > > > > > > > > > > GitHub issue: > https://github.com/apache/polaris/issues/5615 > > > > > > > > > > Design doc: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > ZephyrYWZhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
