Hi guys Thanks Dennis, I agree with the "iterative" approach (it was my call as well). I also have no problem to work on both implementations, it's certainly valuable even if we end up keeping only one (assuming we have the bandwidth to work on both :)).
I have a few comments: 1. My bad about TransactWriteItems. Yufei and Yuewei are right: writeEntities is only used in two places in AtomicOperationMetaStoreManager and it is bounded, so it is not where the complexity is. Yuewei's design already uses TransactWriteItems for every create and rename, so it's the regular path, not a special case. 2. Actually, my concern about the Atomic path is not specific to DynamoDB. It's the cross-entity cases that AtomicOperationMetaStoreManager documents as TODO today: dropping a container while a child is created concurrently, dropping an entity without automatically create the cleanup task, rename/drop without validating the parent path, and grant records versus grantRecordsVersion. JDBC has the same gaps, so DynamoDB would not be worse than what we do today. Definitely an improvement to consider (separate from this thread). 3. I have a question for Yuewei: you state that the GSI is used for discovery only and never for correctness, but I don't see hasChildren and hasOverlappingSiblings covered. Can you clarify? 4. I like Dennis' idea to use PolarisResolutionManifest as the dependency set validated with the commit. It maps well to ConditionCheck in TransactWriteItems. So, in order to move forward, I propose: - to work on both implementations (if we have volunteers :)): Yuewei on the Atomic adapter as he described, and the NoSQL backend from Robert. Each can be reviewed with PRs. - we flag both as experimental/beta until we decide, and we pick configuration names to avoid confusing users (jdbc.dynamodb and nosql.dynamodb for instance); - we agree all together about the comparison scenarios between the implementations (for instance drop namespace vs concurrent create table, concurrent commits to different tables in the same catalog, multi-table commit with N tables, and write cost per operation) - we revisit on the dev mailing list once we have the comparison If we don't have volunteer on NoSQL, let's more forward with the current Yuewei's approach. Regards JB On Thu, Oct 1, 2026 at 2:21 AM Dennis Huo <[email protected]> wrote: > > I generally agree with not wanting to strictly *block* any forward progress > on waiting for the alternative impl, especially since a good comparative > assessment here will need a reasonable first-pass impl for both anyways, so > I have nothing against dusting off the existing prototype as Robert and JB > suggest. > > As with any other launch-and-iterate cases, this just means putting more > discernment later in the lifecycle when it comes time to decide on > long-term supportability details, beta/experimental tags, and > evolving/removing vestigial code later on if it becomes obsolete. > > Overall, I think the work we do here for both impls will prove valuable > regardless of how it ends up long-term (even if we end up deleting one or > the other, I'd still count the journey as a win if we learn insights from > it). > > I also forgot to call out earlier my appreciation for Yuewei's well > organized and well-grounded design document; many thanks for putting it > together! Given the concrete mappings, explicit design choices, etc., we > can more easily extract insights from where any of our base assumptions > didn't hold if it hits any limitations, or alternatively it can prove that > the model translates to the real system cleanly. > > I also think this prototyping goes hand-in-hand with the "Consistent > multi-object changes" discussion rather than being extraneous. IIRC we had > some discussion in a community meeting about whether the > PolarisResolutionManifest in itself could already be precisely the > representation of the "set of dependencies" from the "input state" that > need to be validated atomically with the final commit, but I forgot to > follow up on that one in the dev mailing list. > > I'll post some thoughts there soon, but in a nutshell, I think we probably > *don't* want to express a full-fledged "transaction" semantic where a > session sees its own uncommitted copy of an entity through a series of > read + write + read + write operations. Instead, the > PolarisResolutionManifest already carries the entity dependencies of the > current operation, including their entityIds and entityVersions. Now, we > probably don't want to turn *all* single-entity updates into a "conditional > batch update"-equivalent, but we could express the cases where a set of > commits are dependent on the state of another set of dependency states well > in that form, and it seems to align well with DynamoDB's TransactWriteItems. > > > > On Wed, Sep 30, 2026 at 4:33 PM Yuewei Zhou <[email protected]> > wrote: > > > Hi all, thanks for all the insights provided so far. > > > > Short answer to Yufei's question first: no - global versioning isn't > > necessary for the DynamoDB use case I'm targeting, and honestly I agree > > with Yufei that the Atomic path doesn't add significant complexity here. > > > > Polaris's correctness is already per-entity such as entityVersion CAS on > > update, conditional-on-non-existence on create, and a conditional write for > > name uniqueness. The one genuinely multi-entity atomic case, the Iceberg > > multi-table commit (writeEntities), maps directly onto DynamoDB's > > TransactWriteItems. None of this diverges meaningfully from DynamoDB's > > native characteristics or normal usage. Conditional writes and > > TransactWriteItems are exactly the primitives the service is built around. > > > > On the other hand, a global named-pointer/HEAD version would buy > > whole-catalog snapshot consistency we don't need, at the cost of write > > contention on the single reference plus cleanup of immutable objects which > > is exactly what we want to avoid for a serverless, high-write-concurrency > > AWS-native metastore. > > > > Thanks, > > Yuewei > > > > > > On Wed, Sep 30, 2026 at 1:46 PM Yufei Gu <[email protected]> wrote: > > > > > I don't think using AtomicOperationMetaStoreManager for DynamoDB will > > > introduce significant complexity regarding DynamoDB transactions. > > > Additionally, I'm not sure if DynamoDB users need the global versioning > > > feature, given the performance and cleanup overhead it requires. > > > > > > Maybe Yuewei can clarify whether global versioning is necessary for his > > use > > > case. > > > > > > Yufei > > > > > > > > > On Tue, Sep 29, 2026 at 9:59 PM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi all > > > > > > > > I agree with Robert's assessment here: persistence/nosql is the most > > > > natural and architecturally sound home for DynamoDB. > > > > > > > > A few thoughts from my side: > > > > 1. BasePersistence and AtomicOperationMetaStoreManager implicitly > > > > assume relational multi-entity transaction semantics (as implemented > > > > by JDBC via serializable transactions). Forcing DynamoDB into that > > > > shape means orchestrating TransactWriteItems to handle multi-entity > > > > updates (writeEntities), which introduces significant complexity > > > > around DynamoDB's transaction item/size limits, retry strategies, and > > > > ambiguous outcome handling. > > > > 2. In contrast, the persistence/nosql framework was built specifically > > > > around the primitive that KV/NoSQL stores natively provide: single-row > > > > conditional updates (CAS) on named pointers, with immutable objects > > > > written beforehand. This eliminates distributed rollback issues, > > > > naturally fits DynamoDB's consistency model, and stays well within > > > > DynamoDB's 400 KB item limit; > > > > 3. While Dennis makes a fair point about wanting comparative evidence > > > > and understanding the concurrency trade-offs between per-entity CAS > > > > and named-pointer commits, I don't think we should block delivering an > > > > AWS-native DynamoDB backend on that research. We can always run > > > > benchmarks and comparative experiments in parallel or as part of the > > > > broader persistence discussion (we used the same approach when we > > > > moved from OpenJPA to JDBC). > > > > > > > > Given that Robert already has a working implementation on his fork, I > > > > propose we use that as the baseline and work together (Yuewei, Robert, > > > > and anyone interested) to slice it into clean, reviewable, minimal PRs > > > > validated against the existing > > > > persistence/nosql/persistence/correctness test suite. > > > > > > > > Thoughts? > > > > > > > > Regards > > > > JB > > > > > > > > On Tue, Sep 29, 2026 at 4:31 PM Robert Stupp <[email protected]> wrote: > > > > > > > > > > Hi Dennis, > > > > > > > > > > I agree that the borderline cases would make useful apples-to-apples > > > > > validation scenarios. > > > > > > > > > > I do not think we should make implementing a DynamoDB > > `BasePersistence` > > > > > adapter under `AtomicOperationMetaStoreManager` a prerequisite for > > > adding > > > > > DynamoDB, though. > > > > > > > > > > The current atomic-manager API is not limited to single-entity > > updates. > > > > > For example, it calls `writeEntities()` with a list for multi-entity > > > > create > > > > > and update operations. > > > > > JDBC implements that list as one database transaction (using > > > SERIALIZABLE > > > > > isolation). > > > > > A DynamoDB implementation on this path would therefore need to define > > > and > > > > > validate much more than the single-entity CAS case: > > > > > > > > > > - which multi-entity operations use `TransactWriteItems`, and their > > > > > atomicity domain and limits; > > > > > - how conflicts, retries, and ambiguous post-send outcomes are > > > > represented; > > > > > - how grants, cleanup intent, and related state remain consistent; > > and > > > > > - which cases are intentionally unsupported rather than silently > > > partial. > > > > > > > > > > That is worthwhile design work, but it is the broader > > > > persistence-contract > > > > > discussion we already have on dev@. > > > > > It would effectively require us to build and validate a second > > DynamoDB > > > > > persistence model before delivering the stated optional-backend use > > > case. > > > > > > > > > > The existing NoSQL persistence framework is a more natural fit for > > > > DynamoDB. > > > > > Its conditional named-reference update is the publication point for a > > > > > validated catalog state, while immutable objects can be written > > before > > > > that > > > > > point. > > > > > DynamoDB's conditional writes map directly to that model, including > > its > > > > > consistency and conflict behavior. > > > > > > > > > > That path still needs the normal upstream work: > > > > > Reviewable adapter and runtime-integration slice(s), the existing > > > > > correctness/conformance coverage, and DynamoDB-specific tests for > > > > > conditional-write and unknown-outcome behavior. > > > > > > > > > > I am very much in favor of benchmarks and comparative evidence. > > > > > We can define those scenarios and run them against both > > implementations > > > > > where useful. > > > > > I would keep that as validation work, rather than making the optional > > > > > DynamoDB backend wait for a competing implementation to be designed > > and > > > > > built first. > > > > > > > > > > Cheers, > > > > > Robert > > > > > > > > > > On Tue, Sep 29, 2026 at 12:53 AM Dennis Huo <[email protected]> wrote: > > > > > > > > > > > From what I remember of where the persistence discussion left off, > > we > > > > > > wanted some of the borderline examples to actually be used for > > > > > > apples-to-apples comparison between the NoSQL vs > > > AtomicMetaStoreManager > > > > > > approaches. > > > > > > > > > > > > Especially nowadays when prototyping can be done more quickly, I'd > > > > prefer > > > > > > to take this opportunity to use DynamoDB to help quantify the > > > tradeoffs > > > > > > between the two, and following through with at least a first draft > > of > > > > > > Yuewei's proposed design. > > > > > > > > > > > > The design's assessment of the single-entity CAS basic-case + > > > > > > TrasactWriteItems special-case makes sense to me as aligning with > > the > > > > core > > > > > > design of the AtomicMetaStoreManager. I guess we might have to look > > > at > > > > > > whether anything has drifted further from that model in newer > > > features > > > > > > though. > > > > > > > > > > > > On Mon, Sep 28, 2026 at 10:59 AM Yuewei Zhou < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks Robert - and it's great you already have a DynamoDB > > backend > > > > > > > implementation in your fork. > > > > > > > > > > > > > > I'd like to stay involved in bringing DynamoDB support to > > Polaris. > > > Is > > > > > > there > > > > > > > anything I can pick up to help move it forward? Happy to dig > > > > > > > into whatever's useful. > > > > > > > > > > > > > > Thanks, > > > > > > > Yuewei > > > > > > > > > > > > > > On Mon, Sep 28, 2026 at 3:19 AM Robert Stupp <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Just to chime in here: I don't think we need to turn this into > > a > > > > choice > > > > > > > > between two persistence models. > > > > > > > > > > > > > > > > I would prefer DynamoDB to be added as another backend in the > > > > existing > > > > > > > > NoSQL persistence framework, rather than as a new > > BasePersistence > > > > > > adapter > > > > > > > > below AtomicOperationMetaStoreManager. > > > > > > > > > > > > > > > > The NoSQL reference update CAS is intentional. > > > > > > > > That is the point where a validated catalog change becomes > > > visible > > > > as > > > > > > one > > > > > > > > consistent state. > > > > > > > > It gives us a way to handle changes involving related entities > > > and > > > > > > > > cross-entity validation consistently. > > > > > > > > > > > > > > > > There are broader metastore questions behind this, but they are > > > > already > > > > > > > > being discussed in the "[DISCUSS] Consistent multi-object > > changes > > > > in > > > > > > > > Polaris persistence" thread [1]. > > > > > > > > > > > > > > > > This includes how we obtain current state, what belongs to one > > > > logical > > > > > > > > operation versus one backend commit attempt, and how conflicts, > > > > > > retries, > > > > > > > > and unknown outcomes are handled. > > > > > > > > Those questions apply to JDBC and NoSQL alike, so I don't think > > > we > > > > need > > > > > > > to > > > > > > > > resolve them again specifically for DynamoDB. > > > > > > > > > > > > > > > > FWIW, I already have a DynamoDB implementation in my public > > fork > > > > [2]. > > > > > > > > > > > > > > > > It may be useful as a starting point for the DynamoDB adapter > > and > > > > its > > > > > > > > runtime integration. > > > > > > > > We could then split the work into smaller reviewable pieces and > > > > use the > > > > > > > > existing correctness tests. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Robert > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/vx0k8ow4k87m4y7cxpmojb0zy17t5ldy > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/snazy/polaris/tree/persistence-nosql-all-backends/persistence/nosql/persistence/db/dynamodb > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 11:57 PM Yuewei Zhou < > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > Following the discussion on the GitHub issue, and as > > suggested > > > > there, > > > > > > > I'd > > > > > > > > > like to bring this proposal to the list for wider visibility > > > and > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > Summary: a new, opt-in persistence.type=dynamodb backend that > > > > reuses > > > > > > > the > > > > > > > > > existing AtomicOperationMetaStoreManager (per-entity > > > > compare-and-set) > > > > > > > > over > > > > > > > > > a new DynamoDbBasePersistence adapter - the same shape > > > > > > relational-jdbc > > > > > > > > > uses, so the manager, catalog API, and entity model are > > > > unchanged. > > > > > > The > > > > > > > > > motivation is a serverless, AWS-native metastore, so AWS > > > > deployments > > > > > > > > don't > > > > > > > > > have to run a stateful Postgres or MongoDB purely to hold > > > catalog > > > > > > > > metadata. > > > > > > > > > > > > > > > > > > On the issue we aligned that this fits Polaris's usual > > pattern > > > > for > > > > > > > > optional > > > > > > > > > backends - a self-contained, pluggable module, selected by > > > > config, > > > > > > with > > > > > > > > no > > > > > > > > > default change, no migration, and no new dependency or > > > > enforcement of > > > > > > > any > > > > > > > > > kind on users or downstream projects that don't enable it - > > > > which is > > > > > > a > > > > > > > > > reasonable basis to start building on and kick off a series > > of > > > > PRs. > > > > > > I'd > > > > > > > > of > > > > > > > > > course still welcome input on the design as it takes shape. > > > > > > > > > > > > > > > > > > One trade-off I'd especially like thoughts on: this > > > > per-entity-CAS > > > > > > > > approach > > > > > > > > > versus layering DynamoDB under the existing NoSQL framework > > > > (single > > > > > > > > > named-pointer / HEAD commit) - roughly, independent-write > > > > concurrency > > > > > > > on > > > > > > > > > one side vs. atomic multi-entity commits and cross-entity > > > > validation > > > > > > on > > > > > > > > the > > > > > > > > > other. > > > > > > > > > > > > > > > > > > GitHub issue: https://github.com/apache/polaris/issues/5615 > > > > > > > > > Design doc: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1vc7FCvznlfdBoHe0DJQo1vRzcd0AXHFX4cGrFhuiF9Y/edit?usp=sharing > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > ZephyrYWZhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
