Hi Qiegang, Thanks for the design doc, it reads well, and moving the writes onto table branches makes the publish much easier to reason about. Some thoughts on a few of the open questions, from the #3469 work and from adding multi-table commits to PyIceberg (apache/iceberg-python#4123):
1/ Server-side publish (14.2, question 5). I'd lean towards offering it, at least as an option. The client-side publish rules are hard to get the same across clients. PyIceberg, for example, can fast-forward a branch but has no cherry-pick, so a Python client couldn't re-apply appends on a moved main today without implementing it first. While adding commit_transactions to PyIceberg, I also found that a delete that matched no rows staged nothing, and the transaction went out without its main-hasn't-moved check, so it wasn't validated. Small gaps like that are easier to close once on the server than in every client. The trade-off you describe (no client-side replay) still lets clients that want replay use the standard commit. 2/ The fast-forward case of the publish (section 9: main still at S0, fast-forward main to the branch head, assert the branch head) is the recipe that #5740 documents and tests: BranchPromotionRecipeTest covers both the success case and "one table moved on main, nothing changes". It may be useful as a baseline test for the publish, so the two describe the same commit the same way. 3/ Snapshot expiry (14.3, question 16). Rejecting remove-snapshots for snapshots that a registered read point still covers seems the safer default to me, since registration is opt-in and explicit. For unregistered handles inside the retention period, I'd rather keep the 410 than fail maintenance jobs, so expire_snapshots doesn't start failing on catalogs where nobody uses long transactions. Since you're taking questions 11-16 to the Iceberg dev thread, I'll add this there as well. 4/ Branch visibility (14.1, question 1). If the branches stay visible, a reserved name prefix would help: users and tools could tell staging branches apart from their own, and it avoids name collisions with user branches. Happy to review the follow-up document and the split PRs. Regards, Ankur On Mon, Oct 19, 2026 11:33 PM, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Qiegang, > > Welcome to the Polaris community! > > Thanks for starting this thread. > > Multi-table transactions come up regularly here :) (for instance see #3469 > about catalog-level branching for atomic multi-table operations, #5807 > about multi-table commit not validating requirements on tables without > updates, etc). > So yes, I think there is interest, and starting as an optional Polaris > extension is a good idea (waiting for an Iceberg proposal). > > For a change of this size, I would suggest: > - Let's keep the design discussion here in this thread. Decisions are made > on the mailing list. > - I think it's great to create a GH Issue with the "proposal" label, > pointing to this thread and the design doc. It will be the umbrella issue > to track the work and the PRs. > - If you can, share the design doc as a Google Doc open for comments, or PR > with md files. > - Once we have consensus on the API and the persistence SPI, we can go with > small and focused PRs. I checked and your branch is around 19k lines across > ~ 100 files, so I think split is welcome, else the review will be tricky. A > a reminder, new public interfaces and extension points should be discussed > and agreed (lazy consensus at least) on the dev mailing list. > > As you are moving the write path to Iceberg table branches, I would wait > for the updated design before a detailed review. > > In the meantime, a few early thoughts from a first read :) : > - the PoC relies on PostgreSQL for atomicity (row locks, the commit_scn > count row, etc). Polaris has several persistence backends now (JDBC, NoSQL > with MongoDB/DynamoDB/...). Especially the NoSQL ones don't rely on > relational transactions. Upstream, I think the transaction state should go > through a persistence SPI, with JDBC as the first implementation. It would > help to describe the minimal operations that backends have to provide. > - With "Universal CTS", all commits of a catalog serialize one one row, > even for users who never use transactions. Maybe a feature flag for that > would be appropriate. > - The txn_snapshot/txn_id parameters on loadTable extend an Iceberg > endpoint, and the snapshot lifecycle is an Iceberg question. We try to stay > aligned with the spec on the Iceberg endpoint, so I suggest sharing the PoC > on the Iceberg dev mailing list also. > - I'm completely fine with autorization on the transaction endpoints not > complete yet. I think the new operations have to be mapped to the Polaris > privilege model, so they also work with the pluggable authorizers (OPA, > Ranger, ...). > - Today, COMMIT reuses the transaction workspace buffering, which PR #5088 > (currently in review) is replacing with an explicit buffering. Probably a > good idea to rebase your branch on main and following #5088. > > Also, the README says the copyright is held by Cloudera. It will probably > change before any merge (and depending of what has been done, maybe a SGA > would be required and an IP-Clearance). Let me know if I can help on this. > > Regards > JB > > On 2026-10-19 19:30 UTC Qiegang Long wrote: > > Hi all, > > > > Thanks to everyone who came to the multi-table transactions talk at > > Lakehouse Day in Glasgow, and for the conversations afterwards. > > Several of you suggested continuing here. I'm new to the Polaris > > community: I built this as a proof of concept on Polaris, and this is > > the first time I'm bringing it to the list. > > > > Iceberg already has most of the foundation for transactions engines > > can use: the REST commitTransaction bundle commits several tables > > atomically, and table branches can stage writes. What's missing is the > > engine usable transaction around them: BEGIN ... COMMIT across > > statements, with one consistent read point across tables. The > > discussion of this on the Iceberg list has been quiet for a while, and > > I built the PoC to restart it with something concrete. > > > > The PoC runs Spark and Trino against a Polaris fork: multi-statement > > transactions at READ COMMITTED and REPEATABLE READ, a read point > > shared across engines, and fan-out writes from independent workers > > that commit atomically. After conversations with Iceberg folks at the > > conference, I'm moving the write path onto Iceberg table branches, so > > writes become standard REST updates; I'll follow up in this thread > > with the updated code. > > > > For example, in Spark with the Polaris plugin: > > > > BEGIN TRANSACTION; > > INSERT INTO customers VALUES (22, 'joe', '[email protected]'); > > INSERT INTO orders VALUES (2002, 22, 9.99, 'pending'); > > COMMIT; -- both tables become visible together, or neither does > > > > Readers in Spark or Trino, at either isolation level, never see the > > order without its customer. > > > > The change to the Iceberg REST surface is small: one optional > > loadTable parameter (the read point). Everything else is Polaris-only > > endpoints (minting a read point, fan-out coordination). So it could > > live in Polaris first as an opt-in extension, with an Iceberg proposal > > later if the community wants one. > > > > Would Polaris be interested in carrying it as an extension first? And > > since I'm new here, I'd appreciate guidance on the usual path for a > > change of this size: for example a design review here first, an issue > > to track it, time at a community sync, or going straight to small PRs. > > I'm happy to work whichever way suits the project. It's an early PoC > > with known gaps (listed in the design doc), and JDBC/PostgreSQL only > > for now. > > > > Talk, slides, demo video, design doc and code: > > https://github.com/qlong/lakehouse-transactions > > > > Thanks, > > Qiegang > > >
