Hi Ankur, Yufei, Thanks to both of you, and Ankur, sorry for the slow reply.
Yufei: yes, that's the design doc: https://docs.google.com/document/d/1ew2dOI_BHQFP1C8Q46Y_m_AvNNH4rzzLKaSsB089GTU/edit?tab=t.0 Reviewing it here first and splitting the implementation into small PRs afterwards is the plan. Comments directly in the doc work too. The earlier messages are in the archive, in case they went to spam: https://lists.apache.org/thread/d5ft79ogcxx15ccr1n1bc0rvsl14tf6v Ankur, on your points: 1. Server-side publish (question 10): thanks, the missing cherry-pick in PyIceberg is a good example of how hard the publish rules are to keep the same across clients. The server already publishes fan-out transactions this way, so offering it for single-session is mainly a question of moving that work, and its rebase cost, onto the catalog for the common case. Scalability, and leaving the client free to resolve a conflict itself (a server can only re-validate a change, not replay it), were the reasons the design publishes single-session transactions on the client. Definitely open on this, IMHO it is a critical decision point. On the delete that matched no rows: in the design, a branch that was never written is dropped without an assertion on main. Under snapshot isolation that's correct, since such a delete is a read. Under SERIALIZABLE it has to be validated like any other read, which is part of the read-set validation listed as future work. 2. #5740: thanks, I'll use BranchPromotionRecipeTest as the baseline for the fast-forward case, so the recipe and the publish describe the same commit. 3. Snapshot expiry (question 16): rejecting remove-snapshots only for snapshots a registered read point still needs, and keeping 410 for unregistered handles, sounds right to me. 4. Branch visibility (question 1): a reserved prefix is a good option (the PoC already names staging branches with a txn_ prefix), will add this along with alternatives. I'll add these to the open questions in the doc. Thanks, Qiegang On Fri, Nov 6, 2026 at 7:50 PM Yufei Gu <[email protected]> wrote: > > Hi Qiegang, > > Thanks for sharing this and putting together the PoC! Sorry for the > late reply. This would be a great feature to have in Polaris as we > discussed at Coc Glasgow. > > I'd suggest starting with a review of the design doc on the dev list > so we can align on the scope, transaction semantics, and API changes. > Once we agree on the design, we can break the implementation into > smaller PRs. I believe this is the design doc, right? > https://docs.google.com/document/d/1ew2dOI_BHQFP1C8Q46Y_m_AvNNH4rzzLKaSsB089GTU/edit?tab=t.0 > > Sorry if I missed other discussions, some ASF mails have recently > landed in my spam folder. > > Yufei > > On Mon, Oct 19, 2026 at 12:32 PM Qiegang Long <[email protected]> wrote: > > > > Hi all, > > > > Thanks to everyone who came to the multi-table transactions talk at > > Lakehouse Day in Glasgow, and for the conversations afterwards. > > Several of you suggested continuing here. I'm new to the Polaris > > community: I built this as a proof of concept on Polaris, and this is > > the first time I'm bringing it to the list. > > > > Iceberg already has most of the foundation for transactions engines > > can use: the REST commitTransaction bundle commits several tables > > atomically, and table branches can stage writes. What's missing is the > > engine usable transaction around them: BEGIN ... COMMIT across > > statements, with one consistent read point across tables. The > > discussion of this on the Iceberg list has been quiet for a while, and > > I built the PoC to restart it with something concrete. > > > > The PoC runs Spark and Trino against a Polaris fork: multi-statement > > transactions at READ COMMITTED and REPEATABLE READ, a read point > > shared across engines, and fan-out writes from independent workers > > that commit atomically. After conversations with Iceberg folks at the > > conference, I'm moving the write path onto Iceberg table branches, so > > writes become standard REST updates; I'll follow up in this thread > > with the updated code. > > > > For example, in Spark with the Polaris plugin: > > > > BEGIN TRANSACTION; > > INSERT INTO customers VALUES (22, 'joe', '[email protected]'); > > INSERT INTO orders VALUES (2002, 22, 9.99, 'pending'); > > COMMIT; -- both tables become visible together, or neither does > > > > Readers in Spark or Trino, at either isolation level, never see the > > order without its customer. > > > > The change to the Iceberg REST surface is small: one optional > > loadTable parameter (the read point). Everything else is Polaris-only > > endpoints (minting a read point, fan-out coordination). So it could > > live in Polaris first as an opt-in extension, with an Iceberg proposal > > later if the community wants one. > > > > Would Polaris be interested in carrying it as an extension first? And > > since I'm new here, I'd appreciate guidance on the usual path for a > > change of this size: for example a design review here first, an issue > > to track it, time at a community sync, or going straight to small PRs. > > I'm happy to work whichever way suits the project. It's an early PoC > > with known gaps (listed in the design doc), and JDBC/PostgreSQL only > > for now. > > > > Talk, slides, demo video, design doc and code: > > https://github.com/qlong/lakehouse-transactions > > > > Thanks, > > Qiegang
