Thanks, Maninder! Good idea. Is any meeting for this already scheduled?
Cheers, Dmitri. On Fri, Jun 27, 2025 at 1:52 AM Maninderjit Singh < parmar.maninder...@gmail.com> wrote: > Thanks Dmitri! > I will add this to the doc. Also, it might be a good idea to discuss it in > a meeting so we can hash out the details. > > On Wed, Jun 25, 2025, 9:23 AM Dmitri Bourlatchkov <di...@apache.org> > wrote: > >> Hi Maninder, >> >> Thanks for adding a section on opaque IDs and apologies for delayed reply >> from my side. I could not find a place where to fit my text in the doc, so >> I'm sending it in this email :) >> >> This option is mostly related to option 2 (CSN) but proposes to use >> commit IDs (alternative to CSN) that are opaque to clients - this is the >> same as in your opaque ID section in the doc, but I hope that thoughts >> below might help to clarity how it is intended to work. The main difference >> is delegating the resolution of commit IDs to snapshots to catalog servers. >> >> Catalog Servers are free to use any implementation for commit IDs, >> including monotonically increasing numbers (but they are not limited to >> CSN). >> >> Catalog Servers produce a commit ID for every change, which will be >> exposed to clients as a reasonably short string. Multi-table changes >> naturally get the same commit ID. >> >> Commit IDs are part of REST Catalog responses, but do not have to be in >> the metadata files. No Iceberg spec changes are required. REST API changes >> are needed, but they are optional and transparent to clients, unless the >> client wishes extra consistency guarantees. >> >> Clients can request table metadata for any table using a particular >> commit ID. This mechanism can be used to ensure consistency in time-travel >> queries. >> >> An engine can proceed as follows, while executing a multi-table change: >> 1. Load table A - receive metadata and commit ID C1 >> 2. Load table B by providing C1 as a request parameter to the Catalog >> server >> 3. Load table C by providing C1 as a request parameter to the server >> 4. Process data in tables A, B, C >> 5. Update table A >> 6. Update table B >> 7. Submit metadata updates for A and B to the Catalog, passing C1 as the >> “base” commit ID to the server. Additionally submit the name C as a “read >> but not changed” table. >> 8. The Catalog server checks whether the change has any conflicts between >> C1 and the current state of the catalog (including validating that C has >> not changed) >> 9. The Catalog commits changes and returns commit ID C2 to the client >> (this commit ID represents the committed state of the submitted metadata >> changes). >> >> If the commit fails due to conflicts, the client receives a “conflict” >> error and a commit ID C3, which represents the most up-to-date state of the >> catalog (the state that was conflicting with the submitted changes). The >> client then re-loads tables based on C3 and retries its workflows. >> >> Load table responses when a commit ID is provided do not have to return >> all of the table's metadata. It is sufficient to return only the most >> relevant snapshots (usually the latest plus its parent). This is similar to >> the partial metadata loading proposal, but not critical for consistency >> guarantees. The critical part is that the Catalog communicates to engines >> what snapshot is current for a particular commit ID. >> >> Resolving Time Travels Queries: When a client executes a time travel >> query, the client provides a timestamp when loading the first table that is >> included in the query. The Catalog will resolve the timestamp to a commit >> ID and include it in the response. Client using the returned commit ID to >> load subsequent tables. >> >> Optionally a new endpoint may be added to the REST Catalog API to handle >> the resolution of timestamps to commit IDs. >> >> Caching Metadata on the Client Side: Reloading table metadata for a >> particular snapshot could leverage the ETag mechanism to reduce the amount >> of network traffic. >> >> Servers do not need to keep any in-progress state for transactions. The >> same multi-table commit mechanism servers have for the existing commit >> endpoint can be extended to also produce commit IDs. Resolving timestamps >> to commit ID is an implementation detail. Some changes in existing servers >> will probably be required for that. Conceptually this problem does not >> appear to be more complex than providing a monotonic CSN or implementing >> the existing multi-table commit endpoint. >> >> Retention of the data related to time-travel is a server-side concern. If >> a client wishes to time travel to a point that no longer has commit >> tracking information and error is returned. >> >> WDYT? >> >> Thanks, >> Dmitri. >> >> On Thu, Jun 19, 2025 at 6:24 PM Maninderjit Singh < >> parmar.maninder...@gmail.com> wrote: >> >>> Thanks Dmitri for the review! >>> >>> We have been deliberate about not including server side implementation >>> for brevity and to allow each vendor to choose the best option for them. >>> Having said that, I have included a few papers that you can reference. >>> >>> I have also added a new >>> <https://docs.google.com/document/d/1KVgUJc1WgftHfLz118vMbEE7HV8_pUDk4s-GJFDyAOE/edit?pli=1&tab=t.0#bookmark=id.typa1ivjs7pw> >>> section under alternative to explore opaque ids further. Could you validate >>> and fill in the details? There are a few open questions and >>> dependencies that would be required for this proposal: >>> >>> Why do we even need an opaque id, could we use tableIdentifier + >>> Sequence number as an implicit opaque id? >>> How are opaque ids compared across tables and with time? >>> Not clear on who issues the timestamp for opaque ids and how it >>> is achieving consistency beyond repeatable reads? >>> Would this require dependency on the partial metadata load proposal >>> <https://docs.google.com/document/d/1eXnT0ZiFvdm_Zvk6fLGT_UxVWO-HsiqVywqu1Uk8s7E/edit?tab=t.0#heading=h.t6emwabb4tkr> >>> ? >>> >>> Regards, >>> Maninder >>> >>> On Thu, Jun 19, 2025 at 12:12 PM Dmitri Bourlatchkov <di...@apache.org> >>> wrote: >>> >>>> Thanks for the quick response, Jagdeep! >>>> >>>> I can certainly add a section to the doc. Could you clarify what you >>>> mean by "chatty protocol", though. I did not find that term in the >>>> linked email discussion :) >>>> >>>> Thanks, >>>> Dmitri. >>>> >>>> On Thu, Jun 19, 2025 at 2:28 PM Jagdeep Sidhu <sidhujagde...@gmail.com> >>>> wrote: >>>> >>>>> Hi Dmitri, >>>>> >>>>> Thank you for reviewing. As you said, we previously explored and >>>>> dropped TransactionContext APIs with opaque IDs because it created a very >>>>> chatty protocol and also led to complex transaction state management on >>>>> Server side, link to old thread below. >>>>> >>>>> Would you add a section to the existing document on the approach you >>>>> are thinking - Opaque IDs without the chatty protocol and complex >>>>> transaction state management on Catalog Server? Then we can compare all of >>>>> them and discuss the best path forward. Thank you! >>>>> >>>>> Older thread - >>>>> https://lists.apache.org/thread/q7vgnfwdxng5q6mq45m0psghzy7553r7 >>>>> >>>>> -Jagdeep >>>>> >>>>> On Thu, Jun 19, 2025 at 10:42 AM Dmitri Bourlatchkov <di...@apache.org> >>>>> wrote: >>>>> >>>>>> Thanks for driving this proposal, Maninder! >>>>>> >>>>>> From my POV the need for Catalogs to provide a monotonic sequence >>>>>> number has deep implications on the catalog implementations. I added a >>>>>> related comment to the doc as well. >>>>>> >>>>>> The document does a good job at discussing the client operation. I'd >>>>>> appreciate it if the server-side impact were considered in more depth >>>>>> too, >>>>>> since the proposal implies changes on both sides. >>>>>> >>>>>> I know that an opaque "commit ID" was considered before, however, if >>>>>> I'm not mistaken previous discussions revolved around the idea of >>>>>> a TransactionContext as an entity exposed via new APIs for sharing state >>>>>> between clients/engines and the catalog. I'd like to revisit the idea of >>>>>> opaque transaction IDs (managed by the catalog) but without the use >>>>>> of TransactionContext. I made a brief comment about that in the doc, and >>>>>> I'm willing to expand on this. I believe it can be implemented >>>>>> without having a durable context object to represent a transaction >>>>>> between >>>>>> the client and the catalog. >>>>>> >>>>>> The main idea for "opaque commit IDs" is to allow more flexibility >>>>>> for Catalog implementations, while keeping the same client-side >>>>>> guarantees >>>>>> (snapshot isolation, causally consistent multi-table changes, etc.). >>>>>> >>>>>> Thanks, >>>>>> Dmitri. >>>>>> >>>>>> On Mon, Jun 16, 2025 at 9:32 PM Maninderjit Singh < >>>>>> parmar.maninder...@gmail.com> wrote: >>>>>> >>>>>>> Hi Iceberg dev community, >>>>>>> >>>>>>> We have been iterating on the Multi Table Transactions proposal and >>>>>>> have merged the proposals for using catalog authored timestamps and >>>>>>> sequence numbers together as well incorporated feedback from the >>>>>>> community: >>>>>>> Proposal: Multi-table multi-statement transactions for >>>>>>> Apache Iceberg REST Catalog >>>>>>> <https://drive.google.com/open?id=1KVgUJc1WgftHfLz118vMbEE7HV8_pUDk4s-GJFDyAOE> >>>>>>> >>>>>>> We have captured the tradeoffs involved with each approach as well >>>>>>> as the reasoning for making those choices. We would love to hear your >>>>>>> opinions on the consolidated proposal and which approach is more >>>>>>> suitable >>>>>>> for your requirements and why. >>>>>>> >>>>>>> Thank you in advance! >>>>>>> >>>>>>
