Dear all, thank you for the great discussion in the last Catalog community sync, sorry I couldn't be there.
I worked some results of the discussion into the proposal, most importantly, the TableMetadata was removed from the create & register table and create view events. It is easy to add this as optional in the future if we see the need. I believe the information we miss out on can be obtained easily enough by just loading the table. Please note that due to this change, the event endpoint is now no full Event Log anymore. That is, the current state of a table cannot be re-created solely from events. I also changed a handful of names & comments based on comments from Robert, thanks! The diff to the previous state is linked at the top of the proposal doc: https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing There is one remaining open discussion point from the last sync: Should the `UpdateTableOperation` require the field `updates`. The main reason brought up against this field being required are security considerations, as TableUpdates might contain sensitive data such as Schemas. A `TableUpdate` without any updates information can't be used well for Federation, except for basic CacheInvalidation, which could already be done with etags. Also for Auditing purposes we typically want to know what change happened (i.e. which column was removed). Thus, I don't see much value in a `UpdateTableOperation` without the `updates` field. Catalog implementations should of course only return data for entities if the principal making the request is authorized to see it. But as authorization is implementation specific, we don't need to specify this behaviour in the spec. Audit information is never public. Even the presence of a table can be sensitive information. If I missed any other point open for discussion, please respond here in the Mailing List, so that we have a clear view of what needs further discussion. I hope we can clear all remaining requests in the next Catalog Community Sync next week (9th of July): https://docs.google.com/document/d/1iPGVCIcr-M0XtAiudOguWAvmqIdVgpYN5vz5ohO8PKw/edit?usp=sharing Thanks everyone! Christian On Tue, 17 Jun 2025 at 19:56, Christian Thiel <christian.t.b...@gmail.com> wrote: > There are currently no open change requests that I am aware of. > I would ask all interested parties to review the current proposal and post > open points here or bring them up in the Catalog sync tomorrow. > Kevin (thanks!) will moderate this topic tomorrow as I am travelling and > probably won't make it in time. > > > On Mon, 16 Jun 2025 at 15:29, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi >> >> I think Christian updated the spec with the latest discussion >> (especially about actor object). >> I would suggest making a new review and maybe sync-up during the next >> Catalog Community meeting. >> >> Regards >> JB >> >> On Sat, Jun 14, 2025 at 1:18 AM Adnan Hemani >> <adnan.hem...@snowflake.com.invalid> wrote: >> > >> > Hi all, >> > >> > Wanted to see if there were any remaining action items or discussions >> remaining on this topic. I’m trying to get a gauge of how close we might be >> on getting this merged! >> > >> > Best, >> > Adnan Hemani >> > >> > On 2025/05/28 17:33:20 Christian Thiel wrote: >> > > Thank you all for the great discussion today! >> > > I have updated the proposal. Key changes are: >> > > >> > > - >> > > >> > > Specify that clients should ignore unknown operation & event types >> > > - >> > > >> > > Specify the `actor` field as part of the `Event` schema as an >> opaque >> > > string. Remove the `Actor` type. >> > > - Remove the `actors` filter from the `GetEventsRequest`, add an >> > > extendable `custom-filters` object instead (`additionalProperties: >> true`) >> > > >> > > The diff for the changes since the sync today is available in Github: >> > > >> https://github.com/apache/iceberg/pull/12584/commits/3de9a7c5d128b1100c38ce688603c94491008d35 >> > > The google doc is also updated, >> > > >> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing >> > > >> > > Looking forward to more feedback, especially regarding custom >> operations! >> > > >> > > On Tue, 27 May 2025 at 10:15, Christian Thiel <ch...@gmail.com> >> > > wrote: >> > > >> > > > Dear all, >> > > > >> > > > I think we have reached mostly consensus here. >> > > > There is one more change since our last discussion: We removed the >> > > > recursive "assumed-by" field of actors in favor or an "actor-chain" >> list. >> > > > >> > > > If there is any more need for discussion please voice it here on the >> > > > Mailing List or in the Catalog sync tomorrow. I would otherwise >> start a >> > > > vote. >> > > > >> > > > Best, >> > > > Christian >> > > > >> > > > On Wed, 7 May 2025 at 11:18, Christian Thiel <ch...@gmail.com> >> > > > wrote: >> > > > >> > > >> Dear all, >> > > >> >> > > >> I worked the changes discussed in the last catalog sync into the >> Events >> > > >> proposal [1]. >> > > >> Those include: >> > > >> - Using request-id instead of transaction >> > > >> - A more flexible User (now called Actor) >> > > >> - Custom Operation type >> > > >> >> > > >> The specific diff compared to the last discussion can be best seen >> in my >> > > >> latest commit in git [2]. >> > > >> It would be good to still comment in Google Docs so that we have >> > > >> everything in one place. >> > > >> >> > > >> Best, >> > > >> Christian >> > > >> >> > > >> [1]: >> > > >> >> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing >> > > >> [2]: >> > > >> >> https://github.com/apache/iceberg/pull/12584/commits/4d67051e03d5345687566b3900db3af23ce15766 >> > > >> >> > > >> >> > > >> On Wed, 16 Apr 2025 at 14:53, Christian Thiel <ch...@gmail.com> >> > > >> wrote: >> > > >> >> > > >>> Dear all, >> > > >>> after the last Catalog sync I updated the proposal. >> > > >>> Changes are in the original proposal Document [1] and the >> original PR [2] >> > > >>> >> > > >>> Best, >> > > >>> Christian >> > > >>> >> > > >>> [1] >> > > >>> >> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing >> > > >>> [2] https://github.com/apache/iceberg/pull/12584 >> > > >>> >> > > >>> >> > > >>> On Thu, 20 Mar 2025 at 10:49, Christian Thiel < >> > > >>> christian.t.b...@gmail.com> wrote: >> > > >>> >> > > >>>> Dear all, >> > > >>>> >> > > >>>> We have recently discussed in the Iceberg Catalog Community Sync >> [1] >> > > >>>> and the Mailing List [2] different ways on how federation >> between Catalogs >> > > >>>> could be standardized. >> > > >>>> >> > > >>>> This proposal introduces a /events endpoint to the IRC >> specification. >> > > >>>> The endpoint provides events of modifications to objects managed >> by the >> > > >>>> Catalog (tables, namespaces, views), allowing consumers to >> efficiently >> > > >>>> track metadata changes using persistent offsets for reliable >> consumption >> > > >>>> and resumability. >> > > >>>> >> > > >>>> Proposal Document: >> > > >>>> >> https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?usp=sharing >> > > >>>> >> > > >>>> I am looking forward to your thoughts and hope we find time in >> next >> > > >>>> week's Catalog sync to discuss this further. >> > > >>>> >> > > >>>> Best >> > > >>>> Christian >> > > >>>> >> > > >>>> [1] Catalog Community Sync Feb. 2025 >> > > >>>> https://www.youtube.com/watch?v=hYcehreE8Nk >> > > >>>> [2] Mailing List, September 2024 - Notifications Endpoint: >> > > >>>> https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht >> > > >>>> >> > > >>> >> > > >> >
