Hi Andrew, It is difficult to move this mechanism to the Iceberg sink. The table structure change in Iceberg's design requires generating a new snapshot, which poses significant challenges to schema evolution.
Another approach is to create a snapshot compatible way for Paimon to generate Iceberg, which is what we are working on. Best, Jingsong On Fri, May 24, 2024 at 8:11 PM Andrew Otto <[email protected]> wrote: > > Hi! > > How coupled to Paimon catalogs and tables is the cdc part of Paimon? > RichCdcMultiplexRecord and related code seem incredibly useful even outside > of the context of the Paimon table format. > > I'm asking because the database sync action feature is amazing. At the > Wikimedia Foundation, we are on an all-in journey with Iceberg. I'm > wondering how hard it would be to extract the CDC logic from Paimon and > abstract the Sink bits. > > Could the table/database sync with schema evolution (without Flink job > restarts!) potentially work with the Iceberg sink? > > Thanks! > -Andrew Otto > Wikimedia Foundation > > > >
