> I’m curious if there is any reason for choosing Iceberg instead of Paimon
No technical reason that I'm aware of. We are using it mostly because of momentum. We looked at Flink Table Store (before it was Paimon), but decided it was too early and the docs were too sparse at the time to really consider it. > Especially for a use case like CDC that iceberg struggles to support. We aren't doing any CDC right now (for many reasons), but I have never seen a feature like Paimon's database sync before. One job to sync and evolve an entire database? That is amazing. If we could do this with Iceberg, we might be able to make an argument to product managers to push for CDC. On Fri, May 24, 2024 at 8:36 AM Giannis Polyzos <ipolyzos...@gmail.com> wrote: > I’m curious if there is any reason for choosing Iceberg instead of Paimon > (other than - iceberg is more popular). > Especially for a use case like CDC that iceberg struggles to support. > > On Fri, 24 May 2024 at 3:22 PM, Andrew Otto <o...@wikimedia.org> wrote: > >> Interesting thank you! >> >> I asked this in the Paimon users group: >> >> How coupled to Paimon catalogs and tables is the cdc part of Paimon? >> RichCdcMultiplexRecord >> <https://github.com/apache/paimon/blob/cc7d308d166a945d8d498231ed8e2fc9c7a27fc5/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/sink/cdc/RichCdcMultiplexRecord.java> >> and >> related code seem incredibly useful even outside of the context of the >> Paimon table format. >> >> I'm asking because the database sync action >> <https://paimon.apache.org/docs/master/flink/cdc-ingestion/mysql-cdc/#synchronizing-databases> >> feature >> is amazing. At the Wikimedia Foundation, we are on an all-in journey with >> Iceberg. I'm wondering how hard it would be to extract the CDC logic from >> Paimon and abstract the Sink bits. >> >> Could the table/database sync with schema evolution (without Flink job >> restarts!) potentially work with the Iceberg sink? >> >> >> >> >> On Thu, May 23, 2024 at 4:34 PM Péter Váry <peter.vary.apa...@gmail.com> >> wrote: >> >>> If I understand correctly, Paimon is sending `CdcRecord`-s [1] on the >>> wire which contain not only the data, but the schema as well. >>> With Iceberg we currently only send the row data, and expect to receive >>> the schema on job start - this is more performant than sending the schema >>> all the time, but has the obvious issue that it is not able to handle the >>> schema changes. Another part of the dynamic schema synchronization is the >>> update of the Iceberg table schema - the schema should be updated for all >>> of the writers and the committer / but only a single schema change commit >>> is needed (allowed) to the Iceberg table. >>> >>> This is a very interesting, but non-trivial change. >>> >>> [1] >>> https://github.com/apache/paimon/blob/master/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecord.java >>> >>> Andrew Otto <o...@wikimedia.org> ezt írta (időpont: 2024. máj. 23., Cs, >>> 21:59): >>> >>>> Ah I see, so just auto-restarting to pick up new stuff. >>>> >>>> I'd love to understand how Paimon does this. They have a database >>>> sync action >>>> <https://paimon.apache.org/docs/master/flink/cdc-ingestion/mysql-cdc/#synchronizing-databases> >>>> which will sync entire databases, handle schema evolution, and I'm pretty >>>> sure (I think I saw this in my local test) also pick up new tables. >>>> >>>> >>>> https://github.com/apache/paimon/blob/master/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/SyncDatabaseActionBase.java#L45 >>>> >>>> I'm sure that Paimon table format is great, but at Wikimedia Foundation >>>> we are on the Iceberg train. Imagine if there was a flink-cdc full >>>> database sync to Flink IcebergSink! >>>> >>>> >>>> >>>> >>>> On Thu, May 23, 2024 at 3:47 PM Péter Váry <peter.vary.apa...@gmail.com> >>>> wrote: >>>> >>>>> I will ask Marton about the slides. >>>>> >>>>> The solution was something like this in a nutshell: >>>>> - Make sure that on job start the latest Iceberg schema is read from >>>>> the Iceberg table >>>>> - Throw a SuppressRestartsException when data arrives with the wrong >>>>> schema >>>>> - Use Flink Kubernetes Operator to restart your failed jobs by setting >>>>> kubernetes.operator.job.restart.failed >>>>> >>>>> Thanks, Peter >>>>> >>>>> On Thu, May 23, 2024, 20:29 Andrew Otto <o...@wikimedia.org> wrote: >>>>> >>>>>> Wow, I would LOVE to see this talk. If there is no recording, >>>>>> perhaps there are slides somewhere? >>>>>> >>>>>> On Thu, May 23, 2024 at 11:00 AM Carlos Sanabria Miranda < >>>>>> sanabria.miranda.car...@gmail.com> wrote: >>>>>> >>>>>>> Hi everyone! >>>>>>> >>>>>>> I have found in the Flink Forward website the following >>>>>>> presentation: "Self-service ingestion pipelines with evolving >>>>>>> schema via Flink and Iceberg >>>>>>> <https://www.flink-forward.org/seattle-2023/agenda#self-service-ingestion-pipelines-with-evolving-schema-via-flink-and-iceberg>" >>>>>>> by Márton Balassi from the 2023 conference in Seattle, but I cannot find >>>>>>> the recording anywhere. I have found the recordings of the other >>>>>>> presentations in the Ververica Academy website >>>>>>> <https://www.ververica.academy/app>, but not this one. >>>>>>> >>>>>>> Does anyone know where I can find it? Or at least the slides? >>>>>>> >>>>>>> We are using Flink with the Iceberg sink connector to write >>>>>>> streaming events to Iceberg tables, and we are researching how to handle >>>>>>> schema evolution properly. I saw that presentation and I thought it >>>>>>> could >>>>>>> be of great help to us. >>>>>>> >>>>>>> Thanks in advance! >>>>>>> >>>>>>> Carlos >>>>>>> >>>>>>
