Hi Mike, The commit callback can ensure real-time generation of Iceberg snapshots. XTable is another implementation route, and they are developed in parallel.
Best, Jingsong On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user <[email protected]> wrote: > > Btw, the Paimon Support in XTable PR got merged > https://github.com/apache/incubator-xtable/pull/742 > > Given that, would the Paimon community be interested in consolidating Iceberg > support in the XTable codebase rather than using the commit callback approach? > > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <[email protected]> wrote: >> >> Hi Vinish and Nick, here is the draft PR -> >> https://github.com/apache/incubator-xtable/pull/742 >> >> Looking forward to getting your feedback, especially on the Iceberg issue! :) >> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> wrote: >>> >>> Hi Vinish, that's a good idea! >>> >>> I'll tidy up my branch and raise a draft pull request instead of an issue; >>> I think it will be easier to troubleshoot it by seeing the changes and >>> having a way to reproduce it. >>> >>> Thanks, >>> Mike Dias >>> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: >>>> >>>> Hi Mike. >>>> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg >>>> target issues you are seeing for the XTable Paimon source, can you create >>>> a GH issue? The XTable community can help in root causing the problem. >>>> https://github.com/apache/incubator-xtable/issues >>>> >>>> -Vinish >>>> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote: >>>> > Hi Nick, thanks for your reply! >>>> > >>>> > Yeah, I believe that creating the XTable source for Paimon is the best >>>> > approach: fewer hops, fewer dependencies, less maintenance overall. >>>> > >>>> > I have a prototype for the XTable Paimon source that works for Delta and >>>> > Hudi targets, but it does not work for the Iceberg target yet due to >>>> > issues >>>> > with reading parquet fields with incorrect positions that I don't yet >>>> > understand why... >>>> > >>>> > Given that you have experience with Paimon and Iceberg, maybe I could get >>>> > some help troubleshooting that? That would be a huge help! >>>> > >>>> > >>>> > >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> >>>> > wrote: >>>> > >>>> > > Hi Mike, >>>> > > >>>> > > I help maintain Paimon's Iceberg Compatibility. >>>> > > >>>> > > The fastest way to accomplish your goal is the first option. It should >>>> > > not >>>> > > require code changes to Paimon or XTable. That said the feature has >>>> > > some >>>> > > tradeoffs. >>>> > > >>>> > > XTable is greatly preferred to adding Delta specific code to Paimon. I >>>> > > do >>>> > > think it would benefit Paimon adoption to be available in XTable [1]. >>>> > > >>>> > > Paimon's Iceberg Compatibility works differently than XTable - it is a >>>> > > synchronous commit callback to translate metadata. This makes it really >>>> > > easy to enable and it simply supports latest snapshot queries. It does >>>> > > not >>>> > > yet support adding the full snapshot history (which I am working on >>>> > > https://github.com/apache/paimon/issues/6107) but still there will be >>>> > > some practical limitations around very long snapshot histories. >>>> > > >>>> > > Does the current feature set of Paimon-Iceberg compatibility support >>>> > > your >>>> > > use case? >>>> > > Would you be willing to contribute Paimon support in XTable? >>>> > > >>>> > > Nick >>>> > > >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275 >>>> > > >>>> > > >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: >>>> > > > Hello, we are currently evaluating using Paimon as our primary >>>> > > > format for >>>> > > > landing data via Flink, but we still want to maintain compatibility >>>> > > > with >>>> > > > Delta Lake, as we use Delta Sharing >>>> > > > <https://github.com/delta-io/delta-sharing> features extensively. >>>> > > > >>>> > > > Right now, it seems that there are three options to achieve that: >>>> > > > >>>> > > > - Enable the Paimon -> Iceberg >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> >>>> > > Integration, >>>> > > > and then use XTable <https://xtable.apache.org/> to convert >>>> > > > Iceberg >>>> > > > metadata to Delta Lake metadata. >>>> > > > - Implement a Paimon Source for XTable >>>> > > > <https://xtable.apache.org/> >>>> > > and >>>> > > > use it to convert Delta Lake metadata directly from Paimon >>>> > > > metadata >>>> > > > (read-optimize only). >>>> > > > - Implement a Commit Callback in Paimon that writes the Delta Lake >>>> > > > metadata directly, similar to what was done on the Iceberg >>>> > > > integration >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. >>>> > > > >>>> > > > We are playing with the first option right now, but we would like to >>>> > > avoid >>>> > > > that going forward because the Iceberg conversion is unnecessary for >>>> > > > us. >>>> > > So >>>> > > > the question is, would the Paimon community accept contributions to >>>> > > enable >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the community >>>> > > > prefer to rely on XTable for such cross-table format conversions? >>>> > > > >>>> > > > Keen to hear your thoughts! >>>> > > > >>>> > > > Thanks >>>> > > > Mike Dias >>>> > > > >>>> > >>>> > >>>> > >>>> > -- >>>> > Thanks, >>>> > Mike Dias >>>> > >> >> >> >> -- >> Thanks, >> Mike Dias > > > > -- > Thanks, > Mike Dias
