Thanks Jingsong. I believe it should be possible to hook the Xtable conversion as a CommitCallback too, ensuring the same real-time generation.
Then, Paimon codebase doesn't need to worry about Iceberg logic, artifacts, and upgrades anymore, offloading all that to the Xtable codebase. On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <[email protected]> wrote: > Hi Mike, > > The commit callback can ensure real-time generation of Iceberg > snapshots. XTable is another implementation route, and they are > developed in parallel. > > Best, > Jingsong > > On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user > <[email protected]> wrote: > > > > Btw, the Paimon Support in XTable PR got merged > https://github.com/apache/incubator-xtable/pull/742 > > > > Given that, would the Paimon community be interested in consolidating > Iceberg support in the XTable codebase rather than using the commit > callback approach? > > > > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <[email protected]> wrote: > >> > >> Hi Vinish and Nick, here is the draft PR -> > https://github.com/apache/incubator-xtable/pull/742 > >> > >> Looking forward to getting your feedback, especially on the Iceberg > issue! :) > >> > >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> wrote: > >>> > >>> Hi Vinish, that's a good idea! > >>> > >>> I'll tidy up my branch and raise a draft pull request instead of an > issue; I think it will be easier to troubleshoot it by seeing the changes > and having a way to reproduce it. > >>> > >>> Thanks, > >>> Mike Dias > >>> > >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: > >>>> > >>>> Hi Mike. > >>>> > >>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg > target issues you are seeing for the XTable Paimon source, can you create a > GH issue? The XTable community can help in root causing the problem. > >>>> https://github.com/apache/incubator-xtable/issues > >>>> > >>>> -Vinish > >>>> > >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote: > >>>> > Hi Nick, thanks for your reply! > >>>> > > >>>> > Yeah, I believe that creating the XTable source for Paimon is the > best > >>>> > approach: fewer hops, fewer dependencies, less maintenance overall. > >>>> > > >>>> > I have a prototype for the XTable Paimon source that works for > Delta and > >>>> > Hudi targets, but it does not work for the Iceberg target yet due > to issues > >>>> > with reading parquet fields with incorrect positions that I don't > yet > >>>> > understand why... > >>>> > > >>>> > Given that you have experience with Paimon and Iceberg, maybe I > could get > >>>> > some help troubleshooting that? That would be a huge help! > >>>> > > >>>> > > >>>> > > >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano < > [email protected]> wrote: > >>>> > > >>>> > > Hi Mike, > >>>> > > > >>>> > > I help maintain Paimon's Iceberg Compatibility. > >>>> > > > >>>> > > The fastest way to accomplish your goal is the first option. It > should not > >>>> > > require code changes to Paimon or XTable. That said the feature > has some > >>>> > > tradeoffs. > >>>> > > > >>>> > > XTable is greatly preferred to adding Delta specific code to > Paimon. I do > >>>> > > think it would benefit Paimon adoption to be available in XTable > [1]. > >>>> > > > >>>> > > Paimon's Iceberg Compatibility works differently than XTable - it > is a > >>>> > > synchronous commit callback to translate metadata. This makes it > really > >>>> > > easy to enable and it simply supports latest snapshot queries. It > does not > >>>> > > yet support adding the full snapshot history (which I am working > on > >>>> > > https://github.com/apache/paimon/issues/6107) but still there > will be > >>>> > > some practical limitations around very long snapshot histories. > >>>> > > > >>>> > > Does the current feature set of Paimon-Iceberg compatibility > support your > >>>> > > use case? > >>>> > > Would you be willing to contribute Paimon support in XTable? > >>>> > > > >>>> > > Nick > >>>> > > > >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275 > >>>> > > > >>>> > > > >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: > >>>> > > > Hello, we are currently evaluating using Paimon as our primary > format for > >>>> > > > landing data via Flink, but we still want to maintain > compatibility with > >>>> > > > Delta Lake, as we use Delta Sharing > >>>> > > > <https://github.com/delta-io/delta-sharing> features > extensively. > >>>> > > > > >>>> > > > Right now, it seems that there are three options to achieve > that: > >>>> > > > > >>>> > > > - Enable the Paimon -> Iceberg > >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> > >>>> > > Integration, > >>>> > > > and then use XTable <https://xtable.apache.org/> to convert > Iceberg > >>>> > > > metadata to Delta Lake metadata. > >>>> > > > - Implement a Paimon Source for XTable < > https://xtable.apache.org/> > >>>> > > and > >>>> > > > use it to convert Delta Lake metadata directly from Paimon > metadata > >>>> > > > (read-optimize only). > >>>> > > > - Implement a Commit Callback in Paimon that writes the > Delta Lake > >>>> > > > metadata directly, similar to what was done on the Iceberg > integration > >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. > >>>> > > > > >>>> > > > We are playing with the first option right now, but we would > like to > >>>> > > avoid > >>>> > > > that going forward because the Iceberg conversion is > unnecessary for us. > >>>> > > So > >>>> > > > the question is, would the Paimon community accept > contributions to > >>>> > > enable > >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the > community > >>>> > > > prefer to rely on XTable for such cross-table format > conversions? > >>>> > > > > >>>> > > > Keen to hear your thoughts! > >>>> > > > > >>>> > > > Thanks > >>>> > > > Mike Dias > >>>> > > > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > Thanks, > >>>> > Mike Dias > >>>> > > >> > >> > >> > >> -- > >> Thanks, > >> Mike Dias > > > > > > > > -- > > Thanks, > > Mike Dias > -- Thanks, Mike Dias
