Btw, the Paimon Support in XTable PR got merged https://github.com/apache/incubator-xtable/pull/742
Given that, would the Paimon community be interested in consolidating Iceberg support in the XTable codebase rather than using the commit callback approach? On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <[email protected]> wrote: > Hi Vinish and Nick, here is the draft PR -> > https://github.com/apache/incubator-xtable/pull/742 > > Looking forward to getting your feedback, especially on the Iceberg issue! > :) > > On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> wrote: > >> Hi Vinish, that's a good idea! >> >> I'll tidy up my branch and raise a draft pull request instead of an >> issue; I think it will be easier to troubleshoot it by seeing the changes >> and having a way to reproduce it. >> >> Thanks, >> Mike Dias >> >> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: >> >>> Hi Mike. >>> >>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg >>> target issues you are seeing for the XTable Paimon source, can you create a >>> GH issue? The XTable community can help in root causing the problem. >>> https://github.com/apache/incubator-xtable/issues >>> >>> -Vinish >>> >>> On 2025/09/01 05:37:39 Mike Dias via user wrote: >>> > Hi Nick, thanks for your reply! >>> > >>> > Yeah, I believe that creating the XTable source for Paimon is the best >>> > approach: fewer hops, fewer dependencies, less maintenance overall. >>> > >>> > I have a prototype for the XTable Paimon source that works for Delta >>> and >>> > Hudi targets, but it does not work for the Iceberg target yet due to >>> issues >>> > with reading parquet fields with incorrect positions that I don't yet >>> > understand why... >>> > >>> > Given that you have experience with Paimon and Iceberg, maybe I could >>> get >>> > some help troubleshooting that? That would be a huge help! >>> > >>> > >>> > >>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> >>> wrote: >>> > >>> > > Hi Mike, >>> > > >>> > > I help maintain Paimon's Iceberg Compatibility. >>> > > >>> > > The fastest way to accomplish your goal is the first option. It >>> should not >>> > > require code changes to Paimon or XTable. That said the feature has >>> some >>> > > tradeoffs. >>> > > >>> > > XTable is greatly preferred to adding Delta specific code to Paimon. >>> I do >>> > > think it would benefit Paimon adoption to be available in XTable [1]. >>> > > >>> > > Paimon's Iceberg Compatibility works differently than XTable - it is >>> a >>> > > synchronous commit callback to translate metadata. This makes it >>> really >>> > > easy to enable and it simply supports latest snapshot queries. It >>> does not >>> > > yet support adding the full snapshot history (which I am working on >>> > > https://github.com/apache/paimon/issues/6107) but still there will >>> be >>> > > some practical limitations around very long snapshot histories. >>> > > >>> > > Does the current feature set of Paimon-Iceberg compatibility support >>> your >>> > > use case? >>> > > Would you be willing to contribute Paimon support in XTable? >>> > > >>> > > Nick >>> > > >>> > > [1] https://github.com/apache/incubator-xtable/issues/275 >>> > > >>> > > >>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: >>> > > > Hello, we are currently evaluating using Paimon as our primary >>> format for >>> > > > landing data via Flink, but we still want to maintain >>> compatibility with >>> > > > Delta Lake, as we use Delta Sharing >>> > > > <https://github.com/delta-io/delta-sharing> features extensively. >>> > > > >>> > > > Right now, it seems that there are three options to achieve that: >>> > > > >>> > > > - Enable the Paimon -> Iceberg >>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> >>> > > Integration, >>> > > > and then use XTable <https://xtable.apache.org/> to convert >>> Iceberg >>> > > > metadata to Delta Lake metadata. >>> > > > - Implement a Paimon Source for XTable < >>> https://xtable.apache.org/> >>> > > and >>> > > > use it to convert Delta Lake metadata directly from Paimon >>> metadata >>> > > > (read-optimize only). >>> > > > - Implement a Commit Callback in Paimon that writes the Delta >>> Lake >>> > > > metadata directly, similar to what was done on the Iceberg >>> integration >>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. >>> > > > >>> > > > We are playing with the first option right now, but we would like >>> to >>> > > avoid >>> > > > that going forward because the Iceberg conversion is unnecessary >>> for us. >>> > > So >>> > > > the question is, would the Paimon community accept contributions to >>> > > enable >>> > > > Delta Lake compatibility, just like Iceberg's, or would the >>> community >>> > > > prefer to rely on XTable for such cross-table format conversions? >>> > > > >>> > > > Keen to hear your thoughts! >>> > > > >>> > > > Thanks >>> > > > Mike Dias >>> > > > >>> > >>> > >>> > >>> > -- >>> > Thanks, >>> > Mike Dias >>> > >>> >> > > -- > Thanks, > Mike Dias > -- Thanks, Mike Dias
