Hi Vinish, that's a good idea! I'll tidy up my branch and raise a draft pull request instead of an issue; I think it will be easier to troubleshoot it by seeing the changes and having a way to reproduce it.
Thanks, Mike Dias On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: > Hi Mike. > > I'm Vinish one of the PPMC members for XTable. Regarding the iceberg > target issues you are seeing for the XTable Paimon source, can you create a > GH issue? The XTable community can help in root causing the problem. > https://github.com/apache/incubator-xtable/issues > > -Vinish > > On 2025/09/01 05:37:39 Mike Dias via user wrote: > > Hi Nick, thanks for your reply! > > > > Yeah, I believe that creating the XTable source for Paimon is the best > > approach: fewer hops, fewer dependencies, less maintenance overall. > > > > I have a prototype for the XTable Paimon source that works for Delta and > > Hudi targets, but it does not work for the Iceberg target yet due to > issues > > with reading parquet fields with incorrect positions that I don't yet > > understand why... > > > > Given that you have experience with Paimon and Iceberg, maybe I could get > > some help troubleshooting that? That would be a huge help! > > > > > > > > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> > wrote: > > > > > Hi Mike, > > > > > > I help maintain Paimon's Iceberg Compatibility. > > > > > > The fastest way to accomplish your goal is the first option. It should > not > > > require code changes to Paimon or XTable. That said the feature has > some > > > tradeoffs. > > > > > > XTable is greatly preferred to adding Delta specific code to Paimon. I > do > > > think it would benefit Paimon adoption to be available in XTable [1]. > > > > > > Paimon's Iceberg Compatibility works differently than XTable - it is a > > > synchronous commit callback to translate metadata. This makes it really > > > easy to enable and it simply supports latest snapshot queries. It does > not > > > yet support adding the full snapshot history (which I am working on > > > https://github.com/apache/paimon/issues/6107) but still there will be > > > some practical limitations around very long snapshot histories. > > > > > > Does the current feature set of Paimon-Iceberg compatibility support > your > > > use case? > > > Would you be willing to contribute Paimon support in XTable? > > > > > > Nick > > > > > > [1] https://github.com/apache/incubator-xtable/issues/275 > > > > > > > > > On 2025/08/08 05:44:03 Mike Dias via user wrote: > > > > Hello, we are currently evaluating using Paimon as our primary > format for > > > > landing data via Flink, but we still want to maintain compatibility > with > > > > Delta Lake, as we use Delta Sharing > > > > <https://github.com/delta-io/delta-sharing> features extensively. > > > > > > > > Right now, it seems that there are three options to achieve that: > > > > > > > > - Enable the Paimon -> Iceberg > > > > <https://paimon.apache.org/docs/master/iceberg/overview/> > > > Integration, > > > > and then use XTable <https://xtable.apache.org/> to convert > Iceberg > > > > metadata to Delta Lake metadata. > > > > - Implement a Paimon Source for XTable < > https://xtable.apache.org/> > > > and > > > > use it to convert Delta Lake metadata directly from Paimon > metadata > > > > (read-optimize only). > > > > - Implement a Commit Callback in Paimon that writes the Delta Lake > > > > metadata directly, similar to what was done on the Iceberg > integration > > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. > > > > > > > > We are playing with the first option right now, but we would like to > > > avoid > > > > that going forward because the Iceberg conversion is unnecessary for > us. > > > So > > > > the question is, would the Paimon community accept contributions to > > > enable > > > > Delta Lake compatibility, just like Iceberg's, or would the community > > > > prefer to rely on XTable for such cross-table format conversions? > > > > > > > > Keen to hear your thoughts! > > > > > > > > Thanks > > > > Mike Dias > > > > > > > > > > > > -- > > Thanks, > > Mike Dias > > >
