Good point. There is an RFC for Deletion Vectors -> https://github.com/apache/incubator-xtable/blob/main/rfc/rfc-2/2%20-%20Deletion%20Info%20Conversion.md, but it hasn't been implemented yet.
Sounds like it is not something we could replace at this time. Maybe sometime in the future. On Thu, Feb 26, 2026 at 5:38 PM Jingsong Li <[email protected]> wrote: > Hi Mike, > > You are right, but this requires a more detailed evaluation to ensure > that the integration of the two projects will not affect existing > users. > > Also, I am not sure if the ability to delete vectors is currently > implemented in XTable. > > Best, > Jingsong > > On Thu, Feb 26, 2026 at 2:34 PM Mike Dias <[email protected]> wrote: > > > > Thanks Jingsong. I believe it should be possible to hook the Xtable > conversion as a CommitCallback too, ensuring the same real-time generation. > > > > Then, Paimon codebase doesn't need to worry about Iceberg logic, > artifacts, and upgrades anymore, offloading all that to the Xtable codebase. > > > > On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <[email protected]> > wrote: > >> > >> Hi Mike, > >> > >> The commit callback can ensure real-time generation of Iceberg > >> snapshots. XTable is another implementation route, and they are > >> developed in parallel. > >> > >> Best, > >> Jingsong > >> > >> On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user > >> <[email protected]> wrote: > >> > > >> > Btw, the Paimon Support in XTable PR got merged > https://github.com/apache/incubator-xtable/pull/742 > >> > > >> > Given that, would the Paimon community be interested in consolidating > Iceberg support in the XTable codebase rather than using the commit > callback approach? > >> > > >> > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <[email protected]> wrote: > >> >> > >> >> Hi Vinish and Nick, here is the draft PR -> > https://github.com/apache/incubator-xtable/pull/742 > >> >> > >> >> Looking forward to getting your feedback, especially on the Iceberg > issue! :) > >> >> > >> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> > wrote: > >> >>> > >> >>> Hi Vinish, that's a good idea! > >> >>> > >> >>> I'll tidy up my branch and raise a draft pull request instead of an > issue; I think it will be easier to troubleshoot it by seeing the changes > and having a way to reproduce it. > >> >>> > >> >>> Thanks, > >> >>> Mike Dias > >> >>> > >> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> > wrote: > >> >>>> > >> >>>> Hi Mike. > >> >>>> > >> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the > iceberg target issues you are seeing for the XTable Paimon source, can you > create a GH issue? The XTable community can help in root causing the > problem. > >> >>>> https://github.com/apache/incubator-xtable/issues > >> >>>> > >> >>>> -Vinish > >> >>>> > >> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote: > >> >>>> > Hi Nick, thanks for your reply! > >> >>>> > > >> >>>> > Yeah, I believe that creating the XTable source for Paimon is > the best > >> >>>> > approach: fewer hops, fewer dependencies, less maintenance > overall. > >> >>>> > > >> >>>> > I have a prototype for the XTable Paimon source that works for > Delta and > >> >>>> > Hudi targets, but it does not work for the Iceberg target yet > due to issues > >> >>>> > with reading parquet fields with incorrect positions that I > don't yet > >> >>>> > understand why... > >> >>>> > > >> >>>> > Given that you have experience with Paimon and Iceberg, maybe I > could get > >> >>>> > some help troubleshooting that? That would be a huge help! > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano < > [email protected]> wrote: > >> >>>> > > >> >>>> > > Hi Mike, > >> >>>> > > > >> >>>> > > I help maintain Paimon's Iceberg Compatibility. > >> >>>> > > > >> >>>> > > The fastest way to accomplish your goal is the first option. > It should not > >> >>>> > > require code changes to Paimon or XTable. That said the > feature has some > >> >>>> > > tradeoffs. > >> >>>> > > > >> >>>> > > XTable is greatly preferred to adding Delta specific code to > Paimon. I do > >> >>>> > > think it would benefit Paimon adoption to be available in > XTable [1]. > >> >>>> > > > >> >>>> > > Paimon's Iceberg Compatibility works differently than XTable - > it is a > >> >>>> > > synchronous commit callback to translate metadata. This makes > it really > >> >>>> > > easy to enable and it simply supports latest snapshot queries. > It does not > >> >>>> > > yet support adding the full snapshot history (which I am > working on > >> >>>> > > https://github.com/apache/paimon/issues/6107) but still there > will be > >> >>>> > > some practical limitations around very long snapshot histories. > >> >>>> > > > >> >>>> > > Does the current feature set of Paimon-Iceberg compatibility > support your > >> >>>> > > use case? > >> >>>> > > Would you be willing to contribute Paimon support in XTable? > >> >>>> > > > >> >>>> > > Nick > >> >>>> > > > >> >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275 > >> >>>> > > > >> >>>> > > > >> >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: > >> >>>> > > > Hello, we are currently evaluating using Paimon as our > primary format for > >> >>>> > > > landing data via Flink, but we still want to maintain > compatibility with > >> >>>> > > > Delta Lake, as we use Delta Sharing > >> >>>> > > > <https://github.com/delta-io/delta-sharing> features > extensively. > >> >>>> > > > > >> >>>> > > > Right now, it seems that there are three options to achieve > that: > >> >>>> > > > > >> >>>> > > > - Enable the Paimon -> Iceberg > >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> > >> >>>> > > Integration, > >> >>>> > > > and then use XTable <https://xtable.apache.org/> to > convert Iceberg > >> >>>> > > > metadata to Delta Lake metadata. > >> >>>> > > > - Implement a Paimon Source for XTable < > https://xtable.apache.org/> > >> >>>> > > and > >> >>>> > > > use it to convert Delta Lake metadata directly from > Paimon metadata > >> >>>> > > > (read-optimize only). > >> >>>> > > > - Implement a Commit Callback in Paimon that writes the > Delta Lake > >> >>>> > > > metadata directly, similar to what was done on the > Iceberg integration > >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/ > >. > >> >>>> > > > > >> >>>> > > > We are playing with the first option right now, but we would > like to > >> >>>> > > avoid > >> >>>> > > > that going forward because the Iceberg conversion is > unnecessary for us. > >> >>>> > > So > >> >>>> > > > the question is, would the Paimon community accept > contributions to > >> >>>> > > enable > >> >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the > community > >> >>>> > > > prefer to rely on XTable for such cross-table format > conversions? > >> >>>> > > > > >> >>>> > > > Keen to hear your thoughts! > >> >>>> > > > > >> >>>> > > > Thanks > >> >>>> > > > Mike Dias > >> >>>> > > > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > -- > >> >>>> > Thanks, > >> >>>> > Mike Dias > >> >>>> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Thanks, > >> >> Mike Dias > >> > > >> > > >> > > >> > -- > >> > Thanks, > >> > Mike Dias > > > > > > > > -- > > Thanks, > > Mike Dias > -- Thanks, Mike Dias
