Hi Mike, You are right, but this requires a more detailed evaluation to ensure that the integration of the two projects will not affect existing users.
Also, I am not sure if the ability to delete vectors is currently implemented in XTable. Best, Jingsong On Thu, Feb 26, 2026 at 2:34 PM Mike Dias <[email protected]> wrote: > > Thanks Jingsong. I believe it should be possible to hook the Xtable > conversion as a CommitCallback too, ensuring the same real-time generation. > > Then, Paimon codebase doesn't need to worry about Iceberg logic, artifacts, > and upgrades anymore, offloading all that to the Xtable codebase. > > On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <[email protected]> wrote: >> >> Hi Mike, >> >> The commit callback can ensure real-time generation of Iceberg >> snapshots. XTable is another implementation route, and they are >> developed in parallel. >> >> Best, >> Jingsong >> >> On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user >> <[email protected]> wrote: >> > >> > Btw, the Paimon Support in XTable PR got merged >> > https://github.com/apache/incubator-xtable/pull/742 >> > >> > Given that, would the Paimon community be interested in consolidating >> > Iceberg support in the XTable codebase rather than using the commit >> > callback approach? >> > >> > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <[email protected]> wrote: >> >> >> >> Hi Vinish and Nick, here is the draft PR -> >> >> https://github.com/apache/incubator-xtable/pull/742 >> >> >> >> Looking forward to getting your feedback, especially on the Iceberg >> >> issue! :) >> >> >> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> wrote: >> >>> >> >>> Hi Vinish, that's a good idea! >> >>> >> >>> I'll tidy up my branch and raise a draft pull request instead of an >> >>> issue; I think it will be easier to troubleshoot it by seeing the >> >>> changes and having a way to reproduce it. >> >>> >> >>> Thanks, >> >>> Mike Dias >> >>> >> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: >> >>>> >> >>>> Hi Mike. >> >>>> >> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg >> >>>> target issues you are seeing for the XTable Paimon source, can you >> >>>> create a GH issue? The XTable community can help in root causing the >> >>>> problem. >> >>>> https://github.com/apache/incubator-xtable/issues >> >>>> >> >>>> -Vinish >> >>>> >> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote: >> >>>> > Hi Nick, thanks for your reply! >> >>>> > >> >>>> > Yeah, I believe that creating the XTable source for Paimon is the best >> >>>> > approach: fewer hops, fewer dependencies, less maintenance overall. >> >>>> > >> >>>> > I have a prototype for the XTable Paimon source that works for Delta >> >>>> > and >> >>>> > Hudi targets, but it does not work for the Iceberg target yet due to >> >>>> > issues >> >>>> > with reading parquet fields with incorrect positions that I don't yet >> >>>> > understand why... >> >>>> > >> >>>> > Given that you have experience with Paimon and Iceberg, maybe I could >> >>>> > get >> >>>> > some help troubleshooting that? That would be a huge help! >> >>>> > >> >>>> > >> >>>> > >> >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> >> >>>> > wrote: >> >>>> > >> >>>> > > Hi Mike, >> >>>> > > >> >>>> > > I help maintain Paimon's Iceberg Compatibility. >> >>>> > > >> >>>> > > The fastest way to accomplish your goal is the first option. It >> >>>> > > should not >> >>>> > > require code changes to Paimon or XTable. That said the feature has >> >>>> > > some >> >>>> > > tradeoffs. >> >>>> > > >> >>>> > > XTable is greatly preferred to adding Delta specific code to >> >>>> > > Paimon. I do >> >>>> > > think it would benefit Paimon adoption to be available in XTable >> >>>> > > [1]. >> >>>> > > >> >>>> > > Paimon's Iceberg Compatibility works differently than XTable - it >> >>>> > > is a >> >>>> > > synchronous commit callback to translate metadata. This makes it >> >>>> > > really >> >>>> > > easy to enable and it simply supports latest snapshot queries. It >> >>>> > > does not >> >>>> > > yet support adding the full snapshot history (which I am working on >> >>>> > > https://github.com/apache/paimon/issues/6107) but still there will >> >>>> > > be >> >>>> > > some practical limitations around very long snapshot histories. >> >>>> > > >> >>>> > > Does the current feature set of Paimon-Iceberg compatibility >> >>>> > > support your >> >>>> > > use case? >> >>>> > > Would you be willing to contribute Paimon support in XTable? >> >>>> > > >> >>>> > > Nick >> >>>> > > >> >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275 >> >>>> > > >> >>>> > > >> >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: >> >>>> > > > Hello, we are currently evaluating using Paimon as our primary >> >>>> > > > format for >> >>>> > > > landing data via Flink, but we still want to maintain >> >>>> > > > compatibility with >> >>>> > > > Delta Lake, as we use Delta Sharing >> >>>> > > > <https://github.com/delta-io/delta-sharing> features extensively. >> >>>> > > > >> >>>> > > > Right now, it seems that there are three options to achieve that: >> >>>> > > > >> >>>> > > > - Enable the Paimon -> Iceberg >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> >> >>>> > > Integration, >> >>>> > > > and then use XTable <https://xtable.apache.org/> to convert >> >>>> > > > Iceberg >> >>>> > > > metadata to Delta Lake metadata. >> >>>> > > > - Implement a Paimon Source for XTable >> >>>> > > > <https://xtable.apache.org/> >> >>>> > > and >> >>>> > > > use it to convert Delta Lake metadata directly from Paimon >> >>>> > > > metadata >> >>>> > > > (read-optimize only). >> >>>> > > > - Implement a Commit Callback in Paimon that writes the Delta >> >>>> > > > Lake >> >>>> > > > metadata directly, similar to what was done on the Iceberg >> >>>> > > > integration >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. >> >>>> > > > >> >>>> > > > We are playing with the first option right now, but we would like >> >>>> > > > to >> >>>> > > avoid >> >>>> > > > that going forward because the Iceberg conversion is unnecessary >> >>>> > > > for us. >> >>>> > > So >> >>>> > > > the question is, would the Paimon community accept contributions >> >>>> > > > to >> >>>> > > enable >> >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the >> >>>> > > > community >> >>>> > > > prefer to rely on XTable for such cross-table format conversions? >> >>>> > > > >> >>>> > > > Keen to hear your thoughts! >> >>>> > > > >> >>>> > > > Thanks >> >>>> > > > Mike Dias >> >>>> > > > >> >>>> > >> >>>> > >> >>>> > >> >>>> > -- >> >>>> > Thanks, >> >>>> > Mike Dias >> >>>> > >> >> >> >> >> >> >> >> -- >> >> Thanks, >> >> Mike Dias >> > >> > >> > >> > -- >> > Thanks, >> > Mike Dias > > > > -- > Thanks, > Mike Dias
