Here's the vote thread on the iceberg dev list: https://lists.apache.org/thread/ytb9x7x8ff3b7031kcrc05nzjtmh091s
On Wed, Sep 9, 2026 at 11:21 AM Andrew Lamb <[email protected]> wrote: > I have started a vote on the datafusion dev list to accept the code: > https://lists.apache.org/thread/6k5lws31c61y1lx2s34pxo4lmc7334lr > > On Wed, Sep 9, 2026 at 12:45 PM Andrew Lamb <[email protected]> wrote: > > > Thanks Kevin -- that would be great. > > > > If you start the iceberg vote thread, then someone else can start a vote > > thread for DataFusion to accept it. > > > > Andrew > > > > On Wed, Sep 9, 2026 at 12:02 PM Kevin Liu <[email protected]> wrote: > > > >> Thanks Andrew. And thank you Gabriel for porting over the code, I've > +1'ed > >> the PR (https://github.com/apache/datafusion-iceberg/pull/1) > >> > >> Happy to help start a vote on the iceberg side. > >> > >> Best, > >> Kevin Liu > >> > >> On Wed, Sep 9, 2026 at 4:36 AM Andrew Lamb <[email protected]> > >> wrote: > >> > >> > The new repository is available: > >> > https://github.com/apache/datafusion-iceberg > >> > > >> > Gabriel, can you take a look at making a proposed PR with the content > to > >> > move? > >> > > >> > On Wed, Sep 9, 2026 at 7:07 AM Andrew Lamb <[email protected]> > >> wrote: > >> > > >> >> I have filed a INFR request to create the new repository: > >> >> https://issues.apache.org/jira/browse/INFRA-28379 > >> >> > >> >> On Tue, Sep 8, 2026 at 8:02 PM Andrew Lamb <[email protected]> > >> >> wrote: > >> >> > >> >> > I will plan to run both voting threads -- I think first we need a > >> repo > >> >> and > >> >> > a PR to vote on. We will work on that starting tomorrow > >> >> > > >> >> > On Tue, Sep 8, 2026 at 7:52 PM Neelesh Salian < > >> [email protected] > >> >> > > >> >> > wrote: > >> >> > > >> >> > > Sounds good to me. Feel free to start both threads. Glad to see > >> this > >> >> > going > >> >> > > forward. > >> >> > > > >> >> > > On Tue, Sep 8, 2026 at 16:46 Andrew Lamb <[email protected] > > > >> >> wrote: > >> >> > > > >> >> > > > I think we should do a formal vote in a separate thread -- for > >> both > >> >> > > > communities > >> >> > > > > >> >> > > > On Tue, Sep 8, 2026 at 7:11 PM Neelesh Salian < > >> >> > [email protected]> > >> >> > > > wrote: > >> >> > > > > >> >> > > >> Yes, to the new repo apache/datafusion-iceberg > >> >> > > >> > >> >> > > >> Question to the folks here: Do we need a formal community vote > >> >> across > >> >> > > >> communities or this thread suffices? > >> >> > > >> > >> >> > > >> On Tue, Sep 8, 2026 at 03:37 Andrew Lamb < > [email protected]> > >> >> > wrote: > >> >> > > >> > >> >> > > >> > Thank for all the feedback so far -- unless anyone has > >> concerns, > >> >> I > >> >> > > will > >> >> > > >> > file a request a new repository for > apache/datafusion-iceberg > >> >> > tomorrow > >> >> > > >> > > >> >> > > >> > Andrew > >> >> > > >> > > >> >> > > >> > On Tue, Sep 8, 2026 at 5:49 AM Renjie Liu < > >> >> [email protected]> > >> >> > > >> wrote: > >> >> > > >> > > >> >> > > >> > > +1 for Andrew's proposal. > >> >> > > >> > > > >> >> > > >> > > On Sat, Sep 5, 2026 at 1:56 AM Kevin Liu < > >> >> [email protected]> > >> >> > > >> wrote: > >> >> > > >> > > > >> >> > > >> > > > + dev@iceberg > >> >> > > >> > > > > >> >> > > >> > > > (lesson learned, emailing 2 devlist might not be the > best > >> >> idea > >> >> > > >> > > logistically > >> >> > > >> > > > haha) > >> >> > > >> > > > > >> >> > > >> > > > On Fri, Sep 4, 2026 at 10:48 AM Kevin Liu < > >> >> > [email protected]> > >> >> > > >> > wrote: > >> >> > > >> > > > > >> >> > > >> > > > > Thanks for all the great discussions so far. I'm glad > we > >> >> > found a > >> >> > > >> > > solution > >> >> > > >> > > > > that benefits the broader ecosystem. > >> >> > > >> > > > > I'm +1 to moving to an apache-governed location, > >> >> > > >> > > > > apache/datafusion-iceberg seems like a great place. I > >> like > >> >> the > >> >> > > >> > process > >> >> > > >> > > > > Andrew proposed, happy to help with the logistics. > >> >> > > >> > > > > > >> >> > > >> > > > > Best, > >> >> > > >> > > > > Kevin Liu > >> >> > > >> > > > > > >> >> > > >> > > > > PS I have also removed the iceberg python binding that > >> >> exports > >> >> > > >> > > > > DataFusion's TableProvider, so that's 1 less thing we > >> have > >> >> to > >> >> > > >> worry > >> >> > > >> > > > about. > >> >> > > >> > > > > https://github.com/apache/iceberg-rust/issues/3036 > >> >> > > >> > > > > > >> >> > > >> > > > > On Fri, Sep 4, 2026 at 6:28 AM Gabriel Musat < > >> >> > > >> [email protected]> > >> >> > > >> > > > wrote: > >> >> > > >> > > > > > >> >> > > >> > > > >> Sounds like a good direction, +1 (non binding) > >> >> > > >> > > > >> > >> >> > > >> > > > >> I can help porting the commit history of the > >> >> > DataFusion-Iceberg > >> >> > > >> > > > >> integration to the new repo if PMCs agree. > >> >> > > >> > > > >> > >> >> > > >> > > > >> On 2026/09/04 12:27:30 Andrew Lamb wrote: > >> >> > > >> > > > >> > I think this is a great idea as well -- thank you > for > >> >> > > bringing > >> >> > > >> it > >> >> > > >> > up > >> >> > > >> > > > and > >> >> > > >> > > > >> > for the great discussions so far. > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > While at VLDB this past week, I spoke to at least > >> three > >> >> > > people > >> >> > > >> > from > >> >> > > >> > > > >> > companies adding Apache Iceberg support to their > >> >> products. > >> >> > > All > >> >> > > >> of > >> >> > > >> > > them > >> >> > > >> > > > >> had > >> >> > > >> > > > >> > to fork iceberg-rust for one reason or another. > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > I think we have a huge need to improve our ability > to > >> >> work > >> >> > > >> > together > >> >> > > >> > > > and > >> >> > > >> > > > >> > accelerate everyone's efforts. Giving the > DataFusion > >> >> > > >> integration > >> >> > > >> > > > access > >> >> > > >> > > > >> to > >> >> > > >> > > > >> > more expert maintainers with bandwidth I think will > >> help > >> >> > > >> everyone. > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > There seems to be consensus that we should move the > >> >> > > integration > >> >> > > >> > code > >> >> > > >> > > > to > >> >> > > >> > > > >> > DataFusion governance, and that it should remain in > >> the > >> >> > ASF, > >> >> > > >> > though > >> >> > > >> > > > some > >> >> > > >> > > > >> > open technical questions remain. > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > If that is the case, I propose the following > specific > >> >> > > process: > >> >> > > >> > > > >> > 1. Create the new gitub repository in Apache for > the > >> >> code > >> >> > > (e.g. > >> >> > > >> > > > >> > apache/datafusion-iceberg) > >> >> > > >> > > > >> > 2. Create a PR in the new repo with the proposed > code > >> >> > > >> > > > >> > 3. Hold a formal vote on the iceberg dev list to > move > >> >> the > >> >> > > >> > > integration > >> >> > > >> > > > >> code > >> >> > > >> > > > >> > to DataFusion > >> >> > > >> > > > >> > 4. Hold a formal vote on the DataFusion dev list to > >> >> accept > >> >> > > the > >> >> > > >> new > >> >> > > >> > > > code > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > I am happy to help with the logistics (e.g. ASF > INFRA > >> >> > ticket > >> >> > > to > >> >> > > >> > > create > >> >> > > >> > > > >> the > >> >> > > >> > > > >> > new repo, votes, etc) but I am not expert enough to > >> >> create > >> >> > > the > >> >> > > >> > > > proposed > >> >> > > >> > > > >> PR. > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > Please let me know your thoughts, > >> >> > > >> > > > >> > Andrew > >> >> > > >> > > > >> > (PMC Chair of DataFusion) > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > On Tue, Sep 1, 2026 at 9:48 PM Renjie Liu < > >> >> > > >> > [email protected]> > >> >> > > >> > > > >> wrote: > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > > To add more background about the relationship > >> between > >> >> > comet > >> >> > > >> and > >> >> > > >> > > > >> datafusion > >> >> > > >> > > > >> > > for those who are not familiar with them. > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > Apache datafusion is a popular extensible compute > >> >> engine > >> >> > > >> written > >> >> > > >> > > in > >> >> > > >> > > > >> rust. > >> >> > > >> > > > >> > > Apache comet is an apache spark accelerator > >> builton on > >> >> > > apache > >> >> > > >> > > > >> datafusion, > >> >> > > >> > > > >> > > and also a subproject of apache datafusion. > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > datafusion-iceberg is an apache datafusion > >> extension > >> >> > built > >> >> > > on > >> >> > > >> > > > >> iceberg-rust, > >> >> > > >> > > > >> > > and comet's iceberg support is built on it. > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > On Tue, Sep 1, 2026 at 9:39 PM Andy Grove < > >> >> > > >> > [email protected]> > >> >> > > >> > > > >> wrote: > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > > +1 for moving to DataFusion PMC. The > >> >> iceberg-datafusion > >> >> > > >> > > > integration > >> >> > > >> > > > >> is > >> >> > > >> > > > >> > > > very important for Comet. > >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > On Mon, Aug 31, 2026 at 11:42 PM Xuanwo < > >> >> > > [email protected] > >> >> > > >> > > >> >> > > >> > > > wrote: > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > TBH, I also support moving to the DataFusion > >> PMC. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > - DataFusion is the largest dependency in > >> >> > > >> > iceberg-datafusion. > >> >> > > >> > > > >> > > > > - The largest downstream user of > >> >> iceberg-datafusion > >> >> > is > >> >> > > >> > Comet, > >> >> > > >> > > > >> which > >> >> > > >> > > > >> > > > shares many of the same PMC members as > >> DataFusion. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > It feels natural to be part of the DataFusion > >> >> PMC. As > >> >> > > >> long > >> >> > > >> > as > >> >> > > >> > > DF > >> >> > > >> > > > >> PMC is > >> >> > > >> > > > >> > > > willing to accept this project, it LGTM. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > On Tue, Sep 1, 2026, at 12:03, Renjie Liu > >> wrote: > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > I don't think the testing should be a blocker > >> of > >> >> > moving > >> >> > > >> > > > >> > > > iceberg-datafusion out of iceberg-rust repo. > From > >> >> what > >> >> > I > >> >> > > >> > learn, > >> >> > > >> > > > >> most of > >> >> > > >> > > > >> > > the > >> >> > > >> > > > >> > > > sqllogictests are in pr of modifying > >> >> iceberg-datafusion > >> >> > > >> > > > integration, > >> >> > > >> > > > >> > > there > >> >> > > >> > > > >> > > > are only few cases where we rely on > >> sqllogictests to > >> >> > > verify > >> >> > > >> > > > >> features. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > How would that sound for the `iceberg-rust` > >> >> > community > >> >> > > >> to > >> >> > > >> > > > grant a > >> >> > > >> > > > >> > > > couple of Apache DataFusion PMCs committer > >> access to > >> >> > the > >> >> > > >> > > > repository, > >> >> > > >> > > > >> > > > limited to the `iceberg-datafusion` crate via a > >> >> > > CODEOWNERS > >> >> > > >> > file, > >> >> > > >> > > > so > >> >> > > >> > > > >> that > >> >> > > >> > > > >> > > > they can help push reviews and PRs forward > >> >> > independently? > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > I'm not sure if this is feasible, but moving > >> the > >> >> > > >> > > > >> iceberg-datafusion > >> >> > > >> > > > >> > > > crate to apache datafusion project sounds a > more > >> >> > > reasonable > >> >> > > >> > > > >> approach to > >> >> > > >> > > > >> > > me. > >> >> > > >> > > > >> > > > It's still governed by Apache, and most of the > >> code > >> >> is > >> >> > > >> related > >> >> > > >> > > to > >> >> > > >> > > > >> > > > DataFusion, so I think the DataFusion community > >> is > >> >> in a > >> >> > > >> better > >> >> > > >> > > > >> position > >> >> > > >> > > > >> > > to > >> >> > > >> > > > >> > > > define the vision and design for it. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > On Mon, Aug 31, 2026 at 10:45 PM Gabriel > Musat > >> < > >> >> > > >> > > > >> [email protected]> > >> >> > > >> > > > >> > > > wrote: > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > Hi everyone, > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > Based on these two facts: > >> >> > > >> > > > >> > > > > - There's a current reviewer bandwidth > problem > >> >> that > >> >> > > hurts > >> >> > > >> > > > >> development > >> >> > > >> > > > >> > > > velocity of the `iceberg-datafusion` crate > >> currently > >> >> > > hosted > >> >> > > >> > > under > >> >> > > >> > > > >> > > > `apache/iceberg-rust`. > >> >> > > >> > > > >> > > > > - There is value in maintaining the > >> >> > > `iceberg-datafusion` > >> >> > > >> > crate > >> >> > > >> > > > >> inside > >> >> > > >> > > > >> > > > `apache/iceberg-rust` for testing, design and > >> >> > governance. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > How would that sound for the `iceberg-rust` > >> >> community > >> >> > > to > >> >> > > >> > > grant a > >> >> > > >> > > > >> couple > >> >> > > >> > > > >> > > > of Apache DataFusion PMCs committer access to > the > >> >> > > >> repository, > >> >> > > >> > > > >> limited to > >> >> > > >> > > > >> > > > the `iceberg-datafusion` crate via a CODEOWNERS > >> >> file, > >> >> > so > >> >> > > >> that > >> >> > > >> > > they > >> >> > > >> > > > >> can > >> >> > > >> > > > >> > > help > >> >> > > >> > > > >> > > > push reviews and PRs forward independently? > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > Based on review history, I'd propose Matt > >> >> Butrovich > >> >> > and > >> >> > > >> Tim > >> >> > > >> > > > >> Saucer as > >> >> > > >> > > > >> > > > two good candidates, but this would be > completely > >> >> up to > >> >> > > the > >> >> > > >> > > > >> > > `iceberg-rust` > >> >> > > >> > > > >> > > > community. > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > On 2026/08/26 17:56:19 Shawn Chang wrote: > >> >> > > >> > > > >> > > > > > Hi all, > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Summarizing the discussion so far, > including > >> a > >> >> few > >> >> > > >> points > >> >> > > >> > > > >> raised in > >> >> > > >> > > > >> > > > today’s > >> >> > > >> > > > >> > > > > > community sync. > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > There seems to be general agreement that > the > >> >> > current > >> >> > > >> > > > DataFusion > >> >> > > >> > > > >> > > > integration > >> >> > > >> > > > >> > > > > > has a velocity/reviewer bandwidth problem, > >> and > >> >> > moving > >> >> > > >> it > >> >> > > >> > to > >> >> > > >> > > a > >> >> > > >> > > > >> > > separate > >> >> > > >> > > > >> > > > > > repository could help DataFusion > contributors > >> >> > iterate > >> >> > > >> more > >> >> > > >> > > > >> > > > independently. > >> >> > > >> > > > >> > > > > > At the same time, several open questions > >> remain: > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > - > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Whether repo separation is the right > >> >> solution, > >> >> > > >> versus > >> >> > > >> > > > >> expanding > >> >> > > >> > > > >> > > > > > DataFusion reviewer/committer > >> participation > >> >> in > >> >> > > >> > > > iceberg-rust. > >> >> > > >> > > > >> > > > > > - > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Where the boundary should be between > >> engine > >> >> > > specific > >> >> > > >> > > > >> integration > >> >> > > >> > > > >> > > and > >> >> > > >> > > > >> > > > > > Iceberg core functionality, and how to > >> avoid > >> >> > > >> duplicated > >> >> > > >> > > or > >> >> > > >> > > > >> forked > >> >> > > >> > > > >> > > > Iceberg > >> >> > > >> > > > >> > > > > > implementations. (how to avoid the case > >> where > >> >> > > >> > > > >> Iceberg-datafusion > >> >> > > >> > > > >> > > > moving > >> >> > > >> > > > >> > > > > > much faster than the core and eventually > >> >> need a > >> >> > > >> forked > >> >> > > >> > > core > >> >> > > >> > > > >> API > >> >> > > >> > > > >> > > > > > implementation) > >> >> > > >> > > > >> > > > > > - > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > How compatibility and end-to-end > >> correctness > >> >> > > testing > >> >> > > >> > > should > >> >> > > >> > > > >> work > >> >> > > >> > > > >> > > > across > >> >> > > >> > > > >> > > > > > repositories, since iceberg-rust still > >> >> relies on > >> >> > > >> > > DataFusion > >> >> > > >> > > > >> for > >> >> > > >> > > > >> > > > integration > >> >> > > >> > > > >> > > > > > testing. > >> >> > > >> > > > >> > > > > > - > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Where the integration should live and > how > >> to > >> >> > keep > >> >> > > it > >> >> > > >> > > under > >> >> > > >> > > > >> Apache > >> >> > > >> > > > >> > > > > > governance while making it easy for both > >> >> Iceberg > >> >> > > and > >> >> > > >> > > > >> DataFusion > >> >> > > >> > > > >> > > > > > contributors to maintain. > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > So I think the main question is not only > >> >> whether to > >> >> > > >> move > >> >> > > >> > the > >> >> > > >> > > > >> code, > >> >> > > >> > > > >> > > but > >> >> > > >> > > > >> > > > how > >> >> > > >> > > > >> > > > > > to improve development velocity without > >> losing > >> >> the > >> >> > > >> close > >> >> > > >> > > > design, > >> >> > > >> > > > >> > > > testing, > >> >> > > >> > > > >> > > > > > and governance relationship between the > >> engine > >> >> > > >> integration > >> >> > > >> > > and > >> >> > > >> > > > >> > > > iceberg-rust > >> >> > > >> > > > >> > > > > > core. > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Best, > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > Shawn > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > On Mon, Aug 24, 2026 at 4:14 AM Manu Zhang > < > >> >> > > >> > > > >> [email protected]> > >> >> > > >> > > > >> > > > wrote: > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > > +1 moving the DataFusion integration and > >> tests > >> >> > > into a > >> >> > > >> > > > separate > >> >> > > >> > > > >> > > > > > > apache-governed repository. Can we bring > >> this > >> >> > > >> discussion > >> >> > > >> > > to > >> >> > > >> > > > >> the > >> >> > > >> > > > >> > > > Community > >> >> > > >> > > > >> > > > > > > Sync[1] this week? > >> >> > > >> > > > >> > > > > > > > >> >> > > >> > > > >> > > > > > > 1. > >> >> > > >> > > > >> > > > > > > > >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > >> >> > > >> > > > > >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > > >> >> > > >> >> > >> > https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit?tab=t.0 > >> >> > > >> > > > >> > > > > > > > >> >> > > >> > > > >> > > > > > > On Mon, Aug 24, 2026 at 3:37 PM Renjie > Liu > >> < > >> >> > > >> > > > >> > > [email protected]> > >> >> > > >> > > > >> > > > > > > wrote: > >> >> > > >> > > > >> > > > > > > > >> >> > > >> > > > >> > > > > > >> Hi, Matt: > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> Thanks for raising this. > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> 1) Apache governance: > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> I would +1 for putting this in an apache > >> >> repo, > >> >> > for > >> >> > > >> > > example > >> >> > > >> > > > a > >> >> > > >> > > > >> sub > >> >> > > >> > > > >> > > > repo of > >> >> > > >> > > > >> > > > > > >> datafusion project. Shawn has stated > most > >> of > >> >> the > >> >> > > >> > reasons, > >> >> > > >> > > > so > >> >> > > >> > > > >> I > >> >> > > >> > > > >> > > > don't want > >> >> > > >> > > > >> > > > > > >> to repeat it again. > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> 2) Where do tests live/how tests should > be > >> >> > > >> maintained? > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> Initially I was thinking about putting > >> >> > > sqllogictest > >> >> > > >> in > >> >> > > >> > > > >> > > > iceberg-rust, but > >> >> > > >> > > > >> > > > > > >> after second thought I'm leaning towards > >> to > >> >> put > >> >> > it > >> >> > > >> in > >> >> > > >> > the > >> >> > > >> > > > >> new repo > >> >> > > >> > > > >> > > > for two > >> >> > > >> > > > >> > > > > > >> reasons: > >> >> > > >> > > > >> > > > > > >> 1. It would be easier for developer of > the > >> >> > > >> > > > datafusion-iceberg > >> >> > > >> > > > >> > > > integration > >> >> > > >> > > > >> > > > > > >> to add tests > >> >> > > >> > > > >> > > > > > >> 2. It would make the dependency graph > and > >> >> > version > >> >> > > >> > release > >> >> > > >> > > > >> easier. > >> >> > > >> > > > >> > > > Though > >> >> > > >> > > > >> > > > > > >> the dependency is on crate level rather > >> than > >> >> > repo > >> >> > > >> > level, > >> >> > > >> > > > the > >> >> > > >> > > > >> > > > bi-direction > >> >> > > >> > > > >> > > > > > >> dependency may make version management > >> weird > >> >> and > >> >> > > >> > > difficult. > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> The downside of this approach is that > >> it's a > >> >> > > little > >> >> > > >> > > > >> unfriendly for > >> >> > > >> > > > >> > > > > > >> iceberg-rust developers, but I think > it's > >> >> less > >> >> > > >> frequent > >> >> > > >> > > for > >> >> > > >> > > > >> > > > iceberg-rust > >> >> > > >> > > > >> > > > > > >> developers to add sqllogictests compared > >> with > >> >> > > >> > > > >> datafusion-iceberg > >> >> > > >> > > > >> > > > developers. > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> 3) Dropping DataFusion dependencies from > >> >> > > >> pyiceberg-core > >> >> > > >> > > > >> binding > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> I'm not quite familiar with this part, > >> but it > >> >> > > sounds > >> >> > > >> > > > >> reasonable to > >> >> > > >> > > > >> > > > me. > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >> On Sat, Aug 22, 2026 at 6:50 AM Shawn > >> Chang < > >> >> > > >> > > > >> > > [email protected] > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > >> wrote: > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > >>> Hi Matt, > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> Thanks for raising this! I generally > >> agree > >> >> that > >> >> > > we > >> >> > > >> can > >> >> > > >> > > > move > >> >> > > >> > > > >> the > >> >> > > >> > > > >> > > > > > >>> datafusion > >> >> > > >> > > > >> > > > > > >>> integration to a separate repo if that > >> helps > >> >> > more > >> >> > > >> > > > DataFusion > >> >> > > >> > > > >> > > > experts to > >> >> > > >> > > > >> > > > > > >>> work on the integrations > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> On the three considerations: > >> >> > > >> > > > >> > > > > > >>> 1) Apache governance: > >> >> > > >> > > > >> > > > > > >>> I think this is my biggest concern so > >> far. > >> >> This > >> >> > > >> could > >> >> > > >> > > not > >> >> > > >> > > > >> only > >> >> > > >> > > > >> > > > affect > >> >> > > >> > > > >> > > > > > >>> contributors, but also whether the > >> >> downstream > >> >> > > users > >> >> > > >> > > could > >> >> > > >> > > > >> > > continue > >> >> > > >> > > > >> > > > to use > >> >> > > >> > > > >> > > > > > >>> the integration. > >> >> > > >> > > > >> > > > > > >>> Even if the code remains > Apache-licensed, > >> >> > > >> governance, > >> >> > > >> > > > >> release of > >> >> > > >> > > > >> > > > > > >>> artifacts, > >> >> > > >> > > > >> > > > > > >>> and contribution policies can matter > for > >> >> > > adoption. > >> >> > > >> > > > >> > > > > > >>> We should explore more about the option > >> to > >> >> keep > >> >> > > it > >> >> > > >> > under > >> >> > > >> > > > an > >> >> > > >> > > > >> > > > > > >>> Apache-governed > >> >> > > >> > > > >> > > > > > >>> repository. > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> 2) Where do tests live/how tests should > >> be > >> >> > > >> maintained? > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> I think this is closely related to the > >> >> > governance > >> >> > > >> > > > question. > >> >> > > >> > > > >> To > >> >> > > >> > > > >> > > me, > >> >> > > >> > > > >> > > > the > >> >> > > >> > > > >> > > > > > >>> broader question is: *how do we make it > >> easy > >> >> > for > >> >> > > >> > people > >> >> > > >> > > > >> from both > >> >> > > >> > > > >> > > > the > >> >> > > >> > > > >> > > > > > >>> Iceberg Rust and DataFusion communities > >> to > >> >> > > maintain > >> >> > > >> > the > >> >> > > >> > > > >> > > > integration?* > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> If we move it to a non-Apache > >> repository, I > >> >> > worry > >> >> > > >> that > >> >> > > >> > > it > >> >> > > >> > > > >> could > >> >> > > >> > > > >> > > > > > >>> eventually > >> >> > > >> > > > >> > > > > > >>> look somewhat like the current Iceberg > >> Java > >> >> <> > >> >> > > >> Trino > >> >> > > >> > > > >> integration: > >> >> > > >> > > > >> > > > the > >> >> > > >> > > > >> > > > > > >>> integration primarily lives on the > Trino > >> >> side > >> >> > and > >> >> > > >> is > >> >> > > >> > > > >> therefore > >> >> > > >> > > > >> > > > mostly > >> >> > > >> > > > >> > > > > > >>> maintained by people who are already > >> deeply > >> >> > > >> involved > >> >> > > >> > in > >> >> > > >> > > > >> Trino. > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> There is an important difference here, > >> >> though. > >> >> > > For > >> >> > > >> > > Iceberg > >> >> > > >> > > > >> Java, > >> >> > > >> > > > >> > > > Spark is > >> >> > > >> > > > >> > > > > > >>> arguably the primary engine integration > >> and > >> >> > has a > >> >> > > >> > large > >> >> > > >> > > > >> Iceberg > >> >> > > >> > > > >> > > > > > >>> contributor > >> >> > > >> > > > >> > > > > > >>> base around it, so having the Trino > >> >> integration > >> >> > > >> > > maintained > >> >> > > >> > > > >> more > >> >> > > >> > > > >> > > > > > >>> independently is relatively natural. In > >> >> Iceberg > >> >> > > >> Rust > >> >> > > >> > > > today, > >> >> > > >> > > > >> > > > DataFusion > >> >> > > >> > > > >> > > > > > >>> has > >> >> > > >> > > > >> > > > > > >>> a much more central role. It is by far > >> the > >> >> most > >> >> > > >> mature > >> >> > > >> > > > >> engine > >> >> > > >> > > > >> > > > integration > >> >> > > >> > > > >> > > > > > >>> in the project, is used by our > >> SQLLogicTest > >> >> > > >> > > > infrastructure, > >> >> > > >> > > > >> and > >> >> > > >> > > > >> > > > many > >> >> > > >> > > > >> > > > > > >>> users > >> >> > > >> > > > >> > > > > > >>> building on Iceberg Rust are also > >> building > >> >> on > >> >> > > >> > > DataFusion. > >> >> > > >> > > > >> The > >> >> > > >> > > > >> > > > overlap > >> >> > > >> > > > >> > > > > > >>> between the two communities is > therefore > >> >> much > >> >> > > >> larger. > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> Because of that, I would prefer that > >> >> extracting > >> >> > > the > >> >> > > >> > > > >> integration > >> >> > > >> > > > >> > > > does not > >> >> > > >> > > > >> > > > > > >>> turn it into something that is > >> effectively > >> >> > owned > >> >> > > >> only > >> >> > > >> > by > >> >> > > >> > > > the > >> >> > > >> > > > >> > > > DataFusion > >> >> > > >> > > > >> > > > > > >>> side. Ideally, both Iceberg Rust > >> >> contributors > >> >> > and > >> >> > > >> > > > DataFusion > >> >> > > >> > > > >> > > > contributors > >> >> > > >> > > > >> > > > > > >>> should be able to review changes, > >> maintain > >> >> > > >> > > compatibility, > >> >> > > >> > > > >> and > >> >> > > >> > > > >> > > > evolve the > >> >> > > >> > > > >> > > > > > >>> integration together. > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> 3) Dropping DataFusion dependencies > from > >> >> > > >> > pyiceberg-core > >> >> > > >> > > > >> binding: > >> >> > > >> > > > >> > > I > >> >> > > >> > > > >> > > > think > >> >> > > >> > > > >> > > > > > >>> it > >> >> > > >> > > > >> > > > > > >>> makes sense to make things simpler and > >> have > >> >> > left > >> >> > > a > >> >> > > >> > > comment > >> >> > > >> > > > >> on the > >> >> > > >> > > > >> > > > github > >> >> > > >> > > > >> > > > > > >>> issue with more detailed thoughts: > >> >> > > >> > > > >> > > > > > >>> > >> >> > > https://github.com/apache/iceberg-rust/issues/3036 > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> Best, > >> >> > > >> > > > >> > > > > > >>> Shawn > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> On Fri, Aug 21, 2026 at 11:23 AM Matt > >> >> > Butrovich < > >> >> > > >> > > > >> > > > [email protected]> > >> >> > > >> > > > >> > > > > > >>> wrote: > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > Hello all, > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > I've never tried emailing two > different > >> >> > project > >> >> > > >> > lists > >> >> > > >> > > at > >> >> > > >> > > > >> once, > >> >> > > >> > > > >> > > > but it > >> >> > > >> > > > >> > > > > > >>> was > >> >> > > >> > > > >> > > > > > >>> > suggested that I do so to try to > track > >> the > >> >> > > >> > > conversation > >> >> > > >> > > > >> across > >> >> > > >> > > > >> > > > both > >> >> > > >> > > > >> > > > > > >>> > communities. We'll see how this > >> threads on > >> >> > the > >> >> > > >> > mailing > >> >> > > >> > > > >> lists. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > There has been a GitHub Discussion > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > >> >> > > >> > > > > >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > > >> >> > > >> >> > >> > https://github.com/apache/iceberg-rust/discussions/2992#discussioncomment-18082533 > >> >> > > >> > > > >> > > > > > >>> , > >> >> > > >> > > > >> > > > > > >>> > a GitHub Issue > >> >> > > >> > > > >> > > > > >> https://github.com/apache/iceberg-rust/issues/3029, > >> >> > and > >> >> > > >> > > > >> > > > > > >>> > it's been a long topic of > conversation > >> in > >> >> the > >> >> > > >> past > >> >> > > >> > two > >> >> > > >> > > > >> weeks in > >> >> > > >> > > > >> > > > both > >> >> > > >> > > > >> > > > > > >>> the > >> >> > > >> > > > >> > > > > > >>> > DataFusion and Iceberg Rust Community > >> >> Calls > >> >> > to > >> >> > > >> > discuss > >> >> > > >> > > > >> moving > >> >> > > >> > > > >> > > the > >> >> > > >> > > > >> > > > > > >>> > DataFusion integration from Iceberg > >> Rust > >> >> to a > >> >> > > >> > separate > >> >> > > >> > > > >> > > > repository. I > >> >> > > >> > > > >> > > > > > >>> will > >> >> > > >> > > > >> > > > > > >>> > try to summarize some of the major > >> points > >> >> as > >> >> > I > >> >> > > >> > > > understand > >> >> > > >> > > > >> them, > >> >> > > >> > > > >> > > > but the > >> >> > > >> > > > >> > > > > > >>> > conversations are the ground truth > and > >> >> please > >> >> > > >> feel > >> >> > > >> > > free > >> >> > > >> > > > to > >> >> > > >> > > > >> > > > correct me > >> >> > > >> > > > >> > > > > > >>> here. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > The DataFusion TableProvider > >> integration > >> >> in > >> >> > > >> Iceberg > >> >> > > >> > > Rust > >> >> > > >> > > > >> makes > >> >> > > >> > > > >> > > > > > >>> DataFusion > >> >> > > >> > > > >> > > > > > >>> > a dependency for Iceberg Rust. The > >> >> > integration > >> >> > > >> > exists > >> >> > > >> > > > for > >> >> > > >> > > > >> > > > multiple > >> >> > > >> > > > >> > > > > > >>> reasons: > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 1) an engine to execute Iceberg > Rust's > >> >> corpus > >> >> > > of > >> >> > > >> > > > >> sqllogictest > >> >> > > >> > > > >> > > > files > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 2) a TableProvider integration for > >> >> DataFusion > >> >> > > >> users > >> >> > > >> > to > >> >> > > >> > > > >> interact > >> >> > > >> > > > >> > > > with > >> >> > > >> > > > >> > > > > > >>> > Iceberg tables > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > Some of the motivations to break out > >> the > >> >> > > >> > integration: > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 1) There have been a number of issues > >> and > >> >> > pull > >> >> > > >> > > requests > >> >> > > >> > > > >> against > >> >> > > >> > > > >> > > > the > >> >> > > >> > > > >> > > > > > >>> > DataFusion TableProvider in Iceberg > >> Rust > >> >> as > >> >> > > users > >> >> > > >> > want > >> >> > > >> > > > to > >> >> > > >> > > > >> add > >> >> > > >> > > > >> > > > more > >> >> > > >> > > > >> > > > > > >>> > features, and they often go stale. I > >> don't > >> >> > > >> believe > >> >> > > >> > > there > >> >> > > >> > > > >> are > >> >> > > >> > > > >> > > many > >> >> > > >> > > > >> > > > > > >>> > committers/PMC members familiar with > or > >> >> using > >> >> > > the > >> >> > > >> > > > >> DataFusion > >> >> > > >> > > > >> > > > > > >>> integration. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 2) Iceberg Rust would like to stay as > >> >> > > >> > engine-agnostic > >> >> > > >> > > as > >> >> > > >> > > > >> > > > possible. A > >> >> > > >> > > > >> > > > > > >>> > recent DataFusion Ballista > integration > >> was > >> >> > > >> declined > >> >> > > >> > > for > >> >> > > >> > > > >> this > >> >> > > >> > > > >> > > > reason > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > https://github.com/apache/iceberg-rust/pull/2613 > >> >> > > >> . > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 3) Other projects that rely on both > >> >> Iceberg > >> >> > > Rust > >> >> > > >> and > >> >> > > >> > > > >> DataFusion > >> >> > > >> > > > >> > > > (e.g., > >> >> > > >> > > > >> > > > > > >>> > Comet) are blocked by Iceberg Rust > >> >> upgrading > >> >> > > its > >> >> > > >> > > > >> DataFusion and > >> >> > > >> > > > >> > > > Arrow > >> >> > > >> > > > >> > > > > > >>> > dependencies before they can upgrade. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > Some considerations for both > >> communities: > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 1) Where would this DataFusion > >> >> TableProvider > >> >> > > >> live? > >> >> > > >> > > Most > >> >> > > >> > > > >> > > > specifically, > >> >> > > >> > > > >> > > > > > >>> > would it be an Apache-governed > >> project? As > >> >> > > folks > >> >> > > >> > like > >> >> > > >> > > > >> > > @andygrove > >> >> > > >> > > > >> > > > point > >> >> > > >> > > > >> > > > > > >>> out, > >> >> > > >> > > > >> > > > > > >>> > this can affect whether some > community > >> >> > members > >> >> > > >> could > >> >> > > >> > > > >> contribute > >> >> > > >> > > > >> > > > to it. > >> >> > > >> > > > >> > > > > > >>> > There is a datafusion-contrib org for > >> >> > > >> > > DataFusion-related > >> >> > > >> > > > >> > > > projects to > >> >> > > >> > > > >> > > > > > >>> have > >> >> > > >> > > > >> > > > > > >>> > visibility but no Apache governance, > >> but > >> >> > there > >> >> > > >> may > >> >> > > >> > be > >> >> > > >> > > > >> options > >> >> > > >> > > > >> > > to > >> >> > > >> > > > >> > > > put it > >> >> > > >> > > > >> > > > > > >>> > under an Apache repository. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 2) How would Iceberg Rust continue to > >> run > >> >> > > >> > > sqllogictests > >> >> > > >> > > > >> for > >> >> > > >> > > > >> > > > regression > >> >> > > >> > > > >> > > > > > >>> > testing? Does this live in a > different > >> >> > > repository > >> >> > > >> > that > >> >> > > >> > > > >> depends > >> >> > > >> > > > >> > > > on this > >> >> > > >> > > > >> > > > > > >>> new > >> >> > > >> > > > >> > > > > > >>> > Iceberg Rust TableProvider crate? > Would > >> >> we be > >> >> > > >> able > >> >> > > >> > to > >> >> > > >> > > > >> test Pull > >> >> > > >> > > > >> > > > > > >>> Requests on > >> >> > > >> > > > >> > > > > > >>> > Iceberg Rust with sqllogictests? > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > 3) @kevinqliu is familiar with the > >> Python > >> >> > > >> bindings > >> >> > > >> > in > >> >> > > >> > > > >> Iceberg > >> >> > > >> > > > >> > > > Rust, > >> >> > > >> > > > >> > > > > > >>> and I > >> >> > > >> > > > >> > > > > > >>> > believe that DataFusion dependency > >> might > >> >> be > >> >> > > >> removed > >> >> > > >> > as > >> >> > > >> > > > >> well, > >> >> > > >> > > > >> > > but > >> >> > > >> > > > >> > > > that > >> >> > > >> > > > >> > > > > > >>> is > >> >> > > >> > > > >> > > > > > >>> > not the core focus of this > >> conversation. > >> >> He > >> >> > has > >> >> > > >> also > >> >> > > >> > > > >> proposed > >> >> > > >> > > > >> > > > removing > >> >> > > >> > > > >> > > > > > >>> that > >> >> > > >> > > > >> > > > > > >>> > and using the DataFusion Python > >> bindings > >> >> > > >> directly. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > I'm sure I'm forgetting things, but > >> this > >> >> > email > >> >> > > is > >> >> > > >> > long > >> >> > > >> > > > >> enough. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > Thanks everyone for the discussion > thus > >> >> far, > >> >> > > and > >> >> > > >> > > looking > >> >> > > >> > > > >> > > forward > >> >> > > >> > > > >> > > > to > >> >> > > >> > > > >> > > > > > >>> more > >> >> > > >> > > > >> > > > > > >>> > input on this. > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > -Matt > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >> > >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > Xuanwo > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > https://xuanwo.io/ > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > >> >> > > >> > > > >> > >> >> > > >> > > >> >> > > --------------------------------------------------------------------- > >> >> > > >> > > > >> To unsubscribe, e-mail: > >> >> > [email protected] > >> >> > > >> > > > >> For additional commands, e-mail: > >> >> > > [email protected] > >> >> > > >> > > > >> > >> >> > > >> > > > >> > >> >> > > >> > > > > >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > > > >> >> > > > >> >> > > >> >> > >> > > >> > > >
