Thanks Andy for raising this. +1 for the iceberg-rust solution. This could benefit the whole oss community in the long term and avoid duplicated efforts. As with the technique challenges you mentioned, this seems more like a missing feature rather than limitation.
On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <[email protected]> wrote: > Thanks Andy for starting this discussion! > > I agree we should prioritize the iceberg-rust path as the long term > default for Comet/Iceberg scans, while it's still valuable to keep the > Iceberg Java integration as an option, especially for users who need > JVM-specific features. I'm hoping we can get consensus on a path to resolve > the shading issues (e.g. the approach proposed in > github.com/apache/iceberg/pull/13786) so the java option can move > forward. > > Thanks, > Huaxin > > On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <[email protected]> > wrote: > >> Thanks Andy for starting this thread! >> >> I've replied to the issue above but am posting here as well for better >> visibility: >> >> I’m leaning toward prioritizing the iceberg-rust path to avoid the >> shading and circular dependency headaches of the Java approach. Easier >> maintenance typically lowers the entry barrier and encourages more >> community involvement, which is important for the project’s long-term >> health. >> >> It also makes sense to keep the Iceberg Java integration available as an >> experimental option for users who need its richer JVM-specific features >> while Rust becomes the long-term default. We don’t want to block >> contributions from anyone who truly needs the Java path. >> >> Best, >> Shawn >> >> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <[email protected]> >> wrote: >> >>> Hi, >>> >>> There is a discussion in the DataFusion Comet community about the future >>> direction of our Iceberg support [1]. For example, should we continue the >>> efforts to integrate with the Iceberg Java project or focus more on the >>> iceberg-rust project. >>> >>> It would be great to also get input from the Iceberg community. >>> >>> Thanks, >>> >>> Andy. >>> >>> [1] https://github.com/apache/datafusion-comet/issues/2921 >>> >>
