GitHub user gabotechs created a discussion: Moving the DataFusion integration into a separate repository
I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing [DataFusion integration](https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion) into a crate maintained in a separate repository. As discussed in [datafusion-distributed #599](https://github.com/datafusion-contrib/datafusion-distributed/issues/599), we are exploring incorporating the integration into [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed), making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg. Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors. Some reasons why this may be worth considering: - There appears to be a vision of making iceberg-rust engine-agnostic ([related discussion](https://github.com/apache/iceberg-rust/pull/2613#issuecomment-4830211788), cc @blackmwk). - Several integration-related PRs have had difficulty receiving review attention in this repository, such as [#2671](https://github.com/apache/iceberg-rust/pull/2671) and [#2521](https://github.com/apache/iceberg-rust/pull/2521). - Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including [Ballista #2174](https://github.com/apache/datafusion-ballista/issues/2174) and [datafusion-distributed #599](https://github.com/datafusion-contrib/datafusion-distributed/issues/599). I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as [Ballista](https://github.com/apache/datafusion-ballista) and [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle. GitHub link: https://github.com/apache/iceberg-rust/discussions/2992 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
