Dear Developers, I am opening this discussion on the dev mailing list as an Iceberg Improvement Plan (IIP) for the following issue: https://github.com/apache/iceberg-python/issues/3737.
I have opened an initial PR at https://github.com/apache/iceberg-python/pull/3738 to gather feedback, building on constructive input already received from the community. The goal is to introduce a pluggable backend that allows swapping the default Python+PyArrow compute workload for Apache DataFusion. This will provide spill-to-disk capabilities to prevent out-of-memory (OOM) errors on production-sized data during key operations. Before moving forward, I would like to build broader consensus. This proposal follows several iterations and discussions on Slack and during Python sync meetings. Previous approaches include: - Issue & Design Doc (#3554): https://github.com/apache/iceberg-python/issues/3554 - Feedback: Rather than fixing specific operations for DataFusion, we should explore a fully pluggable backend architecture. - Consensus: Create a separate issue and PR for a pluggable architecture. - Issue, Design Doc, & PR (#3715 / #3716): https://github.com/apache/iceberg-python/issues/3715 / https://github.com/apache/iceberg-python/pull/3716 - Feedback: The diff was too large to review effectively (20k+ LOC, primarily tests), even though core changes were only about 3k LOC. - Consensus: Refactor the existing 3k+ line pyarrow.py monolith first, making future changes like integrating DataFusion easier to understand. This history brings us to #3737 and #3738, which outline a multi-stage refactor plan to separate the core components of pyarrow.py into logical sections (e.g., FileIO, schema, write, read). The initial phase will introduce no functional changes to ensure there are no regressions. Once this refactoring is complete, we can build a protocol or interface to support alternative backends (such as DataFusion for compute or DuckDB for reading), which will be much more feasible once decoupled from the current monolith. Currently, several PyIceberg features face OOM risks on large datasets, including core features like reading equality delete files and critical maintenance operations like data compaction. Feedback indicates the community would highly value these scaling improvements. I am seeking guidance on how to execute this change meaningfully, ensuring a stable and well-considered implementation. I look forward to your feedback. -- Best regards, Jared Yu B.S. Statistics - University of California, Davis '19 M.S. Data Science - Johns Hopkins University '22
