Hi Airflow community, I would like to propose a new provider for Apache DataFusion: apache-airflow-providers-apache-datafusion.
Background: During the AIP-99 (common.ai) discussions, we agreed to use Apache DataFusion to support querying files on object stores. This functionality powers the SQLToolset and the LLM data-quality operators within common.ai, but the engine code currently lives inside the common.sql provider and only supports Amazon S3 and local files. Proposal: I propose moving the DataFusion functionality out of common.sql into a dedicated provider under providers/apache/datafusion, and extending it to support additional storage backends and file formats. This keeps common.sql focused on DB-API databases while giving DataFusion a dedicated home to be discovered, used, and extended independently. Why DataFusion? - In-process, Arrow-native engine: Runs directly inside the worker without requiring an external service or cluster. - Vectorized, streaming execution: Queries datasets larger than the worker's RAM efficiently. - Built-in object store support: Straightforward extension to S3, GCS, Azure, and HTTP. On a personal note, I use DataFusion daily to analyze datasets of over 50 million rows on a single machine, which convinces me it is an excellent fit for Airflow. Performance References: - ClickBench Benchmarks: DataFusion consistently ranks among the fastest single-node engines for querying Parquet files: https://benchmark.clickhouse.com/ - Version 54.0.0 Release (June 2026): Significant join, scan, and planning improvements: https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0 - Dynamic Filters (September 2025): Up to 25x faster queries for common patterns: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters - SIGMOD 2024 Paper: Peer-reviewed reference on architecture and performance: https://dl.acm.org/doi/10.1145/3626246.3653368 Regards, Pavan
