Hi Pavan, Thanks for proposing this. +1 from my side.
Best, Przemek On 2026/08/08 06:42:41 Pavankumar Gopidesu wrote: > Hi Airflow community, > > I would like to propose a new provider for Apache DataFusion: > apache-airflow-providers-apache-datafusion. > > Background: > > During the AIP-99 (common.ai) discussions, we agreed to use Apache > DataFusion to support querying files on object stores. This functionality > powers the SQLToolset and the LLM data-quality operators within common.ai, > but the engine code currently lives inside the common.sql provider and only > supports Amazon S3 and local files. > > Proposal: > > I propose moving the DataFusion functionality out of common.sql into a > dedicated provider under providers/apache/datafusion, and extending it to > support additional storage backends and file formats. This keeps common.sql > focused on DB-API databases while giving DataFusion a dedicated home to be > discovered, used, and extended independently. > > Why DataFusion? > > - In-process, Arrow-native engine: Runs directly inside the worker > without requiring an external service or cluster. > - Vectorized, streaming execution: Queries datasets larger than the > worker's RAM efficiently. > - Built-in object store support: Straightforward extension to S3, GCS, > Azure, and HTTP. > > On a personal note, I use DataFusion daily to analyze datasets of over 50 > million rows on a single machine, which convinces me it is an excellent fit > for Airflow. > > Performance References: > > - ClickBench Benchmarks: DataFusion consistently ranks among the fastest > single-node engines for querying Parquet files: > https://benchmark.clickhouse.com/ > - Version 54.0.0 Release (June 2026): Significant join, scan, and > planning improvements: > https://datafusion.apache.org/blog/2026/06/12/datafusion-54.0.0 > - Dynamic Filters (September 2025): Up to 25x faster queries for common > patterns: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters > - SIGMOD 2024 Paper: Peer-reviewed reference on architecture and > performance: https://dl.acm.org/doi/10.1145/3626246.3653368 > > Regards, > Pavan > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
