Great to see there is interest in this one, I think the guys at Apache Arrow are delivering amazing work with regular updates and improvements, recently they also added the dbc command tool to facilitate installation of ADBC drivers, see https://columnar.tech/dbc/
While testing a while ago with Postgres I discovered an issue with the ADBC implementation, they quickly solved the issue, so yes I'm pretty confident this would be a great addition, and we should support and integrate other Apache projects as well 😊 I agree regarding AIP-99 concerning the Common Data Access Pattern. I think if I remember correctly, TP already mentioned this during a talk at the Summit last year (https://www.youtube.com/watch?v=KA6iDX5MVi4&t=3swith) with some example syntax of how it could look like with the I/O manager. Back then I was already sold regarding the idea and proposition! I personally saw it more like the ability to chain hooks for example, so like hook.transfer(another_hook, **options), but that's another discussion. Regards, David -----Original Message----- From: Pavankumar Gopidesu <[email protected]> Sent: 03 March 2026 17:06 To: [email protected] Subject: Re: [PROPOSAL] ADBC Provider for Apache Airflow EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto:[email protected]>. Hi David, Thanks for raising this proposal; it would be a great addition. I was actually planning to discuss this with you, as I believe it will support AIP-99 by allowing this hook to provide rich context and sample data for LLMs. I am +1 for this :) Regards Pavan On Tue, Mar 3, 2026 at 3:53 PM Blain David <[email protected]> wrote: > Hello everyone, > > Following some initial discussions with Jarek Potiuk and a previously > opened PR, I would like to formally propose the introduction of an > Apache Arrow / ADBC provider for Airflow. > > Context & Motivation: > > While Airflow has a rich set of database-specific providers, the data > ecosystem is rapidly shifting toward ADBC (Arrow Database Connectivity). > ADBC solves many of the "bottleneck" issues associated with > traditional DB-API 2.0, ODBC or JDBC drivers by leveraging columnar > data access and Arrow-native memory representation. > > We are seeing significant momentum here: > > > * Performance: Significant reduction in serialization overhead for > bulk operations. While results vary by driver maturity and server-side > native Arrow support (e.g., flight endpoints), ADBC provides a much > higher performance ceiling than standard PEP 249 drivers. > * Standardization: Systems like Snowflake, Apache DataFusion and > DuckDB are increasingly treating Arrow as a first-class citizen. > * Future-proofing: Tools like dbt-fusion and various lakehouse > architectures are moving toward Arrow-based execution. > > The Proposal: > > I propose adding an apache-airflow-providers-apache-arrow (or similar) > that introduces an AdbcHook. > > Key Technical Highlights: > > > * Compatibility: By implementing DbApiHook, the AdbcHook will be > immediately compatible with existing SQL operators. > * Efficiency: It will offer a high-performance alternative to > traditional row-based drivers without requiring users to rewrite their > DAG logic. > * Scope: Focus on providing a standardized interface for Arrow-native > bulk reads and writes (future enhancement in AdbcHook). > > Community & Maintenance: > > I have already started the groundwork in a Draft PR (#52330). > > I believe this aligns with the project's goal of supporting > high-performance data engineering patterns. I'm looking for feedback on: > > > * Naming: Should this be a standalone adbc provider or part of an > apache.arrow provider? I chose the later but to be discussed. > * Scope: At the moment I was only focusing purely on the > Hook/Connection, as it extends the DbAPiHook and implements all > required methods, it's already directly useable in SQL-operators. > > I'd love to gather your thoughts and gauge interest before moving to a > formal voting thread. > > Draft PR: > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fapache%2Fairflow%2Fpull%2F52330&data=05%7C02%7Cdavid.blain%40 > infrabel.be%7C83703c4a38864b35d2a708de793eedc4%7Cb82bc314ab8e4d6fb1894 > 6f02e1f27f2%7C0%7C0%7C639081508341385655%7CUnknown%7CTWFpbGZsb3d8eyJFb > XB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCI > sIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kq9Dw3GJB67UTw2Zw3rLWuMItwbNo0JSX > 7rGTZr23rk%3D&reserved=0 > > Best regards, > David >
