Sounds good! On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <[email protected]> wrote:
> Hi folks, > > I'd like to propose adding a new community provider for DuckDB. > > Integration overview: > --------------------- > > DuckDB is an in-process analytical (OLAP) database. Unlike most SQL > providers there is no server to connect to: queries run inside the task > process, which makes it a cheap way to run an analytical transform that > doesn't justify standing up a cluster. It reads and writes Parquet, CSV and > JSON directly from object storage, so it fits naturally between an extract > and a load step in a > Dag. > > It also fits Airflow's existing shape well: The proposed DuckDB provider > speaks DbAPI, so it plugs into the common-sql ecosystem rather than needing > new machinery, and it fits in the data-processing/ETL space. DuckDB Labs > was acquired by Amazon but DuckDB remains MIT licensed and controlled by > the DuckDB Foundation. It is usable with no account, no signup and no > hosted service. > > The provider exposes a DuckDBHook extending DbApiHook, so it works with > SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a > DuckDBExecuteQueryOperator and a duckdb connection type. The connection is > optional: with none configured the hook opens an in-memory database. > > Provider specific integrations would live in their relevant provider > package (e.g. for AWS, an AwsDuckDBHook in apache-airflow-providers-amazon > for credential management). > > System tests: > ------------- > > DuckDB runs in-process with no live external service, so system tests are > not > required for this provider; the unit tests exercise a real in-memory > DuckDB. > An AWS-specific system test could be added in the Amazon provider package, > if folks think that's required. > > Proposed stewards: > ------------------ > > Myself and the AWS team plan to sponsor and steward this package, you know > us and our track record well :) > > - Niko Oliveira (@o-nikolas) > - Vincent Beck (@vincbeck) > > Working implementation > ---------------------- > > https://github.com/apache/airflow/pull/72845 > > Incubation commitment > --------------------- > > We commit to: > > - Maintaining the provider and responding to issues within a reasonable > time > - Meeting the incubation health metrics within 6 months > - Participating in quarterly governance updates > > Cheers, > Niko >
