This is a good initiative, I'm in favor for this one! Is this based on the Astronomer one? https://github.com/astronomer/airflow-provider-duckdb
Kind regards, David General (Internal Property) ________________________________ From: Niko Oliveira <[email protected]> Sent: Thursday, September 10, 2026 08:03 To: [email protected] <[email protected]> Subject: [DISCUSS] New Community Provider: DuckDB EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto:[email protected]>. Hi folks, I'd like to propose adding a new community provider for DuckDB. Integration overview: --------------------- DuckDB is an in-process analytical (OLAP) database. Unlike most SQL providers there is no server to connect to: queries run inside the task process, which makes it a cheap way to run an analytical transform that doesn't justify standing up a cluster. It reads and writes Parquet, CSV and JSON directly from object storage, so it fits naturally between an extract and a load step in a Dag. It also fits Airflow's existing shape well: The proposed DuckDB provider speaks DbAPI, so it plugs into the common-sql ecosystem rather than needing new machinery, and it fits in the data-processing/ETL space. DuckDB Labs was acquired by Amazon but DuckDB remains MIT licensed and controlled by the DuckDB Foundation. It is usable with no account, no signup and no hosted service. The provider exposes a DuckDBHook extending DbApiHook, so it works with SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a DuckDBExecuteQueryOperator and a duckdb connection type. The connection is optional: with none configured the hook opens an in-memory database. Provider specific integrations would live in their relevant provider package (e.g. for AWS, an AwsDuckDBHook in apache-airflow-providers-amazon for credential management). System tests: ------------- DuckDB runs in-process with no live external service, so system tests are not required for this provider; the unit tests exercise a real in-memory DuckDB. An AWS-specific system test could be added in the Amazon provider package, if folks think that's required. Proposed stewards: ------------------ Myself and the AWS team plan to sponsor and steward this package, you know us and our track record well :) - Niko Oliveira (@o-nikolas) - Vincent Beck (@vincbeck) Working implementation ---------------------- https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F72845&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8a280eeef1984f1ea80808df0f015e29%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639246170692897451%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Rc9w5Ilzv4w0Ki1kpwMyP6sNAR0no5m7OtN1WdwSt24%3D&reserved=0<https://github.com/apache/airflow/pull/72845> Incubation commitment --------------------- We commit to: - Maintaining the provider and responding to issues within a reasonable time - Meeting the incubation health metrics within 6 months - Participating in quarterly governance updates Cheers, Niko
