I wrote our original DuckDB provider at Astronomer, would be more than happy to deprecate in favor of yours! Definitely would love to see something more first-class and better supported.
> On Sep 10, 2026, at 11:23 AM, Niko Oliveira <[email protected]> wrote: > > Glad to see the traction, thanks for weighing in everyone! > >> Is this based on the Astronomer one? > https://github.com/astronomer/airflow-provider-duckdb > > No code was based on that provider. It hasn't been updated in 3 years and > it focused on a minimal set of DuckDB features. There wasn't much to reuse > there. > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <[email protected]> wrote: > >> +10 >> >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <[email protected]> wrote: >> >>> Sounds good! >>> >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <[email protected]> wrote: >>> >>>> Hi folks, >>>> >>>> I'd like to propose adding a new community provider for DuckDB. >>>> >>>> Integration overview: >>>> --------------------- >>>> >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL >>>> providers there is no server to connect to: queries run inside the task >>>> process, which makes it a cheap way to run an analytical transform that >>>> doesn't justify standing up a cluster. It reads and writes Parquet, CSV >>> and >>>> JSON directly from object storage, so it fits naturally between an >>> extract >>>> and a load step in a >>>> Dag. >>>> >>>> It also fits Airflow's existing shape well: The proposed DuckDB >> provider >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather than >>> needing >>>> new machinery, and it fits in the data-processing/ETL space. DuckDB >> Labs >>>> was acquired by Amazon but DuckDB remains MIT licensed and controlled >> by >>>> the DuckDB Foundation. It is usable with no account, no signup and no >>>> hosted service. >>>> >>>> The provider exposes a DuckDBHook extending DbApiHook, so it works with >>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus >> a >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The connection >>> is >>>> optional: with none configured the hook opens an in-memory database. >>>> >>>> Provider specific integrations would live in their relevant provider >>>> package (e.g. for AWS, an AwsDuckDBHook in >>> apache-airflow-providers-amazon >>>> for credential management). >>>> >>>> System tests: >>>> ------------- >>>> >>>> DuckDB runs in-process with no live external service, so system tests >> are >>>> not >>>> required for this provider; the unit tests exercise a real in-memory >>>> DuckDB. >>>> An AWS-specific system test could be added in the Amazon provider >>> package, >>>> if folks think that's required. >>>> >>>> Proposed stewards: >>>> ------------------ >>>> >>>> Myself and the AWS team plan to sponsor and steward this package, you >>> know >>>> us and our track record well :) >>>> >>>> - Niko Oliveira (@o-nikolas) >>>> - Vincent Beck (@vincbeck) >>>> >>>> Working implementation >>>> ---------------------- >>>> >>>> https://github.com/apache/airflow/pull/72845 >>>> >>>> Incubation commitment >>>> --------------------- >>>> >>>> We commit to: >>>> >>>> - Maintaining the provider and responding to issues within a reasonable >>>> time >>>> - Meeting the incubation health metrics within 6 months >>>> - Participating in quarterly governance updates >>>> >>>> Cheers, >>>> Niko >>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
