Thanks Julian, much appreciated! On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev < [email protected]> wrote:
> I wrote our original DuckDB provider at Astronomer, would be more than > happy to deprecate in favor of yours! Definitely would love to see > something more first-class and better supported. > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <[email protected]> wrote: > > > > Glad to see the traction, thanks for weighing in everyone! > > > >> Is this based on the Astronomer one? > > https://github.com/astronomer/airflow-provider-duckdb > > > > No code was based on that provider. It hasn't been updated in 3 years and > > it focused on a minimal set of DuckDB features. There wasn't much to > reuse > > there. > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <[email protected]> wrote: > > > >> +10 > >> > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <[email protected]> wrote: > >> > >>> Sounds good! > >>> > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <[email protected]> wrote: > >>> > >>>> Hi folks, > >>>> > >>>> I'd like to propose adding a new community provider for DuckDB. > >>>> > >>>> Integration overview: > >>>> --------------------- > >>>> > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL > >>>> providers there is no server to connect to: queries run inside the > task > >>>> process, which makes it a cheap way to run an analytical transform > that > >>>> doesn't justify standing up a cluster. It reads and writes Parquet, > CSV > >>> and > >>>> JSON directly from object storage, so it fits naturally between an > >>> extract > >>>> and a load step in a > >>>> Dag. > >>>> > >>>> It also fits Airflow's existing shape well: The proposed DuckDB > >> provider > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather than > >>> needing > >>>> new machinery, and it fits in the data-processing/ETL space. DuckDB > >> Labs > >>>> was acquired by Amazon but DuckDB remains MIT licensed and controlled > >> by > >>>> the DuckDB Foundation. It is usable with no account, no signup and no > >>>> hosted service. > >>>> > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it works > with > >>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus > >> a > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The > connection > >>> is > >>>> optional: with none configured the hook opens an in-memory database. > >>>> > >>>> Provider specific integrations would live in their relevant provider > >>>> package (e.g. for AWS, an AwsDuckDBHook in > >>> apache-airflow-providers-amazon > >>>> for credential management). > >>>> > >>>> System tests: > >>>> ------------- > >>>> > >>>> DuckDB runs in-process with no live external service, so system tests > >> are > >>>> not > >>>> required for this provider; the unit tests exercise a real in-memory > >>>> DuckDB. > >>>> An AWS-specific system test could be added in the Amazon provider > >>> package, > >>>> if folks think that's required. > >>>> > >>>> Proposed stewards: > >>>> ------------------ > >>>> > >>>> Myself and the AWS team plan to sponsor and steward this package, you > >>> know > >>>> us and our track record well :) > >>>> > >>>> - Niko Oliveira (@o-nikolas) > >>>> - Vincent Beck (@vincbeck) > >>>> > >>>> Working implementation > >>>> ---------------------- > >>>> > >>>> https://github.com/apache/airflow/pull/72845 > >>>> > >>>> Incubation commitment > >>>> --------------------- > >>>> > >>>> We commit to: > >>>> > >>>> - Maintaining the provider and responding to issues within a > reasonable > >>>> time > >>>> - Meeting the incubation health metrics within 6 months > >>>> - Participating in quarterly governance updates > >>>> > >>>> Cheers, > >>>> Niko > >>>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
