+1 On 2026/09/10 23:54:59 Vikram Koka via dev wrote: > +1 > > On Thu, Sep 10, 2026 at 3:43 PM Kaxil Naik <[email protected]> wrote: > > > +1 > > > > On Thu, 10 Sept 2026 at 18:04, Shubham Raj <[email protected]> wrote: > > > > > Sounds good, +1 > > > > > > On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <[email protected]> > > > wrote: > > > > > > > Arguably, long overdue. > > > > > > > > +1 from me. I will review the PR and offer my thoughts if that is ok > > > with > > > > you. > > > > > > > > Thanks, > > > > Sameer. > > > > > > > > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <[email protected]> > > wrote: > > > > > > > > > Sounds great, let's bring it in! > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <[email protected]> > > > > wrote: > > > > > > > > > > > Thanks Julian, much appreciated! > > > > > > > > > > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > I wrote our original DuckDB provider at Astronomer, would be more > > > > than > > > > > > > happy to deprecate in favor of yours! Definitely would love to > > see > > > > > > > something more first-class and better supported. > > > > > > > > > > > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Glad to see the traction, thanks for weighing in everyone! > > > > > > > > > > > > > > > >> Is this based on the Astronomer one? > > > > > > > > https://github.com/astronomer/airflow-provider-duckdb > > > > > > > > > > > > > > > > No code was based on that provider. It hasn't been updated in 3 > > > > years > > > > > > and > > > > > > > > it focused on a minimal set of DuckDB features. There wasn't > > much > > > > to > > > > > > > reuse > > > > > > > > there. > > > > > > > > > > > > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > >> +10 > > > > > > > >> > > > > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <[email protected]> > > > wrote: > > > > > > > >> > > > > > > > >>> Sounds good! > > > > > > > >>> > > > > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, < > > [email protected] > > > > > > > > > > wrote: > > > > > > > >>> > > > > > > > >>>> Hi folks, > > > > > > > >>>> > > > > > > > >>>> I'd like to propose adding a new community provider for > > > DuckDB. > > > > > > > >>>> > > > > > > > >>>> Integration overview: > > > > > > > >>>> --------------------- > > > > > > > >>>> > > > > > > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike > > > most > > > > > SQL > > > > > > > >>>> providers there is no server to connect to: queries run > > inside > > > > the > > > > > > > task > > > > > > > >>>> process, which makes it a cheap way to run an analytical > > > > transform > > > > > > > that > > > > > > > >>>> doesn't justify standing up a cluster. It reads and writes > > > > > Parquet, > > > > > > > CSV > > > > > > > >>> and > > > > > > > >>>> JSON directly from object storage, so it fits naturally > > > between > > > > an > > > > > > > >>> extract > > > > > > > >>>> and a load step in a > > > > > > > >>>> Dag. > > > > > > > >>>> > > > > > > > >>>> It also fits Airflow's existing shape well: The proposed > > > DuckDB > > > > > > > >> provider > > > > > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem > > rather > > > > > than > > > > > > > >>> needing > > > > > > > >>>> new machinery, and it fits in the data-processing/ETL space. > > > > > DuckDB > > > > > > > >> Labs > > > > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and > > > > > > controlled > > > > > > > >> by > > > > > > > >>>> the DuckDB Foundation. It is usable with no account, no > > signup > > > > and > > > > > > no > > > > > > > >>>> hosted service. > > > > > > > >>>> > > > > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it > > > > works > > > > > > > with > > > > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql > > > > ecosystem, > > > > > > plus > > > > > > > >> a > > > > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The > > > > > > > connection > > > > > > > >>> is > > > > > > > >>>> optional: with none configured the hook opens an in-memory > > > > > database. > > > > > > > >>>> > > > > > > > >>>> Provider specific integrations would live in their relevant > > > > > provider > > > > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in > > > > > > > >>> apache-airflow-providers-amazon > > > > > > > >>>> for credential management). > > > > > > > >>>> > > > > > > > >>>> System tests: > > > > > > > >>>> ------------- > > > > > > > >>>> > > > > > > > >>>> DuckDB runs in-process with no live external service, so > > > system > > > > > > tests > > > > > > > >> are > > > > > > > >>>> not > > > > > > > >>>> required for this provider; the unit tests exercise a real > > > > > in-memory > > > > > > > >>>> DuckDB. > > > > > > > >>>> An AWS-specific system test could be added in the Amazon > > > > provider > > > > > > > >>> package, > > > > > > > >>>> if folks think that's required. > > > > > > > >>>> > > > > > > > >>>> Proposed stewards: > > > > > > > >>>> ------------------ > > > > > > > >>>> > > > > > > > >>>> Myself and the AWS team plan to sponsor and steward this > > > > package, > > > > > > you > > > > > > > >>> know > > > > > > > >>>> us and our track record well :) > > > > > > > >>>> > > > > > > > >>>> - Niko Oliveira (@o-nikolas) > > > > > > > >>>> - Vincent Beck (@vincbeck) > > > > > > > >>>> > > > > > > > >>>> Working implementation > > > > > > > >>>> ---------------------- > > > > > > > >>>> > > > > > > > >>>> https://github.com/apache/airflow/pull/72845 > > > > > > > >>>> > > > > > > > >>>> Incubation commitment > > > > > > > >>>> --------------------- > > > > > > > >>>> > > > > > > > >>>> We commit to: > > > > > > > >>>> > > > > > > > >>>> - Maintaining the provider and responding to issues within a > > > > > > > reasonable > > > > > > > >>>> time > > > > > > > >>>> - Meeting the incubation health metrics within 6 months > > > > > > > >>>> - Participating in quarterly governance updates > > > > > > > >>>> > > > > > > > >>>> Cheers, > > > > > > > >>>> Niko > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
