Hi everyone, I wanted to share a blog post documenting the Hugging Face provider and the work that went into bringing it to the Apache Airflow ecosystem: https://huggingface.co/blog/AINovice2005/airflow-hf-provider <https://huggingface.co/blog/AINovice2005/airflow-hf-provider>
Many thanks to Kaxil Naik for sponsoring and championing the integration and to the wider Airflow community for their support throughout the process. Best regards, Parag Ekbote On Wed, Nov 11, 2026 at 2:52 PM Yeonguk Choo <[email protected]> wrote: > +1 as well > Thanks for the contribution 🙌 > > Best Regards, > Yeonguk > > 2026년 11월 11일 (수) 오후 3:40, Amogh Desai via dev <[email protected]>님이 > 작성: > > > Nice! Thanks for the contribution, Parag! > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Wed, Nov 11, 2026 at 6:44 AM Kaxil Naik <[email protected]> wrote: > > > > > We are going to merge this once CI goes green: > > > https://github.com/apache/airflow/pull/76317 > > > > > > On Tue, 3 Nov 2026 at 15:26, Blain David <[email protected]> > > wrote: > > > > > > > +1 as well, don't forget about the async counterparts as well in the > > hook > > > > ;-) > > > > > > > > General (Internal Property) > > > > ________________________________ > > > > From: Parag Ekbote <[email protected]> > > > > Sent: Tuesday, November 3, 2026 02:12 > > > > To: [email protected] <[email protected]> > > > > Subject: [DISCUSS] New Community Provider: Hugging Face > > > > > > > > [You don't often get email from [email protected]. Learn why > > this > > > > is important at https://aka.ms/LearnAboutSenderIdentification ] > > > > > > > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en > deze > > > > niet vertrouwt, klik niet op een link of open geen bijlages. Bij > > twijfel, > > > > stuur deze e-mail als bijlage naar [email protected]<mailto: > > > > [email protected]>. > > > > > > > > Hi Airflow community, > > > > > > > > I'd like to propose introducing a new community provider for Hugging > > > Face. > > > > Integration overview > > > > > > > > Hugging Face < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuggingface.co%2F&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896862729%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ict4zTMBmiCURE63cK9QHoTYw185JlnNyRkZIqCAOkU%3D&reserved=0 > > > > <https://huggingface.co/>> is an open-source AI platform and > > > > community that provides tools and infrastructure for building, > sharing > > > and > > > > deploying machine learning models, datasets and applications. Hugging > > > Face > > > > Datasets is widely used in data engineering and machine learning > > > workflows, > > > > and a first-class Airflow provider would make dataset orchestration, > > > > transformation, and publishing simpler. > > > > > > > > The provider initially focuses on Hugging Face Datasets, exposing a > > > > HuggingFaceHook and native operators for loading datasets with > revision > > > > pinning, applying user-defined transformations and publishing > datasets > > to > > > > the Hugging Face Hub. > > > > > > > > Future iterations may extend the provider to Hugging Face Models and > > > > Inference Endpoints. > > > > System tests > > > > > > > > The provider integrates with the live Hugging Face Hub. System tests > > are > > > > included in the PR and exercise dataset loading, transformation and > > > > publishing workflows using the airflow-hf > > > > < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuggingface.co%2Fairflow-hf&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896876633%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UjmjHX5ugIgjLwZJXRq4kBxoHLbWr2%2BVTPMJyBuWvIc%3D&reserved=0 > > > > <https://huggingface.co/airflow-hf>> Hugging Face organization. > > > > Proposed stewards > > > > > > > > Parag Ekbote, @ParagEkbote < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FParagEkbote&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896883162%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=a%2F9QlVqphMckSJyeBKyhkJfNyHTfIXct6z9zAyu5sWo%3D&reserved=0 > > > > <https://github.com/ParagEkbote>>, < > > > > [email protected]> > > > > > > > > Quentin Lhoest, @lhoestq < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flhoestq&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896889012%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=pyCnGTAr0x4%2FD3F7H7Utnm08g%2B1IRxAN0s29n2bMgOM%3D&reserved=0 > > > > <https://github.com/lhoestq>>, < > > > > [email protected]> > > > > Sponsoring committer(s) > > > > > > > > Kaxil Naik, @kaxil < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkaxil&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896894587%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=g3fovALx3UMCnlhP42ZBHPcQz1nZn7gcAsLWcNCIx3o%3D&reserved=0 > > > > <https://github.com/kaxil>>, <[email protected]> > > > > Working implementation > > > > < > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F76317&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896900255%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rUt3Z2gABvZjlcfaM9NInLl4PPUF6C4WslBLzykoVro%3D&reserved=0 > > > > <https://github.com/apache/airflow/pull/76317>> > > > > > > > > Incubation commitment > > > > > > > > We commit to: > > > > > > > > - > > > > > > > > Maintaining the provider and responding to issues within a > > reasonable > > > > time. > > > > - > > > > > > > > Meeting the incubation health metrics within six months. > > > > - > > > > > > > > Participating in quarterly governance updates. > > > > > > > > Best regards, > > > > > > > > Parag Ekbote > > > > > > > > > >
