We are going to merge this once CI goes green: https://github.com/apache/airflow/pull/76317
On Tue, 3 Nov 2026 at 15:26, Blain David <[email protected]> wrote: > +1 as well, don't forget about the async counterparts as well in the hook > ;-) > > General (Internal Property) > ________________________________ > From: Parag Ekbote <[email protected]> > Sent: Tuesday, November 3, 2026 02:12 > To: [email protected] <[email protected]> > Subject: [DISCUSS] New Community Provider: Hugging Face > > [You don't often get email from [email protected]. Learn why this > is important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > stuur deze e-mail als bijlage naar [email protected]<mailto: > [email protected]>. > > Hi Airflow community, > > I'd like to propose introducing a new community provider for Hugging Face. > Integration overview > > Hugging Face < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuggingface.co%2F&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896862729%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ict4zTMBmiCURE63cK9QHoTYw185JlnNyRkZIqCAOkU%3D&reserved=0 > <https://huggingface.co/>> is an open-source AI platform and > community that provides tools and infrastructure for building, sharing and > deploying machine learning models, datasets and applications. Hugging Face > Datasets is widely used in data engineering and machine learning workflows, > and a first-class Airflow provider would make dataset orchestration, > transformation, and publishing simpler. > > The provider initially focuses on Hugging Face Datasets, exposing a > HuggingFaceHook and native operators for loading datasets with revision > pinning, applying user-defined transformations and publishing datasets to > the Hugging Face Hub. > > Future iterations may extend the provider to Hugging Face Models and > Inference Endpoints. > System tests > > The provider integrates with the live Hugging Face Hub. System tests are > included in the PR and exercise dataset loading, transformation and > publishing workflows using the airflow-hf > < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhuggingface.co%2Fairflow-hf&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896876633%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UjmjHX5ugIgjLwZJXRq4kBxoHLbWr2%2BVTPMJyBuWvIc%3D&reserved=0 > <https://huggingface.co/airflow-hf>> Hugging Face organization. > Proposed stewards > > Parag Ekbote, @ParagEkbote < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FParagEkbote&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896883162%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=a%2F9QlVqphMckSJyeBKyhkJfNyHTfIXct6z9zAyu5sWo%3D&reserved=0 > <https://github.com/ParagEkbote>>, < > [email protected]> > > Quentin Lhoest, @lhoestq < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flhoestq&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896889012%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=pyCnGTAr0x4%2FD3F7H7Utnm08g%2B1IRxAN0s29n2bMgOM%3D&reserved=0 > <https://github.com/lhoestq>>, < > [email protected]> > Sponsoring committer(s) > > Kaxil Naik, @kaxil < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkaxil&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896894587%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=g3fovALx3UMCnlhP42ZBHPcQz1nZn7gcAsLWcNCIx3o%3D&reserved=0 > <https://github.com/kaxil>>, <[email protected]> > Working implementation > < > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F76317&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cd58f4ed265d74587f72b08df39479e2f%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639292651896900255%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rUt3Z2gABvZjlcfaM9NInLl4PPUF6C4WslBLzykoVro%3D&reserved=0 > <https://github.com/apache/airflow/pull/76317>> > > Incubation commitment > > We commit to: > > - > > Maintaining the provider and responding to issues within a reasonable > time. > - > > Meeting the incubation health metrics within six months. > - > > Participating in quarterly governance updates. > > Best regards, > > Parag Ekbote >
