Yeah. I would absolutely see this as complementary, not even trying to replace Deferrable Operators. I think we should make it clear in the documentation to not confuse people but the use cases and behaviours there are different. Really, it has one thing in common - both DeferrableOperators and Async support for Python Operators can easily leverage "async Hooks".
For me, the name of DeferrableOperators explains it all (and there is a good reason we did not name it AsyncOperators). The distinction I see: * Deferrable Operators are good, when you have generally synchronous operation that should be Deferred for later (usually much later) * AsyncPythonOperator is good when you want to do a number of small, async operations possibly concurrently, but you do not want to defer those, you simply leverage capabilities of async I/O operations being able to run concurrently (note - not in parallel - but concurrently - using single worker CPU and async I/O non-GIL operations for multiplexing many operations. Those are very, very distinct use cases, I would even say they do not have anything in common (except using async hooks). Also, what AsyncPythonOperator does was essentially possible before - with some boilerplate async loop utilisation code. So really what AIP-98 does is adding a syntactic sugar and hiding the async loop management code, to make it a) easier b) native for airflow with `async def task()` c) more discoverable by our users (providing we will iterate on documentation and examples). In the future (what David mentioned) it opens up for better integration with async task monitoring - for example so that we could see progress of those concurrent tasks in Airflow UI other than looking at logs, and things like more reusable "standard" operators (like IterableOperator). I'd say it's a really foundational change to recognise the "single worker async multiplexing" as native-airflow feature - which will bring some nice things in the future. J. On Sat, Jan 17, 2026 at 9:29 AM Blain David <[email protected]> wrote: > Hello Vikram, > > Thank you for your reply. > > To be clear, no I'm not deprecating deferrable operators, it just depends > on what the operator does: > > 1. If the operator is deferrable because it needs to use an async hook to > retrieve huge payloads from a paginated API, then yes, I would prefer the > async operator over the deferred one, like for example the > MSGraphAsyncOperator or the HttpOperator. > The reason why is what was also explained in the devlist discussion > before, you're literally overloading the triggers (in memory) and the > Airflow metadatabase (triggers table) with huge payloads, > something triggers are not designed for (but you could) as triggers don't > have like an XCom backend which you can easily replace with another one, so > you're stuck with storing the payloads (trigger events) in the Airflow > database table. > > 2. If the operator is deferrable because it needs to do polling to > determine it succeeded or not, then yes, it makes sense, for example I just > started a PR (https://github.com/apache/airflow/pull/60651) > to fix an issue related to polling in the WinRMOperator which blocks the > worker for no reason as it just awaits an answer, it's similar to point one > but here the payload in the triggers is small and so is the execution time. > > So yes, in some cases I would advocate to use the BaseAsyncOperator, but > in other cases not, it all depends on the responsibility of the operator > and what you're doing with. > AIP-98 also opens the door to implement the IterableOperator in the future > which was also discussed mostly with Jarek in the devlist ( > https://lists.apache.org/thread/ztnfsqolow4v1zsv4pkpnxc1fk0hbf2p ) as he > knows what the idea behind there is, but that's also still work in progress. > > On the other hand deferrable operators also have a huge advantage as they > rely on triggers and that is that it allows us to implement the "streaming" > mechanism or the lazy dynamic task mapping expansion I explained in AIP-88 ( > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760511 > ) and which I presented on the Summit in Seattle last year. Once PR #55068 > (https://github.com/apache/airflow/pull/55068/) is merged, I will > continue working on that one as well. > > So yes, it all depends on the use case. > > I hope this makes it a bit more clear to you. > > David > > -----Original Message----- > From: Vikram Koka via dev <[email protected]> > Sent: 16 January 2026 19:58 > To: [email protected] > Cc: Vikram Koka <[email protected]> > Subject: Re: [VOTE] AIP-98: Add async support for PythonOperator in > Airflow 3 > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > stuur deze e-mail als bijlage naar [email protected]<mailto: > [email protected]>. > > Hey David, > > Just read the AIP and posted questions on the Confluence page as well. > > I find this very interesting and am *overall supportive*, but I have > several questions about usage and user / developer guidance. > Specifically around what we should be recommending around what user > situations. I put the following question in the confluence page as well: > > > - You are making a strong case for supporting async within the > PythonOperator pattern over Deferrable Operators. > - What I am missing is when should users be using Deferrable Operators > instead? > - Also, are you advocating deprecating Deferrable Operators entirely? I > am not opposed to it, but definitely something I am curious about your > viewpoint here. > > > Until then, I would vote > -0.5 (binding) > > I am absolutely willing and intend to change my vote, just want to get > questions answered first is all. > > These are questions which any user would have and I therefore believe it > is important to address as part of making and merging this change. > > Vikram > > > > On Fri, Jan 16, 2026 at 9:14 AM Dheeraj Turaga <[email protected]> > wrote: > > > +1 (binding) > > > > Sriraj Dheeraj Turaga > > > > On Fri, Jan 16, 2026 at 9:19 AM Shahar Epstein <[email protected]> > wrote: > > > > > +1 (binding) > > > > > > On Fri, Jan 16, 2026 at 3:39 PM Blain David > > > <[email protected]> > > > wrote: > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > I would like to be calling a vote on this AIP: > > > > > > > > > > > > > > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik > > i.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-98%253A%2BAdd%2Bas > > ync%2Bsupport%2Bfor%2BPythonOperator%2Bin%2BAirflow%2B3&data=05%7C02%7 > > Cdavid.blain%40infrabel.be%7C56d6ae2f0e904b8c30d608de55315684%7Cb82bc3 > > 14ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639041867613554992%7CUnknown%7CTW > > FpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs > > IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=IEe7gFlEP5XQwYbFMM > > 8LkPw%2Bp2Yr0IuBS%2BIp1SSAr1o%3D&reserved=0 > > > > > > > > There was already a discussion in the devlist regarding this > proposal: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > > lists.apache.org%2Fthread%2Fztnfsqolow4v1zsv4pkpnxc1fk0hbf2p&data= > > > > 05%7C02%7Cdavid.blain%40infrabel.be%7C56d6ae2f0e904b8c30d608de5531 > > > > 5684%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C6390418676135806 > > > > 65%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > > &sdata=ccy%2Fec1OCrAyvQorRAEvhuPMDuslWEep9fFNNiT6r7o%3D&reserved=0 > > > > > > > > This AIP is already implemented and merged as a PR: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > > github.com%2Fapache%2Fairflow%2Fpull%2F60268&data=05%7C02%7Cdavid. > > > > blain%40infrabel.be%7C56d6ae2f0e904b8c30d608de55315684%7Cb82bc314a > > > > b8e4d6fb18946f02e1f27f2%7C0%7C0%7C639041867613602603%7CUnknown%7CT > > > > WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4 > > > > zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=c4BiguOx6 > > > > kmqIlWBlNIf6SrO4qN2baTwwFuDBaDmBX8%3D&reserved=0 > > > > > > > > The vote will run for 5 days and last till next thursday, the 22th > > > > of > > Jan > > > > 2026 23:30 GMT. > > > > > > > > > > > > > > > > Everyone is encouraged to vote, although only PMC members and > > Committers' > > > > votes are considered binding. > > > > > > > > > > > > > > > > Please vote accordingly > > > > > > > > > > > > > > > > [ ] +1 Approve > > > > > > > > [ ] +0 no opinion > > > > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > > > > > > > I hereby already vote my +1 binding :) > > > > > > > > > > > > > > > > Regards, > > > > > > > > David aka dabla > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
