My non-binding votes: Option A: -1 as it changes the architecture and direction of Airflow with hard to predict implications on scheduler/triggerer/workers e.g. in terms of scheduling decisions and picking up the workloads when enabled
Option B: 0, but if chosen, I would not aim it for 3.2 as it would require excessive testing before it and we are close to 3.2 release Option C: +1 > ________________________________ >From: Blain David <[email protected]> >Sent: 26 March 2026 15:18 >To: [email protected] <[email protected]> >Cc: Vikram Koka <[email protected]> >Subject: Re: [VOTE] Improve problems of additional latency for tasks returning >from Triggerer to Worker > >I'm not fully following the different options, hence why I will ask this >question here because I probably missing >something or I don't fully >understand it, but I understand the problem Jens trying to solve which is in >some sort >avoiding scheduler waiting times when using deferred operators once >the trigger has finished but in the meantime lot's >of new deferred operators >keep getting scheduled. > >So for Option A, it would mean that we would just set the >state to QUEUED >directly instead of SCHEDULED, which >means the task wouldn't be picked up by >the scheduler >anymore and the next_method specified in the trigger would >get >directly executed on the worker without any supervision >of the scheduler, >correct? > >For Option B, I would be against it because this probably >needs more >refactoring on trigger side first so it's loosely >coupled, like the work that >was done with Task SDK >(meaning no direct db access for example), which >requires >more work, but if that would be the pre-requisite and still >doable >in short notice, then I would be in favor. > >For Option C I have no strong opinion, all depends how >quickly this can be >implemented and tested as 3.2.0 is >coming close. > >Kind regards, >David > > >________________________________ >From: Vikram Koka via dev <[email protected]> >Sent: Thursday, March 26, 2026 14:25 >To: [email protected] <[email protected]> >Cc: Vikram Koka <[email protected]> >Subject: Re: [VOTE] Improve problems of additional latency >for tasks >returning from Triggerer to Worker > >EXTERNAL MAIL: Indien je de afzender van deze e-mail niet >kent en deze niet >vertrouwt, klik niet op een link of open >geen bijlages. Bij twijfel, stuur >deze e-mail als bijlage naar >[email protected]<mailto:[email protected]>. > >Very similar to Jarek and Kaxil's analysis: > >Option A: -1 >This is architecturally inconsistent with the direction we have >been moving >towards over the last couple of years. > >Option B: -0.1 >Architecturally better, but the core change has significant >ripple effects, >requiring significant test coverage expansion. >I just don't think we can achieve this quickly and maintain >stability at >this late stage. > >Option C: +1 >An AIP even a mini-AIP would give us some time to work >through the details. >I understand the strong desire for urgency, >but I believe it can be addressed by having Airflow 3.3.0 >follow quickly. >It is currently targeted for June and I believe that >we should make every effort to hit that and have it be early >in June, >before people starting vanishing for summer holidays. > >Best regards, >Vikram > >On Thu, Mar 26, 2026 at 4:59 AM Jarek Potiuk ><[email protected]> wrote: > > Very similar with Kaxil's assessment: > > - Option A: -1: I think it is against the direction of Airflow > architectural changes we work for about 1.5 years. While those aren't > completed, it would basically invalidate that work. > - Option B: -0.1 (for the reasons Kaxil mentioned > - Option C: +1 - and I think this should be addressed quickly with high > priority. We already discussed that 3.3 should be released **way faster** > than 3.2 and I think this one is yet another reason for itV > > J. > > On Thu, Mar 26, 2026 at 4:28 AM Kaxil Naik <[email protected]> wrote: > > > (details cut in last email) > > > > Hi Jens, > > > > My vote: > > > > - Option A: -0.5 > > - Option B: 0 > > - Option C: +1 > > > > Reasoning: > > > > Option A has architectural issues. The triggerer today is a client > > component running user code; it runs user triggers and reports events > back. > > PR #63489 turns it into a mini-scheduler by having it instantiate > executor > > instances > > (init_executors()), generate JWT tokens, and dispatch workloads directly > to > > Celery brokers. That's scheduler territory. > > > > This moves in the opposite direction from AIP-92, which aims to isolate > the > > triggerer further from core services (removing even direct DB access in > > favor of API-only communication). If we're planning to restrict > > what the triggerer can talk to, giving it executor ownership and broker > > dispatch now creates tech debt we'd need to undo. > > > > Option B is a better fit architecturally, but it still adds new surface > > area (XCom push from triggerer, changes to TriggerEvent, KPO behavior > > changes). For a 3.2 release where we should be focused on stability, > > introducing new triggerer capabilities increases what we need to test and > > validate. I'd rather not risk 3.2 stability for this. > > > > Option C is my preference. This is a real problem and worth solving > > properly, but 3.2 should be a stability-focused release. An AIP would > give > > us time to design something that aligns with AIP-92's direction rather > > than working against it. Target 3.3. > > > > While I understand the frustration for waiting for several months, but > any > > new thing added is a new surface to test, validate and potential for > > introducing new bugs. > > > > Thanks, > > Kaxil > > > > On Thu, 26 Mar 2026 at 03:22, Kaxil Naik <[email protected]> wrote: > > > > > Option C: 0 > > > > > > On Wed, 25 Mar 2026 at 23:04, Jens Scheffler <[email protected]> > > wrote: > > > > > >> Hi Airflow Devs, > > >> > > >> Following-up on the discussions in > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F6znvd5rtqnxt5r4hys7qn64j5mflr9g1&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456829672%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kXiLrT39Hi8fFKeUrQWQDgQ6mvh6EMe6UxuYXaSGbHE%3D&reserved=0<https://lists.apache.org/thread/6znvd5rtqnxt5r4hys7qn64j5mflr9g1> > > >> I'd > > >> like to cast a VOTE - even if the discussion is fresh.. but my > aim/call > > >> would be to improve the problem in the core with Airflow 3.2.0 not to > > >> have users stuck in this situation for additional multiple months (as > it > > >> would be a new feature, not a fix most probably not applicable to a > fix > > >> release) > > >> > > >> As we discussed two options we can not have a simple +/- Vote, > therefore > > >> I call for a VOTE with options, please rate from 0 to +1 for the > option > > >> and only -1 on an option to call for veto. > > >> > > >> * OPTION A) Integrate feature to directly queue tasks from Triggerer > > >> as of PR > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F63489&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456865852%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lGZturEQPtN9Lpn0qlRJlb4JBTwyiveydnxQ06E%2Fno8%3D&reserved=0<https://github.com/apache/airflow/pull/63489> > > >> in Airflow > > >> 3.2.0 > > >> * OPTION B) Extend Triggerer to support XCom, change KPO to end from > > >> Triggerer as of PR > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64068&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456889125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ODnVOUcPxzfeLSB7o80%2B7knZCWvXduBJqZww%2FFVrfJA%3D&reserved=0<https://github.com/apache/airflow/pull/64068> > > >> (Core) with Target Airflow 3.2.0 and > > >> > > >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64069&data=05%7C02%7Cdavid.blain%40infrabel.be%7C6fa353dde3a7468e57bf08de8b3b69b1%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639101284456916153%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fyYrUduDAw2LYT0FM4tQugLfedLthAmGps0BD%2BDuyeg%3D&reserved=0<https://github.com/apache/airflow/pull/64069> > > >> (KPO Adjustment) > > >> * OPTION C) Do not immediately change this for Airflow 3.2.0, > further > > >> discuss alternatives, e.g. raise an AIP > > >> > > >> For details of the soluitions please check the discussion thread > (might > > >> need a bit of reading!) and the linked PRs > > >> > > >> The VOTE is open for the next 72 hours to be done before RC1, until > > >> 2026-03-29 23:59 CET - I will sum-up all vores except any potential > veto > > >> and will report the summary. > > >> > > >> My Vote is: > > >> > > >> * Option A: +0.8 > > >> * Option B: +1 > > >> * Option C: 0 > > >> > > >> Thanks, > > >> > > >> Jens > > >> > > > > > >
