Hi Shahar, I have a more fundamental concern here, outside of the performance concern raised earlier.
In the AIP document, you also confirmed that using a "braching task" is an approach for solving this problem today. In my opinion, that is actually a better approach for users. Yes, it adds a task, but the decision-making is explicit in the code written by them and is consolidated within that task. This "branching task" can be observed, logged, and easily audited. And because it is code, it is infinitely flexible for their use case, where certain earlier tasks may be more important or critical than others. As it stands, I agree that trigger rules are needed. They have been part of Airflow for 10 years, yet they remain a source of confusion. I really want to see if we can stop extending them. Today, there are cases where all the prior task information (for DTM tasks) is not necessarily available to the later "branching task." Tamara pointed this out to me in a different discussion. I do think that should be fixed so the branching task can have all the prior task state information. Best regards, Vikram On Tue, May 5, 2026 at 11:21 PM Amogh Desai <[email protected]> wrote: > Hi Shahar, > > > > > > I resonate with Jarek and Ash's performance concerns, trigger rule > evaluation runs on every > task candidate every scheduler cycle, so even a small per-call overhead > will compound really fast at scale. > > > One thing worth trying if you are exploring options (after you manage to > handle the concerns above) -- > Compile the expression into a plain Python function at DAG parse time. I > do not promise that it will make > things better but that way the scheduler just calls a function like it does > today -- no expression parsing in > the hot path at all. Might be worth benchmarking that approach too at some > scale. > > > > Thanks & Regards, > Amogh Desai > > > On Tue, May 5, 2026 at 11:14 PM Shahar Epstein <[email protected]> wrote: > > > Thanks Jarek and Ash for your feedback. > > Your concerns regarding performance are valid - I'll check if it can be > > addressed. > > > > > > Shahar > > > > On Tue, May 5, 2026, 18:13 Ash Berlin-Taylor <[email protected]> wrote: > > > > > +1 to everything Jarek said. On point A, I’d go so far as to ”not > might, > > > but almost certainly will cause performance regression”. > > > > > > Trigger rule evaluation is the hottest of hot paths in the scheduler, > and > > > any change to that area of the code needs to be done with extreme care > > and > > > attention. > > > > > > -ash > > > > > > > On 2 May 2026, at 18:03, Jarek Potiuk <[email protected]> wrote: > > > > > > > > Hi Shahar, > > > > > > > > I am not against this proposal, but I have three concerns regarding > > > points > > > > not fully covered in the AIP: > > > > > > > > a) Performance: Trigger evaluation occurs in the critical scheduler > > loop. > > > > Moving from hard-coded Python criteria to a generalized expression > > model > > > > might introduce a performance hit during the iteration of task > > > candidates. > > > > > > > > b) Complexity and Reasoning: Allowing arbitrary combinations could > lead > > > to > > > > contradicting or nonsensical conditions (e.g., requirements that > exceed > > > the > > > > total number of upstream tasks). This complicates debugging for users > > > > asking "why is my task not running?" and makes it harder for > > maintainers > > > to > > > > discern the original author's intent compared to clear enums like > > > > ALL_FAILED. > > > > > > > > c) Flexibility vs. Intention: The proposed primitives feel more > > > > mathematical than business-logical. If a user doesn't care which > > specific > > > > upstream tasks succeed—only that a certain number do—those tasks are > > > likely > > > > "flavors" of the same work. Such logic might be better handled by > > > extending > > > > "mapped task" success criteria (e.g., a percentage threshold) rather > > than > > > > modifying trigger rules for distinct upstream nodes. > > > > > > > > I'd love to hear your thoughts and those of others on whether this > > > > complexity aligns with what DAG authors actually need. > > > > > > > > Best, > > > > Jarek Potiuk > > > > > > > > On Sat, May 2, 2026 at 11:40 AM Jens Scheffler <[email protected]> > > > wrote: > > > > > > > >> Hi Shahar, > > > >> > > > >> ich think this is a good extension for cases needing this withou > > adding > > > >> too much complexity. Whereas I am not sure whether in our > environment > > we > > > >> have a use case. Would be good to hear from others if there are any. > > > >> Alternatives otherwise in my mind would only be adding more enum > keys > > > >> for other cases (which is not really preferrable) or modelling the > Dag > > > >> with a few EmptyOperators to collect pre-aggregate counts to > leverage > > > >> existing enums... which would in turn create artificial Dag > complexity > > > >> just because of limited enums. > > > >> > > > >> For me OK but would be cool to have other opinions as well. > > > >> > > > >> Jens > > > >> > > > >> On 02.05.26 11:16, Shahar Epstein wrote: > > > >>> Bumping, I'll appreciate your feedback. > > > >>> > > > >>> > > > >>> Shahar > > > >>> > > > >>> On Tue, Apr 21, 2026 at 2:00 PM Shahar Epstein <[email protected]> > > > >> wrote: > > > >>>> Hey everyone, > > > >>>> > > > >>>> I'd like to get your feedback on a new AIP proposal: > > > >>>> > > > >>>> AIP-106: Composable Trigger Rules > > > >>>> https://cwiki.apache.org/confluence/x/64wmGQ > > > >>>> > > > >>>> tl;dr > > > >>>> > > > >>>> This proposal introduces a composable expression model for trigger > > > >> rules, enabling conditions not expressible with the current > enum-based > > > >> system, while fully preserving existing trigger rules for simple > > cases. > > > >>>> For example, "All done, at least two succeeded" can be expressed > as > > > >> TR.expr(done="all", success=">=2"). > > > >>>> > > > >>>> Design Notes/Constraints: > > > >>>> 1. The aggregate form is AND-only by design, to preserve > determinism > > > >> and keep evaluation predictable. > > > >>>> 2. The composable model complements (rather than replaces) the > enum. > > > >> Some trigger rules (e.g. ALL_DONE_SETUP_SUCCESS, ONE_DONE, ALWAYS) > > > remain > > > >> intentionally out of scope, and enums remain the more readable > option > > > for > > > >> simple cases. > > > >>>> > > > >>>> I'd appreciate your feedback on: > > > >>>> > > > >>>> Backward compatibility with existing trigger rules (see Appendix > A) > > > >>>> > > > >>>> Semantics of eager vs complete evaluation > > > >>>> > > > >>>> Scope and limits of the expression model > > > >>>> > > > >>>> Potential edge cases and performance implications > > > >>>> > > > >>>> UI/observability considerations for debugging expressions > > > >>>> > > > >>>> Anything else > > > >>>> > > > >>>> > > > >>>> Thank you! > > > >>>> > > > >>>> > > > >>>> Shahar > > > >>> > --------------------------------------------------------------------- > > > >>> To unsubscribe, e-mail: [email protected] > > > >>> For additional commands, e-mail: [email protected] > > > >>> > > > >> > > > >> > --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: [email protected] > > > >> For additional commands, e-mail: [email protected] > > > >> > > > >> > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
