Yeah dynamic was being over-used and doesn't say much so agreed. On Fri, 20 Nov 2026 at 16:34, Ash Berlin-Taylor via dev < [email protected]> wrote:
> (Yes, though we “soft-renamed” dynamic mapped tasks to just “mapped tasks” > in 3.4.) > > > On 20 Nov 2026, at 16:21, Brent Bovenzi via dev <[email protected]> > wrote: > > > > Planner Tasks sounds good. > > > > We already described mapped tasks as "dynamic" so using that word again > would only confuse people. > > > > On 2026/11/20 12:31:44 Amogh Desai via dev wrote: > >> Thanks for proposing this. I like the idea, and catalog + validator > seems > >> like a reasonable > >> way to keep the LLM planner bounded. > >> > >> I left a few comments on the AIP. > >> > >> Thanks & Regards, > >> Amogh Desai > >> > >> > >> On Thu, Nov 19, 2026 at 12:03 AM Kaxil Naik <[email protected]> > wrote: > >> > >>> I do like Planner Tasks > >>> > >>> On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev < > >>> [email protected]> wrote: > >>> > >>>> For the name, I'm find myself defaulting to calling the proposed > >>>> feature Planner Tasks, especially when discussing it alongside Mapped > >>>> Tasks and Task Loops. Plan mode doesn't fit as well with the other > >>>> concepts we have. > >>>> > >>>> On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> > wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> Picking this thread back up. > >>>>> > >>>>> Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113 > >>>>> simpler, so Ash and I reworked it on top of them: selected tasks are > >>> now > >>>>> created the way loop iterations are, and the plan uses the YAML Dag > >>>> schema. > >>>>> The page is updated: > >>>>> > >>>>> > >>>> > >>> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > >>>>> > >>>>> Thanks Przemysław for the earlier comments. > >>>>> > >>>>> Why now: the Common AI provider reached 1.0 in October, and more of > the > >>>>> Dags people write have a model deciding what happens next. Today an > LLM > >>>> can > >>>>> pick which downstream tasks run (@task.llm_branch), but like any > branch > >>>> it > >>>>> can only select from tasks already wired in the Dag. It can't decide > >>> the > >>>>> order or the dependencies between them, so people either put the > whole > >>>>> agent loop inside one opaque task, losing per-step retries, logs and > >>>>> approvals, or generate Dag files at parse time. A few shapes we keep > >>>> seeing: > >>>>> > >>>>> - Incident triage: an agent reads an alert and picks diagnosis steps, > >>>>> where which probe runs next depends on what the first one found, > ending > >>>> in > >>>>> a fix or a page to a human. > >>>>> - Support tickets: classify, then run some of validate, refund, > >>>>> enrich, escalate and notify, in an order that depends on the ticket, > >>> with > >>>>> an approval task before anything that moves money. > >>>>> - Data-quality failures: quarantine, backfill, re-validate and > notify, > >>>>> chosen per failure. > >>>>> - Selective builds, with no LLM involved: an impact analysis decides > >>>>> which models to rebuild and the edges between exactly those models. > >>>>> > >>>>> In each case the set of allowed actions is fixed and reviewed in the > >>> Dag > >>>>> file; only the selection and wiring change per run. Each action > stays a > >>>>> real task with its own retries, logs and pool, and the catalog plus a > >>>>> validator keep the model inside what the author approved. > >>>>> > >>>>> On the name: neither of us likes "Dynamic Task Groups". It describes > >>> the > >>>>> mechanism, not what you get, which is building part of your Dag at > run > >>>> time. > >>>>> We're keeping it for the AIP so the discussion doesn't get stuck on > it, > >>>> but > >>>>> expect it to change before it ships. > >>>>> > >>>>> Two that came to my mind: "Plan Mode", as in a task group in plan > mode, > >>>> the > >>>>> way coding agents plan before they act, and "Runtime > >>> subgraphs/subdags", > >>>>> which pairs with Task Loops and the @planner decorator. Suggestions > >>>> welcome. > >>>>> > >>>>> What changed since the first draft: > >>>>> > >>>>> - Authoring is a @planner task inside a task group, plus a catalog of > >>>> what > >>>>> it > >>>>> may pick from (@triage.catalog, or catalog= / extra_catalog_items=). > >>>> The > >>>>> TaskFlow and context-manager forms are side by side on the page and > >>> are > >>>>> nearly identical. Catalog items can be operator classes or > >>> import-path > >>>>> strings, so heavy operators don't have to be imported at parse time. > >>>>> > >>>>> - The plan uses the same schema as YAML Dags (tasks, uses, with, > >>> needs), > >>>> so > >>>>> Airflow has one vocabulary for declaring a graph of tasks. > Templating > >>>> is > >>>>> explicit, as in YAML Dags: a value is rendered only when marked with > >>>> $t. > >>>>> > >>>>> - No subclassing. An LLM planner would itself be a planner, shipped > by > >>> a > >>>>> provider, with a validator on top. An @task.agent returning the plan > >>> as > >>>>> structured output should also work with no extra machinery. > >>>>> > >>>>> - Selected entries are real task instances, not map indices of one > >>> mapped > >>>>> task. The planner submits the plan through the Execution API while > it > >>>> is > >>>>> still running, and the API server creates the task instances, which > >>> is > >>>> the > >>>>> approach Task Loops took. The plan is stored against the region, so > >>>> after > >>>>> a > >>>>> clear and re-plan, the earlier plan and the task instances that ran > >>>> under > >>>>> it > >>>>> are still there to look at. > >>>>> > >>>>> - At parse time the group serializes as two nodes: the planner and a > >>>> merge > >>>>> node. The merge is always an EmptyOperator, and anything downstream > >>> of > >>>> the > >>>>> group attaches to it. > >>>>> > >>>>> - Catalog membership is by exact class, so a subclass of an approved > >>>>> operator > >>>>> is rejected. That can be loosened later; tightening it later would > be > >>>> much > >>>>> harder. > >>>>> > >>>>> Comments on the page or here are both fine. > >>>>> > >>>>> Thanks, > >>>>> Kaxil > >>>>> > >>>>> On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Thanks for the update Kaxil. Looks much better. I've added a couple > >>> of > >>>> new > >>>>>> comments. > >>>>>> > >>>>>> On 2026/08/10 21:26:24 Kaxil Naik wrote: > >>>>>>> Thanks for your feedback Przemysław, I updated the AIP > >>>>>>> > >>>>>>> On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected] > >>>> > >>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Kaxil, > >>>>>>>> > >>>>>>>> I've added some comments to the AIP. In general, the idea looks > >>>>>>>> interesting. > >>>>>>>> > >>>>>>>> Could we possibly rearrange the sections a bit? During the > >>> reading, > >>>>>>>> sometimes one thing was stated when the reader does not know a > >>>>>> particular > >>>>>>>> part of the change (e.g. merge policy - I added a comment next to > >>>> it). > >>>>>>>> Also, there are a lot of new definitions - maybe it would be good > >>>> to > >>>>>> create > >>>>>>>> a section which will gather them all and shortly explain? > >>>>>>>> > >>>>>>>> On 2026/07/30 10:16:20 Kaxil Naik wrote: > >>>>>>>>> Hi all, > >>>>>>>>> > >>>>>>>>> I would like to start a discussion on AIP-113: Dynamic Task > >>>> Groups. > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > >>>>>>>>> > >>>>>>>>> tldr: > >>>>>>>>> Dynamic Task Mapping made cardinality dynamic: run the same > >>> task > >>>>>> over a > >>>>>>>>> collection discovered at run time. What is still static is the > >>>> shape, > >>>>>>>>> meaning which tasks > >>>>>>>>> run in a region and how they depend on each other. > >>>>>>>>> > >>>>>>>>> Branching gets you half of that already, and I want to be > >>> precise > >>>>>> about > >>>>>>>>> which half, because > >>>>>>>>> I had this wrong in an earlier draft. `choose_branch` can > >>> return > >>>> a > >>>>>> list, > >>>>>>>> so > >>>>>>>>> a branch already > >>>>>>>>> selects an arbitrary subset of its declared successors. What a > >>>> branch > >>>>>>>>> cannot do is decide > >>>>>>>>> the dependency edges *among* the tasks it selected. Selection > >>> is > >>>>>> solved; > >>>>>>>>> ordering and wiring are not. That is the > >>>>>>>>> gap this AIP is aimed at. > >>>>>>>>> > >>>>>>>>> The proposal is a Task Group whose members and internal edges > >>> are > >>>>>> decided > >>>>>>>>> at run time from a fixed catalog the author declares. A > >>> generator > >>>>>> task > >>>>>>>>> emits a plan, the group materializes > >>>>>>>>> exactly the selected entries as real task instances, and the > >>>> region > >>>>>>>>> converges on a > >>>>>>>>> synthesized merge node. The catalog is what serializes, so two > >>>> runs > >>>>>> of > >>>>>>>>> different shapes share > >>>>>>>>> one DAG version, and an entry the plan did not select never > >>>> becomes a > >>>>>>>> task > >>>>>>>>> instance. The > >>>>>>>>> planner can be a control table, an impact analysis, a triage > >>>> rule, > >>>>>> or an > >>>>>>>>> LLM. > >>>>>>>>> > >>>>>>>>> --- > >>>>>>>>> > >>>>>>>>> Would love to hear your thoughts & get feedback. Please comment > >>>> on > >>>>>> the > >>>>>>>> AIP > >>>>>>>>> doc itself. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Kaxil > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>> --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >>>> > >>> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
