Planner Tasks sounds good. We already described mapped tasks as "dynamic" so using that word again would only confuse people.
On 2026/11/20 12:31:44 Amogh Desai via dev wrote: > Thanks for proposing this. I like the idea, and catalog + validator seems > like a reasonable > way to keep the LLM planner bounded. > > I left a few comments on the AIP. > > Thanks & Regards, > Amogh Desai > > > On Thu, Nov 19, 2026 at 12:03 AM Kaxil Naik <[email protected]> wrote: > > > I do like Planner Tasks > > > > On Wed, 18 Nov 2026 at 17:56, Constance Martineau via dev < > > [email protected]> wrote: > > > > > For the name, I'm find myself defaulting to calling the proposed > > > feature Planner Tasks, especially when discussing it alongside Mapped > > > Tasks and Task Loops. Plan mode doesn't fit as well with the other > > > concepts we have. > > > > > > On Wed, Nov 18, 2026 at 12:49 PM Kaxil Naik <[email protected]> wrote: > > > > > > > > Hi all, > > > > > > > > Picking this thread back up. > > > > > > > > Airflow 3.4 shipped Task Loops and YAML Dags, and both made AIP-113 > > > > simpler, so Ash and I reworked it on top of them: selected tasks are > > now > > > > created the way loop iterations are, and the plan uses the YAML Dag > > > schema. > > > > The page is updated: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > > > > > > > > Thanks Przemysław for the earlier comments. > > > > > > > > Why now: the Common AI provider reached 1.0 in October, and more of the > > > > Dags people write have a model deciding what happens next. Today an LLM > > > can > > > > pick which downstream tasks run (@task.llm_branch), but like any branch > > > it > > > > can only select from tasks already wired in the Dag. It can't decide > > the > > > > order or the dependencies between them, so people either put the whole > > > > agent loop inside one opaque task, losing per-step retries, logs and > > > > approvals, or generate Dag files at parse time. A few shapes we keep > > > seeing: > > > > > > > > - Incident triage: an agent reads an alert and picks diagnosis steps, > > > > where which probe runs next depends on what the first one found, ending > > > in > > > > a fix or a page to a human. > > > > - Support tickets: classify, then run some of validate, refund, > > > > enrich, escalate and notify, in an order that depends on the ticket, > > with > > > > an approval task before anything that moves money. > > > > - Data-quality failures: quarantine, backfill, re-validate and notify, > > > > chosen per failure. > > > > - Selective builds, with no LLM involved: an impact analysis decides > > > > which models to rebuild and the edges between exactly those models. > > > > > > > > In each case the set of allowed actions is fixed and reviewed in the > > Dag > > > > file; only the selection and wiring change per run. Each action stays a > > > > real task with its own retries, logs and pool, and the catalog plus a > > > > validator keep the model inside what the author approved. > > > > > > > > On the name: neither of us likes "Dynamic Task Groups". It describes > > the > > > > mechanism, not what you get, which is building part of your Dag at run > > > time. > > > > We're keeping it for the AIP so the discussion doesn't get stuck on it, > > > but > > > > expect it to change before it ships. > > > > > > > > Two that came to my mind: "Plan Mode", as in a task group in plan mode, > > > the > > > > way coding agents plan before they act, and "Runtime > > subgraphs/subdags", > > > > which pairs with Task Loops and the @planner decorator. Suggestions > > > welcome. > > > > > > > > What changed since the first draft: > > > > > > > > - Authoring is a @planner task inside a task group, plus a catalog of > > > what > > > > it > > > > may pick from (@triage.catalog, or catalog= / extra_catalog_items=). > > > The > > > > TaskFlow and context-manager forms are side by side on the page and > > are > > > > nearly identical. Catalog items can be operator classes or > > import-path > > > > strings, so heavy operators don't have to be imported at parse time. > > > > > > > > - The plan uses the same schema as YAML Dags (tasks, uses, with, > > needs), > > > so > > > > Airflow has one vocabulary for declaring a graph of tasks. Templating > > > is > > > > explicit, as in YAML Dags: a value is rendered only when marked with > > > $t. > > > > > > > > - No subclassing. An LLM planner would itself be a planner, shipped by > > a > > > > provider, with a validator on top. An @task.agent returning the plan > > as > > > > structured output should also work with no extra machinery. > > > > > > > > - Selected entries are real task instances, not map indices of one > > mapped > > > > task. The planner submits the plan through the Execution API while it > > > is > > > > still running, and the API server creates the task instances, which > > is > > > the > > > > approach Task Loops took. The plan is stored against the region, so > > > after > > > > a > > > > clear and re-plan, the earlier plan and the task instances that ran > > > under > > > > it > > > > are still there to look at. > > > > > > > > - At parse time the group serializes as two nodes: the planner and a > > > merge > > > > node. The merge is always an EmptyOperator, and anything downstream > > of > > > the > > > > group attaches to it. > > > > > > > > - Catalog membership is by exact class, so a subclass of an approved > > > > operator > > > > is rejected. That can be loosened later; tightening it later would be > > > much > > > > harder. > > > > > > > > Comments on the page or here are both fine. > > > > > > > > Thanks, > > > > Kaxil > > > > > > > > On Tue, 18 Aug 2026 at 21:37, Przemysław Mirowski <[email protected]> > > > wrote: > > > > > > > > > Thanks for the update Kaxil. Looks much better. I've added a couple > > of > > > new > > > > > comments. > > > > > > > > > > On 2026/08/10 21:26:24 Kaxil Naik wrote: > > > > > > Thanks for your feedback Przemysław, I updated the AIP > > > > > > > > > > > > On Fri, 7 Aug 2026 at 03:20, Przemysław Mirowski <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Hi Kaxil, > > > > > > > > > > > > > > I've added some comments to the AIP. In general, the idea looks > > > > > > > interesting. > > > > > > > > > > > > > > Could we possibly rearrange the sections a bit? During the > > reading, > > > > > > > sometimes one thing was stated when the reader does not know a > > > > > particular > > > > > > > part of the change (e.g. merge policy - I added a comment next to > > > it). > > > > > > > Also, there are a lot of new definitions - maybe it would be good > > > to > > > > > create > > > > > > > a section which will gather them all and shortly explain? > > > > > > > > > > > > > > On 2026/07/30 10:16:20 Kaxil Naik wrote: > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I would like to start a discussion on AIP-113: Dynamic Task > > > Groups. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > > > > > > > > > > > > > > > > tldr: > > > > > > > > Dynamic Task Mapping made cardinality dynamic: run the same > > task > > > > > over a > > > > > > > > collection discovered at run time. What is still static is the > > > shape, > > > > > > > > meaning which tasks > > > > > > > > run in a region and how they depend on each other. > > > > > > > > > > > > > > > > Branching gets you half of that already, and I want to be > > precise > > > > > about > > > > > > > > which half, because > > > > > > > > I had this wrong in an earlier draft. `choose_branch` can > > return > > > a > > > > > list, > > > > > > > so > > > > > > > > a branch already > > > > > > > > selects an arbitrary subset of its declared successors. What a > > > branch > > > > > > > > cannot do is decide > > > > > > > > the dependency edges *among* the tasks it selected. Selection > > is > > > > > solved; > > > > > > > > ordering and wiring are not. That is the > > > > > > > > gap this AIP is aimed at. > > > > > > > > > > > > > > > > The proposal is a Task Group whose members and internal edges > > are > > > > > decided > > > > > > > > at run time from a fixed catalog the author declares. A > > generator > > > > > task > > > > > > > > emits a plan, the group materializes > > > > > > > > exactly the selected entries as real task instances, and the > > > region > > > > > > > > converges on a > > > > > > > > synthesized merge node. The catalog is what serializes, so two > > > runs > > > > > of > > > > > > > > different shapes share > > > > > > > > one DAG version, and an entry the plan did not select never > > > becomes a > > > > > > > task > > > > > > > > instance. The > > > > > > > > planner can be a control table, an impact analysis, a triage > > > rule, > > > > > or an > > > > > > > > LLM. > > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > > > Would love to hear your thoughts & get feedback. Please comment > > > on > > > > > the > > > > > > > AIP > > > > > > > > doc itself. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Kaxil > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
