Hi Kaxil, I've added some comments to the AIP. In general, the idea looks interesting.
Could we possibly rearrange the sections a bit? During the reading, sometimes one thing was stated when the reader does not know a particular part of the change (e.g. merge policy - I added a comment next to it). Also, there are a lot of new definitions - maybe it would be good to create a section which will gather them all and shortly explain? On 2026/07/30 10:16:20 Kaxil Naik wrote: > Hi all, > > I would like to start a discussion on AIP-113: Dynamic Task Groups. > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups > > tldr: > Dynamic Task Mapping made cardinality dynamic: run the same task over a > collection discovered at run time. What is still static is the shape, > meaning which tasks > run in a region and how they depend on each other. > > Branching gets you half of that already, and I want to be precise about > which half, because > I had this wrong in an earlier draft. `choose_branch` can return a list, so > a branch already > selects an arbitrary subset of its declared successors. What a branch > cannot do is decide > the dependency edges *among* the tasks it selected. Selection is solved; > ordering and wiring are not. That is the > gap this AIP is aimed at. > > The proposal is a Task Group whose members and internal edges are decided > at run time from a fixed catalog the author declares. A generator task > emits a plan, the group materializes > exactly the selected entries as real task instances, and the region > converges on a > synthesized merge node. The catalog is what serializes, so two runs of > different shapes share > one DAG version, and an entry the plan did not select never becomes a task > instance. The > planner can be a control table, an impact analysis, a triage rule, or an > LLM. > > --- > > Would love to hear your thoughts & get feedback. Please comment on the AIP > doc itself. > > Regards, > Kaxil > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
