Hi Ramit, +1 to the idea, as we discussed earlier. The AIP has grown much more mature since then. Lookup works for me as a name.
Thanks & Regards, Amogh Desai On Wed, Nov 11, 2026 at 10:59 AM Ramit Kataria via dev < [email protected]> wrote: > Hi Shahar, > > Thanks for the review! I've updated the AIP for all three. > > 1. Overview > > There's now a short Overview section at the start of the proposal. In > short: > > - The Dag author writes a lookup instead of a literal, e.g. > pool=Lookup.param("warehouse_pool"). > - The Dag processor validates it and serializes it with the Dag. > - When a run is created, the lookup is resolved from the run's conf or run > type. > - The value is written onto each task instance whenever its fields are set > (creation, mapped expansion, clear, retry). task_instance_mutation_hook > runs after that. > - The scheduler reads pool, slots, priority and queue from the task > instance, same as today. > > (The name "Lookup" may still change. Ash and I are discussing it on the > wiki page.) > > 2. Changes after a run starts > > Task instances that already exist are never rewritten. A change only > matters the next time a task instance's fields are written, so for new > mapped instances, clears and retries. > > - Override: say "extract" was moved from etl_pool to heavy_pool, and > heavy_pool is then un-nominated or deleted. The existing task instance > keeps heavy_pool. If it is retried or cleared, the override no longer > applies and it goes back to etl_pool. > - Lookup: a deleted pool behaves like a deleted literal pool does today. > The task waits and the Dag shows a warning. > > 3. Scope > > Lookups cover the cases the author can plan for. Overrides are for the > ones nobody planned: someone on call rerunning one run with a heavy task in > a bigger pool, or moving a run off a saturated pool, on a Dag they may not > own and can't redeploy right now. While discussing this with Elad, he > specifically mentioned the need for this override among users he's talked > to. > > Ideally, I'd like to keep both in the AIP. They share the same path > (resolved at run creation, applied at the same points, cluster policy > last), and the part that needs agreement is the security model as a whole. > Overrides are off until an administrator nominates a pool or queue, so > nothing changes for deployments that don't opt in. > > > On Nov 7, 2026, at 10:24 AM, Shahar Epstein via dev < > [email protected]> wrote: > > > > Hey Ramit, > > > > Thanks for bringing this up! The idea makes sense to me, and I’d like to > > see it move forward. > > > > Could you please clarify the following points in the AIP? > > > > 1. > > > > High-level design: Could you add a short overview or diagram showing > how > > a lookup sets a scheduling field, such as pool or queue, and how that > > setting reaches the task instance? The AIP moves quickly from the > problem > > to low-level details, which makes it hard to see the overall flow. > > 2. > > > > Changes after a run starts: What happens if a pool is un-nominated or > > deleted after a run starts? Are task instances that already exist > updated, > > or does the change apply only when task instances are created or > updated > > later? > > 3. > > > > Scope: Which use cases need trigger-time overrides? Do they need to be > > included in the first version alongside lookups? > > > > > > Thanks, > > Shahar > > > >> On Sat, Oct 31, 2026 at 2:25 AM Ramit Kataria via dev < > >> [email protected]> wrote: > >> > >> > >> Hi all, > >> > >> I'd like to start the discussion on AIP-117, Per-run Task Scheduling > >> Controls: > >> > >> > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/446071748/WIP+AIP-117+Per-Run+Task+Scheduling+Controls > >> > >> A task's pool, pool_slots, priority_weight and queue are fixed in the > Dag > >> file. You can't change them for one run when you trigger it, and a Dag > >> author can't make them depend on a param or on the kind of run. Today > >> people duplicate tasks and branch between them, keep two copies of a > Dag, > >> or ask their Deployment Manager for a cluster policy that reads > >> dag_run.conf. > >> > >> The proposal has two parts. > >> > >> 1. Lookups, for Dag authors. A task field can take its value from a Dag > >> param, the run type, or an Airflow Variable instead of a literal, for > >> example pool=Lookup.param("warehouse_pool"). A param used this way must > >> declare enum (or minimum and maximum for numbers), so whoever triggers > the > >> run can only pick values the author listed, and trigger permission is > >> enough. > >> > >> 2. Trigger-time overrides, for operators. An optional "overrides" field > on > >> the trigger request sets these fields for specific tasks, TaskGroups or > >> pools, for Dags whose author didn't prepare anything. It needs Dag edit > >> permission and only works with pools and queues an administrator has > >> nominated. Nothing is nominated by default, so nothing changes on > upgrade. > >> > >> Values are resolved when the run is created and applied whenever task > >> instances are created, cleared or retried. A task_instance_mutation_hook > >> still runs last and wins. The scheduler never runs user code, and its > >> critical section is unchanged. > >> > >> To make this concrete, here is roughly what the docs would say once it > >> lands: > >> > >> - Params: "A task's pool, pool slots, priority weight or queue can come > >> from a param, for example pool=Lookup.param("warehouse_pool") with > >> Param(default="wh_small", enum=["wh_small", "wh_large"]). The trigger > form > >> shows it as a dropdown and notes which tasks it affects. Scheduled runs > use > >> the default. The same value can be passed as conf from the REST API, the > >> CLI, TriggerDagRunOperator or a backfill." > >> > >> - Dag runs: "If you can edit a Dag, you can override these fields for > one > >> run without changing the Dag, for example {"overrides": {"tasks": > >> {"extract": {"pool": "heavy_pool"}}}}. Keys can be task IDs or TaskGroup > >> IDs, and "pools": {"etl_pool": "etl_overflow_pool"} moves every task of > the > >> run from one pool to another. Only pools and queues your administrator > has > >> nominated can be used." > >> > >> - Pools: "A pool can't be used for trigger-time overrides until it's > >> marked as an override target. Treat that as letting anyone who can edit > a > >> Dag send its tasks to this pool. Removing the mark takes effect > >> immediately, and runs already in progress fall back to their normal > pool." > >> > >> - Cluster policies: "Lookups and overrides are applied before > >> task_instance_mutation_hook, so the hook keeps the last word. > task_policy > >> runs at parse time and only sees defaults, so put rules that must hold > for > >> every run in task_instance_mutation_hook." > >> > >> I'd especially like feedback on: > >> > >> - The security model: authors bound what the person triggering can pick, > >> and administrators nominate what Dag editors can override to. > >> - The name "Lookup", which is a working name. > >> > >> Thanks, > >> Ramit > >> >
