Hi Shahar,
Thanks for the review! I've updated the AIP for all three.
1. Overview
There's now a short Overview section at the start of the proposal. In short:
- The Dag author writes a lookup instead of a literal, e.g.
pool=Lookup.param("warehouse_pool").
- The Dag processor validates it and serializes it with the Dag.
- When a run is created, the lookup is resolved from the run's conf or run type.
- The value is written onto each task instance whenever its fields are set
(creation, mapped expansion, clear, retry). task_instance_mutation_hook runs
after that.
- The scheduler reads pool, slots, priority and queue from the task instance,
same as today.
(The name "Lookup" may still change. Ash and I are discussing it on the wiki
page.)
2. Changes after a run starts
Task instances that already exist are never rewritten. A change only matters
the next time a task instance's fields are written, so for new mapped
instances, clears and retries.
- Override: say "extract" was moved from etl_pool to heavy_pool, and heavy_pool
is then un-nominated or deleted. The existing task instance keeps heavy_pool.
If it is retried or cleared, the override no longer applies and it goes back to
etl_pool.
- Lookup: a deleted pool behaves like a deleted literal pool does today. The
task waits and the Dag shows a warning.
3. Scope
Lookups cover the cases the author can plan for. Overrides are for the ones
nobody planned: someone on call rerunning one run with a heavy task in a bigger
pool, or moving a run off a saturated pool, on a Dag they may not own and can't
redeploy right now. While discussing this with Elad, he specifically mentioned
the need for this override among users he's talked to.
Ideally, I'd like to keep both in the AIP. They share the same path (resolved
at run creation, applied at the same points, cluster policy last), and the part
that needs agreement is the security model as a whole. Overrides are off until
an administrator nominates a pool or queue, so nothing changes for deployments
that don't opt in.
> On Nov 7, 2026, at 10:24 AM, Shahar Epstein via dev <[email protected]>
> wrote:
>
> Hey Ramit,
>
> Thanks for bringing this up! The idea makes sense to me, and I’d like to
> see it move forward.
>
> Could you please clarify the following points in the AIP?
>
> 1.
>
> High-level design: Could you add a short overview or diagram showing how
> a lookup sets a scheduling field, such as pool or queue, and how that
> setting reaches the task instance? The AIP moves quickly from the problem
> to low-level details, which makes it hard to see the overall flow.
> 2.
>
> Changes after a run starts: What happens if a pool is un-nominated or
> deleted after a run starts? Are task instances that already exist updated,
> or does the change apply only when task instances are created or updated
> later?
> 3.
>
> Scope: Which use cases need trigger-time overrides? Do they need to be
> included in the first version alongside lookups?
>
>
> Thanks,
> Shahar
>
>> On Sat, Oct 31, 2026 at 2:25 AM Ramit Kataria via dev <
>> [email protected]> wrote:
>>
>>
>> Hi all,
>>
>> I'd like to start the discussion on AIP-117, Per-run Task Scheduling
>> Controls:
>>
>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/446071748/WIP+AIP-117+Per-Run+Task+Scheduling+Controls
>>
>> A task's pool, pool_slots, priority_weight and queue are fixed in the Dag
>> file. You can't change them for one run when you trigger it, and a Dag
>> author can't make them depend on a param or on the kind of run. Today
>> people duplicate tasks and branch between them, keep two copies of a Dag,
>> or ask their Deployment Manager for a cluster policy that reads
>> dag_run.conf.
>>
>> The proposal has two parts.
>>
>> 1. Lookups, for Dag authors. A task field can take its value from a Dag
>> param, the run type, or an Airflow Variable instead of a literal, for
>> example pool=Lookup.param("warehouse_pool"). A param used this way must
>> declare enum (or minimum and maximum for numbers), so whoever triggers the
>> run can only pick values the author listed, and trigger permission is
>> enough.
>>
>> 2. Trigger-time overrides, for operators. An optional "overrides" field on
>> the trigger request sets these fields for specific tasks, TaskGroups or
>> pools, for Dags whose author didn't prepare anything. It needs Dag edit
>> permission and only works with pools and queues an administrator has
>> nominated. Nothing is nominated by default, so nothing changes on upgrade.
>>
>> Values are resolved when the run is created and applied whenever task
>> instances are created, cleared or retried. A task_instance_mutation_hook
>> still runs last and wins. The scheduler never runs user code, and its
>> critical section is unchanged.
>>
>> To make this concrete, here is roughly what the docs would say once it
>> lands:
>>
>> - Params: "A task's pool, pool slots, priority weight or queue can come
>> from a param, for example pool=Lookup.param("warehouse_pool") with
>> Param(default="wh_small", enum=["wh_small", "wh_large"]). The trigger form
>> shows it as a dropdown and notes which tasks it affects. Scheduled runs use
>> the default. The same value can be passed as conf from the REST API, the
>> CLI, TriggerDagRunOperator or a backfill."
>>
>> - Dag runs: "If you can edit a Dag, you can override these fields for one
>> run without changing the Dag, for example {"overrides": {"tasks":
>> {"extract": {"pool": "heavy_pool"}}}}. Keys can be task IDs or TaskGroup
>> IDs, and "pools": {"etl_pool": "etl_overflow_pool"} moves every task of the
>> run from one pool to another. Only pools and queues your administrator has
>> nominated can be used."
>>
>> - Pools: "A pool can't be used for trigger-time overrides until it's
>> marked as an override target. Treat that as letting anyone who can edit a
>> Dag send its tasks to this pool. Removing the mark takes effect
>> immediately, and runs already in progress fall back to their normal pool."
>>
>> - Cluster policies: "Lookups and overrides are applied before
>> task_instance_mutation_hook, so the hook keeps the last word. task_policy
>> runs at parse time and only sees defaults, so put rules that must hold for
>> every run in task_instance_mutation_hook."
>>
>> I'd especially like feedback on:
>>
>> - The security model: authors bound what the person triggering can pick,
>> and administrators nominate what Dag editors can override to.
>> - The name "Lookup", which is a working name.
>>
>> Thanks,
>> Ramit
>>