So, I propose, DAG.max_active_tasks should be evaluated per-dag-run. And
we can change the name accordingly if folks on board.
Agree. And possibly we name it DAG.max_active_tasks_per_run
J.
On Fri, Oct 4, 2024 at 3:30 PM Ryan Hatter
wrote:
> I think I agree with this:
>
> I feel it should be
I think I agree with this:
I feel it should be applied at the dag *run* scope
> and not across all dag runs.
>
Just a thought: If someone *did* want to run multiple DAG runs at the same
time and limit the max active tasks per DAG, they could create a pool for
that DAG and pass the pool in default
Ok, sorry, these concurrency settings are confusing.
Let me clarify.
`max_active_tasks_per_dag` is a core airflow setting and it provides the
default for DAG.max_active_tasks.
DAG.max_active_tasks I think is a reasonable config to have but the problem
in my view is the scope. I feel it should b
The setting max_active_tasks_per_dag seems mostly useless to me / and
footgunish.
Why?
Because you already have a setting for max active dag runs. If you don't
want to run more tasks, don't create the extra dag runs.
We also already have a mechanism (param on base operator) for limiting
indivi
