raphaelauv opened a new pull request, #75552: URL: https://github.com/apache/airflow/pull/75552
closes: https://github.com/apache/airflow/issues/75497 and https://github.com/apache/airflow/issues/40528 before: - DagRun.schedule_tis() deferred tasks as soon as they became ready. That happened outside the scheduler's locked selection step, so pool slots and the max_active_* concurrency limits were never checked. What changes: - One path: schedule_tis() now only sets these tasks to SCHEDULED. The deferral happens in _select_task_instances_to_queue(), under the pool row lock. - Limits enforced: pool slots, max_active_tasks, max_active_tis_per_dag, max_active_tis_per_dagrun, and the per-loop max_tis. This also holds with several schedulers running. - No executor slots used: the scheduler defers these tasks straight to the triggerer. - Resumed tasks still go to a worker: a task resumed by its trigger has next_method set, so it's queued to run execute_complete and isn't deferred again. What users will notice: - One loop later: a task is deferred one scheduler loop after it becomes SCHEDULED, not right away. - Pool slot needed even with include_deferred=False: a full pool now delays these tasks. Once the task is deferred the slot is released, so in that case the pool limits how fast tasks get deferred, not how many run in the triggerer. Set include_deferred=True to cap how many run in the triggerer. What the PR adds: - Unit tests for the pool limit and for resumed tasks; the existing schedule_tis tests now expect SCHEDULED instead of DEFERRED. - Doc notes in deferring.rst and pools.rst covering the user-facing points above. --- ##### Was generative AI tooling used to co-author this PR? - [X] oppus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
