Nataneljpwd opened a new pull request, #68861:
URL: https://github.com/apache/airflow/pull/68861
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
When new dagruns are created in bulk (i.e with triggerDagRunOperator), the
scheduler might struggle with the amount created, and cause other dagruns to
starve.
This is due to the sort order in get_running_dagruns_to_examine which
selects (with a nulls first) by last scheduling decision, which means that if a
lot of new dagruns are created, the scheduler will examine them first, and in
situations where the dags have a lot of tasks (hundreds to tens of thousands)
it can cause the scheduler to stall, as it has to both examine a lot of
dagruns, and create new tasks for those dagruns.
When we have tried to tune the max_dagruns_per_loop_to_schedule we either
got starvation of other dagruns OR the scheduler being reset due to not
returning a heartbeat for a long time and failing the readiness probe.
To fix this, a new configuration is added,
max_new_dagruns_per_loop_to_schedule which can help when a lot of new dagruns
are created in large batches at the same time, and allow the scheduler to both
look at existing dagruns (not starving them and causing them to timeout with no
running / scheduled tasks) and create and manage the new dagruns.
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
- [x] No
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
