xBis7 opened a new pull request, #77032:
URL: https://github.com/apache/airflow/pull/77032
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
This is based on a test environment that I have been using for quite some
time for performance testing or to stress test a new feature under load and
make sure it doesn't introduce any regressions.
The patch includes the test setup along with instructions for agents. The
main idea is that the files will be used as a base that can be easily adjusted
for different test scenarios. Unfortunately there isn't one shape that fits all
for Airflow, and that's why a customizable solution is required.
What can be customized
* Resource profile
* certain Airflow config properties that change performance significantly
when set to the optimal value for the available resources
* Number of Dag Runs and tasks
* Source code under test
* Grafana dashboard
The files in this patch aren't meant to run in the CI. The user asks an
agent to apply the necessary changes to the test setup, runs the tests on the
dev branch, checks the results and then resets all test changes.
## Running the Test
* Start breeze
```bash
breeze shell --backend postgres --integration otel --integration redis
```
`otel` is used for the metrics and `redis` is used for the celery
workers.
* Run the test
```bash
pytest -s airflow-core/tests/integration/perf/test_perf.py --durations=0
```
* Visit the Grafana dashboard
http://localhost:23000/d/airflow-scheduler-perf/scheduler-performance?orgId=1&refresh=10s&from=now-1h&to=now
You can compare timings between the two test runs as well as the before and
after metrics.
## Test Overview
The test
* Creates Dags and stores them in the DB as QUEUED
* Same as unpausing and triggering them but without the overhead of
going through the CLI
* Same facts for every test run, the test becomes more deterministic
* Starts the following processes under breeze
* a scheduler
* an API server
* 3 celery workers
* Waits for all Dags to finish
* While the Dags are running, captures metrics
The test is parametrized and it runs twice. Each execution, set the
environment variable `PERF_TEST__NEW_CODE` at the beggining, once to `True` and
once to `False`.
The environment variable is meant to be be used for testing any new source
code like so
```python
if os.environ.get("PERF_TEST__NEW_CODE", "").lower() == "true":
# new code with changes
ready = select_ready_tis_with_cache(schedulable_tis, finished_tis)
else:
# original code, probably from main
ready = select_ready_tis(schedulable_tis, finished_tis)
```
## Test Dag
The Dag consists of 9 stages where each stage is a mapped task, that turns
into N task instances of the same task. It accepts a param that defines the
number N of the mapped task instances.
This is a representation of the Dag
```
produce_parameters
|
stage_1[0] ... stage_1[N-1]
| |
stage_2[0] ... stage_2[N-1]
| |
...
stage_9[0] ... stage_9[N-1]
```
E.g. for 3 mapped tasks per stage
```
produce_parameters
/ | \
stage_1[0] stage_1[1] stage_1[2]
\ | /
all three done
/ | \
stage_2[0] stage_2[1] stage_2[2]
\ | /
...
/ | \
stage_9[0] stage_9[1] stage_9[2]
```
The dependencies follow this shape and on each stage all task instances are
running in parallel
```
produce_parameters >> stage_1[0..2] >> stage_2[0..2] >> stage_3[0..2] >> ...
>> stage_9[0..2]
```
For N=3, we have 3*9 = 27 tasks in total.
The test also decides how many runs of the Dag are triggered. Therefore, we
have
9 stages * N mapped tis * X Dag Runs
The test can be scaled very easily. E.g. for 50 mapped tis per stage and 30
runs, we end up with
9 * 50 * 30 = 13.500 tis in total.
## Resource Profiles
Performance is greatly affected by Airflow configuration. But the optimal
configuration depends on the machine and the workload.
Currently, there are 2 profiles
* dynamic
* the default profile
* the values are calculated based on the resources of the current machine
* custom
* all values are empty, which is the same as using the Airflow defaults
* the user can adjust the values based on their needs
* e.g. could add a new function that runs a different set of dags and
set certain config values to test a particular scenario
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Claude code Fable 5.1
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
