GitHub user IIPOCTAK created a discussion: Airflow 3.1.3 - DagProccessor CPU throttling
### Apache Airflow version 3.1.3 ### If "Other Airflow 2/3 version" selected, which one? _No response_ ### What happened? We are currently using Airflow 3.1.0. After upgrading to the latest versions: - 3.1.0 -> 3.1.1 - 3.1.0 -> 3.1.2 - 3.1.0 -> 3.1.3 We observed significant CPU throttling in the DagProcessor. As shown in the graphs below, CPU throttling has increased dramatically: <img width="687" height="304" alt="Image" src="https://github.com/user-attachments/assets/f7d85d1b-828a-4520-8cbc-9463a275ad63"; /> <img width="769" height="298" alt="Image" src="https://github.com/user-attachments/assets/c19e264c-6837-478a-9354-453ce1bdf477"; /> CPU usage has also increased, but not nearly as much as throttling: <img width="744" height="300" alt="Image" src="https://github.com/user-attachments/assets/d447d13a-cf2c-499d-b1e9-cdb3d27bb451"; /> <img width="1100" height="297" alt="Image" src="https://github.com/user-attachments/assets/05662bb0-b60f-47cf-ac20-5518b939de89"; /> These spikes on the graph correspond to the activity of the [DagFileProcessorManager](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/dagfile-processing.html) Our Airflow resources run in a Kubernetes cluster. We did not change anything except the package versions. requrements.txt: ``` -apache-airflow[fab,ldap]==3.1.0 -apache-airflow-providers-fab==2.4.4 +apache-airflow[fab,ldap]==3.1.3 +apache-airflow-providers-fab==3.0.1 ``` values.yaml: ``` -airflowVersion: 3.1.0 +airflowVersion: 3.1.3 -defaultAirflowTag: 3.1.0-python3.10 +defaultAirflowTag: 3.1.3-python3.10 ``` I were trying to change these parameters after upgrade: ``` AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE: "random_seeded_by_host" AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES: "1" AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: "300" ``` And increase CPU limits/requests (x2). Nothing helped. ### What you think should happen instead? _No response_ ### How to reproduce Upgrade airflow version, for example, from 3.1.0 -> 3.1.3 version and check CPU usage and throttling. ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers \# pip3 list | grep providers ``` apache-airflow-providers-amazon 9.12.0 apache-airflow-providers-celery 3.12.3 apache-airflow-providers-cncf-kubernetes 10.8.1 apache-airflow-providers-common-compat 1.7.4 apache-airflow-providers-common-io 1.6.3 apache-airflow-providers-common-messaging 2.0.0 apache-airflow-providers-common-sql 1.28.1 apache-airflow-providers-docker 4.4.3 apache-airflow-providers-elasticsearch 6.3.3 apache-airflow-providers-fab 2.4.4 apache-airflow-providers-ftp 3.13.2 apache-airflow-providers-git 0.0.8 apache-airflow-providers-google 18.0.0 apache-airflow-providers-grpc 3.8.2 apache-airflow-providers-hashicorp 4.3.2 apache-airflow-providers-http 5.3.4 apache-airflow-providers-microsoft-azure 12.7.1 apache-airflow-providers-mysql 6.3.4 apache-airflow-providers-odbc 4.10.2 apache-airflow-providers-openlineage 2.7.1 apache-airflow-providers-postgres 6.3.0 apache-airflow-providers-redis 4.3.1 apache-airflow-providers-sendgrid 4.1.3 apache-airflow-providers-sftp 5.4.0 apache-airflow-providers-slack 9.3.0 apache-airflow-providers-smtp 2.2.1 apache-airflow-providers-snowflake 6.5.4 apache-airflow-providers-ssh 4.1.4 apache-airflow-providers-standard 1.8.0 ``` ### Deployment Official Apache Airflow Helm Chart ### Deployment details helm ### Anything else? Constantly after upgrade ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/58403 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
