GitHub user michaelosthege closed a discussion: High CPU and database load caused by Airflow 3 dag-processor and scheduler
### Apache Airflow version 3.0.2 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Yesterday I upgraded my small home setup (Raspberry Pi, 2 dags, 3 tasks) to Airflow 3. In comparison to Airflow 2.9.0 the CPU load has increased dramatically: <img width="2768" height="530" alt="Image" src="https://github.com/user-attachments/assets/7ab02ab9-b3c5-4281-ae6b-69e6dccfe3b3"; /> A few minutes later I also stopped the scheduler and database CPU load went down too. ### What you think should happen instead? Airflow 3 should cause the same or lower CPU load to Airflow 2. ### How to reproduce My docker-compose setup uses this `Dockerfile` ``` FROM apache/airflow:3.0.2 ENV AIRFLOW__CORE__EXECUTOR=LocalExecutor ENV AIRFLOW__API__BASE_URL="http://junas.fritz.box:5000"; ENV AIRFLOW__CORE__AUTH_MANAGER="airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager" # Note: AIRFLOW__API_AUTH__JWT_SECRET is set in the .env file ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="postgresql+psycopg2://airflowu:[email protected]:5001/airflowdb" ENV AIRFLOW__CORE__LOAD_EXAMPLES="False" ENV AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK="True" # Settings to reduce CPU load ENV AIRFLOW__API__WORKERS=1 ENV AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=120 ENV AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=1 ENV AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL=60 COPY requirements.txt /tmp/requirements.txt # one entry: paho-mqtt RUN pip install -r /tmp/requirements.txt ``` The docker-compose services: ```yaml airflow-db: container_name: airflow-db build: context: ./dockerfiles/airflow-postgres dockerfile: Dockerfile restart: unless-stopped ports: - "5001:5432" volumes: - af_pgdata:/var/lib/postgresql/data environment: POSTGRES_DB: airflowdb POSTGRES_USER: airflowu POSTGRES_PASSWORD: xxxxxxx airflow-api-server: container_name: airflow-api-server build: context: ./dockerfiles/airflow dockerfile: Dockerfile depends_on: - airflow-db restart: unless-stopped env_file: ./dockerfiles/airflow/.env ports: - "5000:8080" user: "1000:0" volumes: - /home/admin/airflow/dags:/opt/airflow/dags - /home/admin/airflow/logs:/opt/airflow/logs command: bash -c "airflow db migrate && airflow api-server" healthcheck: test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version";] interval: 30s timeout: 10s retries: 5 start_period: 30s airflow-dag-processor: container_name: airflow-dag-processor build: context: ./dockerfiles/airflow dockerfile: Dockerfile depends_on: - airflow-db - airflow-api-server restart: unless-stopped env_file: ./dockerfiles/airflow/.env user: "1000:0" volumes: - /home/admin/airflow/dags:/opt/airflow/dags - /home/admin/airflow/logs:/opt/airflow/logs command: bash -c "airflow dag-processor" healthcheck: test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob'] interval: 30s timeout: 20s retries: 5 start_period: 30s airflow-scheduler: container_name: airflow-scheduler build: context: ./dockerfiles/airflow dockerfile: Dockerfile depends_on: - airflow-db - airflow-api-server - airflow-dag-processor restart: unless-stopped env_file: ./dockerfiles/airflow/.env user: "1000:0" volumes: - /home/admin/airflow/dags:/opt/airflow/dags - /home/admin/airflow/logs:/opt/airflow/logs command: bash -c "airflow scheduler" healthcheck: test: ["CMD", "curl", "--fail", "http://localhost:8974/health";] interval: 30s timeout: 10s retries: 5 start_period: 30s ``` ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/53177 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
