cesarcgs-tlf opened a new issue, #51284:
URL: https://github.com/apache/airflow/issues/51284
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.0
### What happened?
In our production environment using `CeleryExecutor` and Redis as the
broker, we’re experiencing intermittent but repeatable delays where tasks
remain in the `queued` state for approximately 950 seconds before they begin
execution.
This behavior is unexpected given that:
- There is plenty of free capacity in the pools used by the tasks.
- Four Celery workers are running, each with a concurrency of 32, for a
total of 128 available slots.
- The Redis broker is not under high CPU/memory pressure.
- The scheduler heartbeat is healthy and operational.
- We verified the tasks are coming from different DAGs and using different
pools.
- There is no CPU saturation on any component (scheduler, Redis, workers).
- The logs from the scheduler and workers show no errors or timeouts.
- The configuration key `AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT` is set to
3600.
Despite this, some tasks get queued and then do not start executing until
~950 seconds later, even when there is no obvious bottleneck or backlog.
### What you think should happen instead?
We expected tasks to transition from `queued` to `running` within a few
seconds, as there are ample resources and capacity available.
### How to reproduce
We don't have a minimal reproducible example, but the issue appears randomly
and independently of DAG size or the number of enqueued tasks. For example:
1. A DAG triggers one or more tasks.
2. Some tasks transition from `queued` to `running` quickly as expected.
3. Others remain stuck in `queued` for approximately 950 seconds, even when:
- Pools have available slots.
- Workers are idle or underutilized.
- Redis is healthy and not under pressure.
- There is no backlog of scheduled or queued tasks.
This behavior has been observed:
- Across multiple DAGs.
- Across different pools.
- Even when very few tasks are running in the system.
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-common-io==1.3.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-fab==1.0.2
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-sqlite==3.7.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
