alex-magno opened a new issue, #34013: URL: https://github.com/apache/airflow/issues/34013
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened We are facing a quite interesting problem with our celery workers. **We run on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2**. When the amount of running tasks is high (close to our max cap), we notice an increase in the celery workers memory usage followed by a stale period. At the same time, CPU utilization goes to zero.  During this time, the worker container seems to freeze (no logs are produced). Of the tasks that were running, almost all of them failed with the only log being: ``` [2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not met for <TaskInstance: ***** scheduled__2023-08-31T10:00:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state. [2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run ``` It is important to note that this problem happens sometimes, but not every time. Which makes it even more confusing. Also, workers pods are not killed in the process, nor do they throw any exceptions. Some of our configurations are: ``` AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32 AIRFLOW__CORE__PARALLELISM = 64 AIRFLOW__CELERY__WORKER_CONCURRENCY = 12 ``` We are running at the moment with 5 celery workers, with no CPU or memory limits. The scheduler seems fine during those periods, producing regular logs. The only thing I can think of at the moment is that its probably due to a shortage of resources in the celery workers? However the worker concurrency limit should prevent a high usage of CPU/memory. Any ideas will be appreciated. Some executor metrics on the same time period:   ### What you think should happen instead CPU and Memory resources should not go stale. Also, workers should keep logging as usual, instead of "freezing". ### How to reproduce Deploy through Helm chart, using the `CeleryExecutor`. Run the maximum number of allowed tasks at the same time. However, the problem is not guaranteed to happen. ### Operating System Bottlerocket OS ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details AWS EKS with k8s `v1.26.1` - Nodes (m-series) using Bottlerocket OS. ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
