tienhung2812 commented on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-921858288
Hi, just got this issue again on `2.1.3` but the behavior quite different,
all of schedulers stuck into a DAG check loop, but restart the scheduler will
fix it, all other DAG as stuck in `scheduled`
```
[scheduled]> since the number of tasks running or queued from DAG abc_dag
is >= to the DAG's task concurrency limit of 16
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,511] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and
queued tasks
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,512] {scheduler_job.py:417} INFO - Not executing <TaskInstance:
abc_dag.aa.aa_20.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the
number of tasks running or queued from DAG abc_dag is >= to the DAG's task
concurrency limit of 16
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,512] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and
queued tasks
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,513] {scheduler_job.py:417} INFO - Not executing <TaskInstance:
abc_dag.aa.ff_46.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the
number of tasks running or queued from DAG abc_dag is >= to the DAG's task
concurrency limit of 16
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,513] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and
queued tasks`
airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178 | [2021-09-17
14:29:07,514] {scheduler_job.py:417} INFO - Not executing <TaskInstance:
abc_dag.ff.gg_121.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the
number of tasks running or queued from DAG abc_dag is >= to the DAG's task
concurrency limit of 16
```
The current system have about >= 150 active dags but we have a`abc_dag` have
more than 500 tasks inside
I have already increase the scheduler config to handling more task
- max_tis_per_query: 512
- max_dagruns_to_create_per_loop: 50
- max_dagruns_per_loop_to_schedule: 70
I have also add a scheduler healthcheck but it can not detect this issue
Environment:
- Celery Executor
- MySQL 8.0.23-14
- Airflow: 2.1.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
