eladkal closed issue #34877: Scheduler is spending most of its time in
clear_not_launched_queued_tasks function
URL: https://github.com/apache/airflow/issues/34877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
dirrao commented on issue #34877:
URL: https://github.com/apache/airflow/issues/34877#issuecomment-1763648593
@eladkal They are related. Its more close to #7542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
eladkal commented on issue #34877:
URL: https://github.com/apache/airflow/issues/34877#issuecomment-1763328395
> So, this leads the scheduler to spend most of its time in this function
rather than scheduling the tasks
This is interesting.
I believe this is the K8s equivalent for
mattp- commented on issue #34877:
URL: https://github.com/apache/airflow/issues/34877#issuecomment-1761502806
@dirrao I think that may be a chatgpt AI comment lol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
dirrao commented on issue #34877:
URL: https://github.com/apache/airflow/issues/34877#issuecomment-1761396289
@ROVAN1220
Thanks for the inputs.
I am working on the MR to address the performance issue.
--
This is an automated message from the Apache Git Service.
To respond to
ROVAN1220 commented on issue #34877:
URL: https://github.com/apache/airflow/issues/34877#issuecomment-1759924519
It seems like you've identified a performance bottleneck in your Airflow
setup when running on a large Kubernetes cluster with a high number of queued
tasks. Your proposed
dirrao opened a new issue, #34877:
URL: https://github.com/apache/airflow/issues/34877
### Apache Airflow version
2.7.1
### What happened
Airflow running the clear_not_launched_queued_tasks function on a certain
frequency (default 30 seconds). When we run the airflow on
