GitHub user amit-mittal added a comment to the discussion: Disabling auto-refresh from UI has gone missing
I do agree with you that it shouldn't take that long and there is definitely something wrong with our setup. In pre-prod environments which we have currently upgrade to v3, we allocate less resources, but still it shouldn't have been this slow or get worse after the upgrade (generally performance should improve). Regarding the index use for DB statements, I linked you to the GitHub issue in my previous message where we clearly are not using the index as part of scheduler loop. I enabled `otel` tracing, but I couldn't find any traces for the requests from `WebServer` (I do see traces for `Scheduler`). Is there a way to get more details about this? Sorry, I couldn't understand what you meant by "i section g what is going wrong - which api calls are taking long and looking at corresponding queries in the dn". Some details about our setup: - We are running MySQL 8.x with 16 GB memory and 10 GB InnoDB buffer pool size. Our CPU usage used to be at 10%, but now it hovers at 50%. - The instance handles around 200 DAGs, not too complicated. In database, we have last 2 years history. - We have set page limit in Airflow as 100 - AirflowWebServer has 3 GB memory and can go max up to 8 GB - AirflowScheduler has 4 GB memory and can go max up to 10 GB. We use `LocalExecutor`. Currently, we have 2 replicas. - AirflowDAGProcessor has 2 GB memory and can go max up to 5 GB. - All the DAGs are stored on NFS Production instance and DBs running Airflow v2.10.x have more resources, but currently the instance running on Airflow v3.0.2 has the above resources. If you have a recommendation (or need more details), that will be of huge help. We can increase the allocated resources and tweak the Airflow configurations. GitHub link: https://github.com/apache/airflow/discussions/53175#discussioncomment-13730022 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
