vchiapaikeo opened a new issue, #38776: URL: https://github.com/apache/airflow/issues/38776
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.3 ### What happened? We have started to notice more slowness on initial page load. On our deployment, it takes nearly 6s for the server to respond. <img width="1512" alt="image" src="https://github.com/apache/airflow/assets/9200263/0fc199ff-7594-40cb-99ba-6b679ef87aa4";> We started to do some analysis of the queries that are being run and we found 3 that take over 1500ms (and likely are being run serially). 1. Getting running dags (see screenshot below) -1.4s 1. Getting failed dags (see screenshot below) - 1.8s 1. Populating the dag ids (which defaults to 25 and we've set to 100 but we don't see much improvement when we lower it - only 500ms at most) - 1.6s <img width="325" alt="image" src="https://github.com/apache/airflow/assets/9200263/ddf2ce54-8f8e-4563-b4ec-3705a74fb296";> All of these queries are attached in the following txt file with explain plans and explain analyzes: [slow_queries_on_initial_load.txt](https://github.com/apache/airflow/files/14885733/slow_queries_on_initial_load.txt) ### What you think should happen instead? We are wondering if there are any ways we could possibly improve the performance of initial page load. Some ideas that come to mind: 1. Caching with short ttls that are invalidated intelligently 2. Improved indexing strategies 3. Concurrent query execution 4. Your idea here! ### How to reproduce Add ~2-4k dags to an Airflow deployment and reload the initial page ### Operating System Debian 11 ### Versions of Apache Airflow Providers N/A ### Deployment Official Apache Airflow Helm Chart ### Deployment details We deploy our cluster on GCP's GKE ### Anything else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
