kaveti opened a new issue, #43117: URL: https://github.com/apache/airflow/issues/43117
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.5.3 ### What happened? I have dag which dynamically generates close to 800 tasks. when i hit /tries?day=30 and /duration?days=30 webserver API. it just simply fails with error. - Gunicorn Workers: Increasing the number of workers has not resolved the page loading issues, as the problem isn’t related to incoming request traffic. - Timeout Settings: Adjusted both WORKER_TIMEOUT (3000) and master WORKER_TIMEOUT(3000) to higher values, but the page still fails to load. - Worker Class: Switching to the gevent worker class did not improve performance, despite its ability to handle parallelism. - CPU Limits: Increased the webserver’s CPU limits to 24 cores, yet one of the Gunicorn worker consistently maxes out CPU usage. note: I have checked the webserver_error.log as well. no worker is getting killed abruptly. <img width="1328" alt="Screenshot 2024-10-16 at 9 21 40 PM" src="https://github.com/user-attachments/assets/128f627f-0f2a-473b-adda-2716bc5624aa"> ### What you think should happen instead? _No response_ ### How to reproduce - Create a dag which generation dynamic tasks more than 800. and then hit /dag_id/tries?day=30 or /dag_id/duration?day=30 API. <img width="1707" alt="Screenshot 2024-10-17 at 6 50 31 PM" src="https://github.com/user-attachments/assets/4316359c-bcb9-4227-933a-7bd1902559d9"> ### Operating System Red Hat Enterprise Linux (9.4) - rhel ### Versions of Apache Airflow Providers _No response_ ### Deployment Other 3rd-party Helm chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
