luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2103466344
> > We conducted a throughput benchmarking test, executing 15k DAG's comprised of simple BashOperator tasks. We measured the success rate of tasks completed within a 2-minute interval and observed a 3.7% increase in throughput with this PR. @luoyuliuyin, would like to know how we can replicate the scenario under which your throughput increased by 50%. > > When I did performance testing, I also used a dag composed of a simple BashOperator. You may not see the difference when the load is relatively low. When I tested, the load of the airflow scheduler was very high. There were 40,000 scheduled dags (every 2 hours) and 10,000 manually triggered dags. The number of schedulers was 30, and the number of workers was 100 At the beginning, there may be no obvious difference in the task throughput of the two implementation solutions. After a long time (such as 1 day), the throughput of the existing implementation solution will gradually decrease. What I compare is the stable result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
