Team,
We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance' on Confluence and would very much appreciate feedback and suggestions from the community. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 The original AIP was filed by Xiaodong Deng on March 2nd, 2019 and has stalled after a while, so with his blessing, we are taking the baton on this AIP. We at Astronomer have heard several enterprises ask for both High Availability as well as greater scalability, specifically around starting hundreds and thousands of tasks in a very short time window. We would like to attempt this based on our experience running Airflow as a Service and deploying Airflow at enterprises around the globe. We believe that this will benefit Airflow and fuel greater adoption of Airflow for production pipelines within enterprises. Building on the original AIP, we have proposed an active/active model, where we can scale schedulers, but are staying away from the quorum approach. Xiaodong Deng had put in some really good thinking about the problem including approaches towards reducing contention between multiple schedulers and we have included some of those concepts here. Additional commenters had discussed the possibilities of leader selection and those challenges, and we have incorporated their thinking as well. . Any feedback, suggestions, and comments would be greatly appreciated. Best Regards, Ash Berlin-Taylor and Vikram Koka
