It _shouldn't_, and we will test extensively with mysql. Worse case is we'll have to fall back to managing the lock ourselves with a column rather than relying on db/row level locks. This might be a case where we have different/specialised behaviour for different dbs, or even db versions, if say mysql 8 behaves okay but 5.7/5.6 doesn't.
Ash On 3 March 2020 07:01:15 GMT-05:00, "Kamil Breguła" <[email protected]> wrote: >Hello, > >Will reliance on the database cause problems with MySQL? A lot of my >users use this database. I am afraid that the lock mechanism in MySQL >is much less stable and predictable than PostgresSQL and this can >cause various stability problems. I know that Astronomer uses >PostgreSQL, but Airflow supports RDMS in a production environment and >both must work properly in this AIP. > >Best regards, >Kamil > >On Tue, Mar 3, 2020 at 12:50 PM Kaxil Naik <[email protected]> wrote: >> >> Good work on the Proposal Ash & Vikram. >> >> >> >> On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka ><[email protected]> >> wrote: >> >> > Team, >> > >> > >> > >> > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better >> > Scheduling Performance' on Confluence and would very much >appreciate >> > feedback and suggestions from the community. >> > >> > >> > >> > >https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 >> > >> > >> > >> > The original AIP was filed by Xiaodong Deng on March 2nd, 2019 and >has >> > stalled after a while, so with his blessing, we are taking the >baton on >> > this AIP. We at Astronomer have heard several enterprises ask for >both High >> > Availability as well as greater scalability, specifically around >starting >> > hundreds and thousands of tasks in a very short time window. >> > >> > >> > >> > We would like to attempt this based on our experience running >Airflow as a >> > Service and deploying Airflow at enterprises around the globe. We >believe >> > that this will benefit Airflow and fuel greater adoption of Airflow >for >> > production pipelines within enterprises. >> > >> > >> > >> > Building on the original AIP, we have proposed an active/active >model, >> > where we can scale schedulers, but are staying away from the quorum >> > approach. Xiaodong Deng had put in some really good thinking about >the >> > problem including approaches towards reducing contention between >multiple >> > schedulers and we have included some of those concepts here. >Additional >> > commenters had discussed the possibilities of leader selection and >those >> > challenges, and we have incorporated their thinking as well. . >> > >> > >> > >> > Any feedback, suggestions, and comments would be greatly >appreciated. >> > >> > >> > >> > Best Regards, >> > >> > >> > Ash Berlin-Taylor and Vikram Koka >> >
