The goal would be to support both MySQL and PostgreSQL for production as we know many of Airflow users use MySQL as Metadata DB.
On Tue, Mar 3, 2020 at 12:25 PM Ash Berlin-Taylor <[email protected]> wrote: > It _shouldn't_, and we will test extensively with mysql. > > Worse case is we'll have to fall back to managing the lock ourselves with > a column rather than relying on db/row level locks. This might be a case > where we have different/specialised behaviour for different dbs, or even db > versions, if say mysql 8 behaves okay but 5.7/5.6 doesn't. > > Ash > > On 3 March 2020 07:01:15 GMT-05:00, "Kamil Breguła" < > [email protected]> wrote: > >Hello, > > > >Will reliance on the database cause problems with MySQL? A lot of my > >users use this database. I am afraid that the lock mechanism in MySQL > >is much less stable and predictable than PostgresSQL and this can > >cause various stability problems. I know that Astronomer uses > >PostgreSQL, but Airflow supports RDMS in a production environment and > >both must work properly in this AIP. > > > >Best regards, > >Kamil > > > >On Tue, Mar 3, 2020 at 12:50 PM Kaxil Naik <[email protected]> wrote: > >> > >> Good work on the Proposal Ash & Vikram. > >> > >> > >> > >> On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka > ><[email protected]> > >> wrote: > >> > >> > Team, > >> > > >> > > >> > > >> > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better > >> > Scheduling Performance' on Confluence and would very much > >appreciate > >> > feedback and suggestions from the community. > >> > > >> > > >> > > >> > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > >> > > >> > > >> > > >> > The original AIP was filed by Xiaodong Deng on March 2nd, 2019 and > >has > >> > stalled after a while, so with his blessing, we are taking the > >baton on > >> > this AIP. We at Astronomer have heard several enterprises ask for > >both High > >> > Availability as well as greater scalability, specifically around > >starting > >> > hundreds and thousands of tasks in a very short time window. > >> > > >> > > >> > > >> > We would like to attempt this based on our experience running > >Airflow as a > >> > Service and deploying Airflow at enterprises around the globe. We > >believe > >> > that this will benefit Airflow and fuel greater adoption of Airflow > >for > >> > production pipelines within enterprises. > >> > > >> > > >> > > >> > Building on the original AIP, we have proposed an active/active > >model, > >> > where we can scale schedulers, but are staying away from the quorum > >> > approach. Xiaodong Deng had put in some really good thinking about > >the > >> > problem including approaches towards reducing contention between > >multiple > >> > schedulers and we have included some of those concepts here. > >Additional > >> > commenters had discussed the possibilities of leader selection and > >those > >> > challenges, and we have incorporated their thinking as well. . > >> > > >> > > >> > > >> > Any feedback, suggestions, and comments would be greatly > >appreciated. > >> > > >> > > >> > > >> > Best Regards, > >> > > >> > > >> > Ash Berlin-Taylor and Vikram Koka > >> > >
