Yes. Here is PR. https://github.com/apache/airflow/pull/6697 It made this work easier.
On Thu, Feb 27, 2020 at 12:18 PM Ash Berlin-Taylor <a...@apache.org> wrote: > We're just finializing the AIP document around scheduler HA and hope to > have it published by the end of this week. > > DagFileProcessor is a new class, but the code in it is not new -- it's the > same, gradually evolving scheduler and parsing code, but refactored/moved > to live elsewhere, right? > -ash > On Feb 27 2020, at 5:48 am, Kamil Breguła <kamil.breg...@polidea.com> > wrote: > > Hello, > > > > In my opinion, it will be easier because the database will be less > loaded, > > but more importantly. I added wait_for_update in one method, which means > > that another scheduler will not be able to damage the state of the > > database. Most of these changes even streamline HA, because more things > are > > done atomically/in one transaction. > > > > I wonder what are the plans for HA. If this is based on distributing > files > > to several nodes automatically, then this should not be a problem. These > > changes focus mainly on DagFileProcessor. This is a fairly new class (I > > proposed creating it), which contains code responsible for processing one > > file. > > > > Best regards, > > Kamil > > > > On Thu, Feb 27, 2020 at 6:36 AM Maxime Beauchemin < > > maximebeauche...@gmail.com> wrote: > > > > > Hey, I wanted to echo the awesomeness once more, but also bring up the > > > question as to whether any of this work may make it harder to > distribute / > > > HA the scheduler down the line (?) > > > > > > I almost started analyzing the code and thought it'd just be easier to > ask > > > the authors. > > > > > > Max > > > On Wed, Feb 26, 2020 at 11:44 AM Felix Uellendall > <felue...@pm.me.invalid> > > > wrote: > > > > > > > Really awesome indeed! I hadn’t enough time to look into all of it > yet > > > but > > > > will definitely do. > > > > > > > > Thanks Polidea and Databand for your efforts you put into this! Kamil > > > > especially! > > > > > > > > -Felix > > > > Sent from ProtonMail Mobile > > > > On Wed, Feb 26, 2020 at 08:54, Sumit Maheshwari < > sumeet.ma...@gmail.com> > > > > wrote: > > > > > > > > > Awesome work guys!! Kudos to all of you 👏 > > > > > On Wed, Feb 26, 2020 at 6:59 AM Jiajie Zhong < > > > zhongjiajie...@hotmail.com > > > > > > > > > > wrote: > > > > > > Good work! Thanks Kamil > > > > > > Best Wish > > > > > > — Jiajie > > > > > > > > > > > > > > > > > >
