We're just finializing the AIP document around scheduler HA and hope to have it published by the end of this week.
DagFileProcessor is a new class, but the code in it is not new -- it's the same, gradually evolving scheduler and parsing code, but refactored/moved to live elsewhere, right? -ash On Feb 27 2020, at 5:48 am, Kamil Breguła <kamil.breg...@polidea.com> wrote: > Hello, > > In my opinion, it will be easier because the database will be less loaded, > but more importantly. I added wait_for_update in one method, which means > that another scheduler will not be able to damage the state of the > database. Most of these changes even streamline HA, because more things are > done atomically/in one transaction. > > I wonder what are the plans for HA. If this is based on distributing files > to several nodes automatically, then this should not be a problem. These > changes focus mainly on DagFileProcessor. This is a fairly new class (I > proposed creating it), which contains code responsible for processing one > file. > > Best regards, > Kamil > > On Thu, Feb 27, 2020 at 6:36 AM Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > Hey, I wanted to echo the awesomeness once more, but also bring up the > > question as to whether any of this work may make it harder to distribute / > > HA the scheduler down the line (?) > > > > I almost started analyzing the code and thought it'd just be easier to ask > > the authors. > > > > Max > > On Wed, Feb 26, 2020 at 11:44 AM Felix Uellendall <felue...@pm.me.invalid> > > wrote: > > > > > Really awesome indeed! I hadn’t enough time to look into all of it yet > > but > > > will definitely do. > > > > > > Thanks Polidea and Databand for your efforts you put into this! Kamil > > > especially! > > > > > > -Felix > > > Sent from ProtonMail Mobile > > > On Wed, Feb 26, 2020 at 08:54, Sumit Maheshwari <sumeet.ma...@gmail.com> > > > wrote: > > > > > > > Awesome work guys!! Kudos to all of you 👏 > > > > On Wed, Feb 26, 2020 at 6:59 AM Jiajie Zhong < > > zhongjiajie...@hotmail.com > > > > > > > > wrote: > > > > > Good work! Thanks Kamil > > > > > Best Wish > > > > > — Jiajie > > > > > > > > > > >
