Ah. Thanks! It just popped as the "first" when I looked at it I will mark it as [ARCHIVED} and move to the archive. So my point about updates are not really valid :). It's rather detailed and updated from what I see.
J. On Fri, Sep 11, 2020 at 2:08 PM Kaxil Naik <[email protected]> wrote: > Just a note here: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651> > is > the correct link for the AIP proposed for Scheduler HA. The other link was > an old proposal from someone else. > > On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <[email protected]> > wrote: > > > I started to feel that we need to clarify statements about the HA > Scheduler > > for our community. Not that I am losing sleep regularly over this but it > > did keep me away last night when I started to think about it :). > > > > I have a feeling that while we already defined some - rather aggressive - > > timelines for 2.0, the subject of HA Scheduler was not touched in the > > previous Airflow 2.0 meetings. We are not very far from the release but > the > > HA scheduler is implemented inside Astronomer and we have not seen any > code > > for it yet in the community. I understand that a lot of work (not only > > development but especially testing) has been put into it from the > > Astronomer team internally. > > > > I am actually quite OK with that to be like that. I think Astronomer is a > > super-valuable member of the community and I have no doubts Ash and Kaxil > > and Daniel and others will do an awesome job with it. I am simply afraid > > that when we see it, some of the cases that we see as needed by the > > community will require more work. This will either delay the 2.0 release > or > > we will have to drop it from the 2.0 release. Looking at the number of > > discussions we had with - much simpler IMHO - Smart Sensors, I have the > > feeling that HA scheduler will spark even more discussions. The AIP-15 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler > > > > > was > > not very rich in detail and the last time it has been updated was March > > 2019 (!) and I have no doubt a big number of design decisions, > > observations, learning has happened in Astronomer since. > > > > And to be perfectly honest - I am ok with both of the scenarios I can > see: > > > > 1) We release HA Scheduler in 2.0 > > > > For that, I think we should start looking at the code and discuss it > > already quite some time ago IMHO. It might be too late if we want to fit > > the aggressive timeline we have - especially that there are other things > > the most active people are committing to for 2.0 and they might simply > not > > have enough time to make the quality review rounds and discussions. I > think > > we need to see it first to be even able to assess if we think we can make > > it within the timeline. > > > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer > will > > use the HA Scheduler in their own service as a "commercial" add-on or > > "advantage" of their offering. > > > > In the meantime - between 2.0 and 2.1 Astronomer could donate the code > and > > we could make sure it is reviewed and merged in the way that answers the > > needs of different community members. This has also numerous advantages > to > > the community - similar to the case of Smart Sensors, Astronomer can test > > it in production then and solve all the teething problems of such a > > service. > > > > I cannot speak for the business models of Astronomer of course :), but it > > seems to me like a nice advantage to have for a while, from the business > > point of view. And as a community, we also benefit that we have such a > > strong member of the community with a sustainable and good business > model. > > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially > > (but also others) - Airflow would not be where it is today. And I would > be > > 100% happy with such an approach as a PMC and member of the community > and I > > support it a lot if Astronomer chooses this path. > > > > I think, however, it's the highest time that we decide and clearly > > communicate it to the users as a community. At least I have a feeling > that > > without the community members, committers, and some heavy users being > > involved in the open, and having time for quality review and discussion, > > releasing HA in 2.0 might be not possible. And to just reiterate - this > has > > nothing to do with the expected quality of the code and testing, but more > > about potential differences in expectations, assumptions, understanding, > > performance limitations, and anything else that might (and usually does) > > come up. > > > > I think - since we already started to publish the schedule, this is the > > right time that we make a decision on that and align expectations. > > > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a > > company for the HA Scheduler are? I know as a group of committers we said > > it a number of times that HA Scheduler will be in 2.0 so we built the > > expectations among our users as a community. But maybe you really think > > that pursuing scenario 2) (or maybe another scenario I have not thought > > about) is the way to go for Astronomer? > > > > As I wrote above - I am personally perfectly fine with either of the > > scenarios, and I think they are both beneficial for the community, but I > > think we should discuss it, align expectations, and clearly communicate > as > > the Apache Airflow community. > > > > J. > > > > -- > > Jarek Potiuk > > Polidea | Principal Software Engineer > > > > M: +48 660 796 129 > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
