For what I'm looking for out of a 2.0, as an operator/cluster admin (separate from what I'd like to see as a DAG developer), I'd love to see:
- Combine breaking changes into 2.0, and do as few as possible after - A semver policy for 2.0 and onwards. (For instance we got bit hard by a breaking API change in the k8s operator) - Regularly scheduled releases (like: "minor every other month, major every other year") - A security backport policy - Pinned deps for releases - A way to get integration/cloud vendor operator updates out-of-tree, without having to pull in unrelated Airflow updates For a lot of people, Airflow is an off-the-shelf app rather than a library, but we don't actually ship or support it anything like most comparable off-the-shelf apps. It makes it much harder to support than other applications, unless you're a Python developer yourself. On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > All those are very important and we are going to work on some of them as > well. > > I think if there are breaking changes, we should rather try to fit them in > 2.0 release - at least to the point that they can be base for extending it > in later versions in backwards-compatible way (maybe then we should adopt > SemVer officially and follow it). > > J. > > > On Tue, Sep 24, 2019 at 11:52 PM James Meickle > <jmeic...@quantopian.com.invalid> wrote: > > > My question with that is, how often do we want to do major version > > increments? There's a few API breaking changes I'd love to see, but > > whether to propose them for 2.0 depends on what the wait until 3.0 looks > > like (or whether we'll allow more minor version breakages in the future) > > > > On Tue, Sep 24, 2019, 11:44 Dan Davydov <ddavy...@twitter.com.invalid> > > wrote: > > > > > I think along with "Improve Webserver Performance" we should solve the > > > serialization and task execution isolation problems a little bit more > > > completely since I imagine there could be backwards compatibility > issues. > > > e.g. mapping each task JSON to a Docker image or other serialized > > > representation that workers would then consume. See the attached PDF, > > > AIP-24 is a subset of the DAG Definition Serialization work, but in my > > > opinion we should still work on DAG Isolation too. My only concern is > > that > > > the scope is too big for 2.0. > > > > > > cc @Sumit Maheshwari <smaheshw...@twitter.com> who is also looking at > > > tackling some of these problems. > > > > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > > > > >> I'm also in favour of py-test (and it's what I use for most of my > > >> development) which is why I created > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I don't think > > >> non-user-facing/impacting changes need to go on the road map. > > >> > > >> -ash > > >> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek < > > tomasz.urbas...@polidea.com> > > >> wrote: > > >> > > > >> > I am thinking about proposing migration from nosetest to pytest > > because > > >> > it's "more up to date". I have even a POC but a lot of test fails > due > > to > > >> > probably side effects. > > >> > > > >> > Best, > > >> > Tomek > > >> > > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor <a...@apache.org> > > >> wrote: > > >> > > > >> >> That formatted very badly in plain text. The list was: > > >> >> > > >> >> • Knative Executor (AIP-25, currently draft. Being worked on > > by > > >> >> Daniel Imberman ) > > >> >> • Improve Webserver performance (AIP-24, currently draft. > > Being > > >> >> worked on by myself, Kaxil Naik and Zhou Fang) > > >> >> • Enhanced real-time UI > > >> >> • Improve Scheduler performance > > >> >> • Extend/finish the API (AIP-13 is part of this, but not > all) > > >> >> • Production Docker image + Helm chart > > >> >> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor <a...@apache.org> > wrote: > > >> >>> > > >> >>> Hi everyone, > > >> >>> > > >> >>> I'd like to start working on a concrete plan to get Airflow 2.0 > out, > > >> and > > >> >> as a result I've started updating > > >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 > > >> >>> > > >> >>> In addition to all the tidy up work ("spring cleaning", finish > tidy > > up > > >> >> after dropping Py2 etc) I'd propose the following 6 high level > items: > > >> >>> > > >> >>> Knative Executor (AIP-25, currently draft. Being worked on by > Daniel > > >> >> Imberman ) > > >> >>> Improve Webserver performance (AIP-24, currently draft. Being > worked > > >> on > > >> >> by myself, Kaxil Naik and Zhou Fang) > > >> >>> Enhanced real-time UI > > >> >>> Improve Scheduler performance > > >> >>> Extend/finish the API (AIP-13 is part of this, but not all) > > >> >>> Production Docker image + Helm chart > > >> >>> We at Astronomer are committing to work on these in roughly this > > order > > >> >> if no one else gets to them first. I also propose that we create > SIGs > > >> >> (Special Interest Groups) in slack with weekly/fortnightly (every > 14 > > >> days) > > >> >> "calls"/update sessions. We already have #sig-ui and > > >> #sig-dag-serialisation. > > >> >>> > > >> >>> This roadmap is also not a promise that all of these will be done > > >> before > > >> >> Airflow 2.0 - we may decide later to push something back to v2.1 > etc. > > >> >>> > > >> >>> Does anyone disagree strongly with these priorities, or have > > anything > > >> >> they want to add that you are willing to work on? > > >> >>> > > >> >>> Thanks, > > >> >>> Ash > > >> >> > > >> >> > > >> > > > >> > -- > > >> > > > >> > Tomasz Urbaszek > > >> > Polidea <https://www.polidea.com/> | Junior Software Engineer > > >> > > > >> > M: +48 505 628 493 <+48505628493> > > >> > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > >> > > > >> > Unique Tech > > >> > Check out our projects! <https://www.polidea.com/our-work> > > >> > > >> > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
