I think along with "Improve Webserver Performance" we should solve the serialization and task execution isolation problems a little bit more completely since I imagine there could be backwards compatibility issues. e.g. mapping each task JSON to a Docker image or other serialized representation that workers would then consume. See the attached PDF, AIP-24 is a subset of the DAG Definition Serialization work, but in my opinion we should still work on DAG Isolation too. My only concern is that the scope is too big for 2.0.
cc @Sumit Maheshwari <smaheshw...@twitter.com> who is also looking at tackling some of these problems. On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor <a...@apache.org> wrote: > I'm also in favour of py-test (and it's what I use for most of my > development) which is why I created > https://issues.apache.org/jira/browse/AIRFLOW-4863, but I don't think > non-user-facing/impacting changes need to go on the road map. > > -ash > > > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <tomasz.urbas...@polidea.com> > wrote: > > > > I am thinking about proposing migration from nosetest to pytest because > > it's "more up to date". I have even a POC but a lot of test fails due to > > probably side effects. > > > > Best, > > Tomek > > > > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > >> That formatted very badly in plain text. The list was: > >> > >> • Knative Executor (AIP-25, currently draft. Being worked on by > >> Daniel Imberman ) > >> • Improve Webserver performance (AIP-24, currently draft. Being > >> worked on by myself, Kaxil Naik and Zhou Fang) > >> • Enhanced real-time UI > >> • Improve Scheduler performance > >> • Extend/finish the API (AIP-13 is part of this, but not all) > >> • Production Docker image + Helm chart > >> > >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor <a...@apache.org> wrote: > >>> > >>> Hi everyone, > >>> > >>> I'd like to start working on a concrete plan to get Airflow 2.0 out, > and > >> as a result I've started updating > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 > >>> > >>> In addition to all the tidy up work ("spring cleaning", finish tidy up > >> after dropping Py2 etc) I'd propose the following 6 high level items: > >>> > >>> Knative Executor (AIP-25, currently draft. Being worked on by Daniel > >> Imberman ) > >>> Improve Webserver performance (AIP-24, currently draft. Being worked on > >> by myself, Kaxil Naik and Zhou Fang) > >>> Enhanced real-time UI > >>> Improve Scheduler performance > >>> Extend/finish the API (AIP-13 is part of this, but not all) > >>> Production Docker image + Helm chart > >>> We at Astronomer are committing to work on these in roughly this order > >> if no one else gets to them first. I also propose that we create SIGs > >> (Special Interest Groups) in slack with weekly/fortnightly (every 14 > days) > >> "calls"/update sessions. We already have #sig-ui and > #sig-dag-serialisation. > >>> > >>> This roadmap is also not a promise that all of these will be done > before > >> Airflow 2.0 - we may decide later to push something back to v2.1 etc. > >>> > >>> Does anyone disagree strongly with these priorities, or have anything > >> they want to add that you are willing to work on? > >>> > >>> Thanks, > >>> Ash > >> > >> > > > > -- > > > > Tomasz Urbaszek > > Polidea <https://www.polidea.com/> | Junior Software Engineer > > > > M: +48 505 628 493 <+48505628493> > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > Unique Tech > > Check out our projects! <https://www.polidea.com/our-work> > >
