There's already a concept called priority_weight on tasks http://airflow.apache.org/concepts.html?highlight=priority_weight#pools (the doc about it is in relation to pools, but everything is run in a pool of "default_pool" if not specified.)
Is that what you want? On 9 October 2019 07:38:38 BST, bharath palaksha <bharath...@gmail.com> wrote: >Hi, > >Is there any discussion thread on adding priority to tasks and >cost-based >optimization? >priority and pre-emption as an option to the user. If priority is >specified, scheduler has to schedule high priority tasks and if >pre-emption >is true, it can pre-empt current running task which is of lower >priority > >Thanks, >Bharath > > >On Mon, Sep 30, 2019 at 11:19 PM James Meickle ><jmeic...@quantopian.com.invalid> wrote: > >> For what I'm looking for out of a 2.0, as an operator/cluster admin >> (separate from what I'd like to see as a DAG developer), I'd love to >see: >> >> - Combine breaking changes into 2.0, and do as few as possible after >> - A semver policy for 2.0 and onwards. (For instance we got bit hard >by a >> breaking API change in the k8s operator) >> - Regularly scheduled releases (like: "minor every other month, major >every >> other year") >> - A security backport policy >> - Pinned deps for releases >> - A way to get integration/cloud vendor operator updates out-of-tree, >> without having to pull in unrelated Airflow updates >> >> For a lot of people, Airflow is an off-the-shelf app rather than a >library, >> but we don't actually ship or support it anything like most >comparable >> off-the-shelf apps. It makes it much harder to support than other >> applications, unless you're a Python developer yourself. >> >> On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk ><jarek.pot...@polidea.com> >> wrote: >> >> > All those are very important and we are going to work on some of >them as >> > well. >> > >> > I think if there are breaking changes, we should rather try to fit >them >> in >> > 2.0 release - at least to the point that they can be base for >extending >> it >> > in later versions in backwards-compatible way (maybe then we should >adopt >> > SemVer officially and follow it). >> > >> > J. >> > >> > >> > On Tue, Sep 24, 2019 at 11:52 PM James Meickle >> > <jmeic...@quantopian.com.invalid> wrote: >> > >> > > My question with that is, how often do we want to do major >version >> > > increments? There's a few API breaking changes I'd love to see, >but >> > > whether to propose them for 2.0 depends on what the wait until >3.0 >> looks >> > > like (or whether we'll allow more minor version breakages in the >> future) >> > > >> > > On Tue, Sep 24, 2019, 11:44 Dan Davydov ><ddavy...@twitter.com.invalid> >> > > wrote: >> > > >> > > > I think along with "Improve Webserver Performance" we should >solve >> the >> > > > serialization and task execution isolation problems a little >bit more >> > > > completely since I imagine there could be backwards >compatibility >> > issues. >> > > > e.g. mapping each task JSON to a Docker image or other >serialized >> > > > representation that workers would then consume. See the >attached PDF, >> > > > AIP-24 is a subset of the DAG Definition Serialization work, >but in >> my >> > > > opinion we should still work on DAG Isolation too. My only >concern is >> > > that >> > > > the scope is too big for 2.0. >> > > > >> > > > cc @Sumit Maheshwari <smaheshw...@twitter.com> who is also >looking >> at >> > > > tackling some of these problems. >> > > > >> > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor ><a...@apache.org> >> > > wrote: >> > > > >> > > >> I'm also in favour of py-test (and it's what I use for most of >my >> > > >> development) which is why I created >> > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I >don't >> think >> > > >> non-user-facing/impacting changes need to go on the road map. >> > > >> >> > > >> -ash >> > > >> >> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek < >> > > tomasz.urbas...@polidea.com> >> > > >> wrote: >> > > >> > >> > > >> > I am thinking about proposing migration from nosetest to >pytest >> > > because >> > > >> > it's "more up to date". I have even a POC but a lot of test >fails >> > due >> > > to >> > > >> > probably side effects. >> > > >> > >> > > >> > Best, >> > > >> > Tomek >> > > >> > >> > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor ><a...@apache.org >> > >> > > >> wrote: >> > > >> > >> > > >> >> That formatted very badly in plain text. The list was: >> > > >> >> >> > > >> >> • Knative Executor (AIP-25, currently draft. Being >worked >> on >> > > by >> > > >> >> Daniel Imberman ) >> > > >> >> • Improve Webserver performance (AIP-24, currently >draft. >> > > Being >> > > >> >> worked on by myself, Kaxil Naik and Zhou Fang) >> > > >> >> • Enhanced real-time UI >> > > >> >> • Improve Scheduler performance >> > > >> >> • Extend/finish the API (AIP-13 is part of this, but >not >> > all) >> > > >> >> • Production Docker image + Helm chart >> > > >> >> >> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor ><a...@apache.org> >> > wrote: >> > > >> >>> >> > > >> >>> Hi everyone, >> > > >> >>> >> > > >> >>> I'd like to start working on a concrete plan to get >Airflow 2.0 >> > out, >> > > >> and >> > > >> >> as a result I've started updating >> > > >> >> >https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0 >> > > >> >>> >> > > >> >>> In addition to all the tidy up work ("spring cleaning", >finish >> > tidy >> > > up >> > > >> >> after dropping Py2 etc) I'd propose the following 6 high >level >> > items: >> > > >> >>> >> > > >> >>> Knative Executor (AIP-25, currently draft. Being worked on >by >> > Daniel >> > > >> >> Imberman ) >> > > >> >>> Improve Webserver performance (AIP-24, currently draft. >Being >> > worked >> > > >> on >> > > >> >> by myself, Kaxil Naik and Zhou Fang) >> > > >> >>> Enhanced real-time UI >> > > >> >>> Improve Scheduler performance >> > > >> >>> Extend/finish the API (AIP-13 is part of this, but not >all) >> > > >> >>> Production Docker image + Helm chart >> > > >> >>> We at Astronomer are committing to work on these in >roughly this >> > > order >> > > >> >> if no one else gets to them first. I also propose that we >create >> > SIGs >> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly >(every >> > 14 >> > > >> days) >> > > >> >> "calls"/update sessions. We already have #sig-ui and >> > > >> #sig-dag-serialisation. >> > > >> >>> >> > > >> >>> This roadmap is also not a promise that all of these will >be >> done >> > > >> before >> > > >> >> Airflow 2.0 - we may decide later to push something back to >v2.1 >> > etc. >> > > >> >>> >> > > >> >>> Does anyone disagree strongly with these priorities, or >have >> > > anything >> > > >> >> they want to add that you are willing to work on? >> > > >> >>> >> > > >> >>> Thanks, >> > > >> >>> Ash >> > > >> >> >> > > >> >> >> > > >> > >> > > >> > -- >> > > >> > >> > > >> > Tomasz Urbaszek >> > > >> > Polidea <https://www.polidea.com/> | Junior Software >Engineer >> > > >> > >> > > >> > M: +48 505 628 493 <+48505628493> >> > > >> > E: tomasz.urbas...@polidea.com ><tomasz.urbasz...@polidea.com> >> > > >> > >> > > >> > Unique Tech >> > > >> > Check out our projects! <https://www.polidea.com/our-work> >> > > >> >> > > >> >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> > >>
