Very interesting - also will take a look shortly - and great how it seems to tap into SLA urgency indeed..
On Mon, May 4, 2026 at 10:08 PM Przemysław Mirowski <[email protected]> wrote: > +1 for Weighted Aging and SLA Urgency proposition. > ________________________________ > From: Elad Kalif <[email protected]> > Sent: 30 April 2026 12:08 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation > > I love the idea of dynamic priorities and I think this is a good direction. > > On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected]> wrote: > > > Thanks, this re-drafted version looks interesting. I’m trying to > > internalise and understand the new proposal. I’l leave a few > > comments/questions on the confluence page as I go. > > > > (I have to say, the move away from epic stored procedure def is a welcome > > change!) > > > > -ash > > > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> wrote: > > > > > > Hello community, I want to spark again a discussion that was held in > the > > > past and delayed (due to the focus on releasing airflow 3.2), now that > > 3.2 > > > is released, I think it might be a good Idea to bring up the discussion > > > again. > > > > > > Wiki: > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > > > In the current situation, tasks may starve in airflow, and in large > scale > > > deployments (hundreds of thousands of tasks (or more) per day) we tend > to > > > experience severe starvation, where a group of tasks may starve other > > > tasks, not allowing them to run, as described in the wiki. > > > > > > After the february devcall, where I have proposed the AIP, a few > comments > > > have arised, and so I had begun to research again about different > > > scheduling algorithms and I have added to the considerations as part of > > the > > > AIP. > > > > > > As of now, the state of the AIP is where there are a few ideas proposed > > > (some of which are pretty similar to each other, while others are quite > > > different), as the main concern from the devcall was that the > approaches > > > given might not be the best way to solve the issue, as it is a very > hard > > > problem to solve. > > > > > > After that, I have made some edits to the AIP and to the propositions, > in > > > order to help decide and clarify the advantages and disadvantages of > each > > > approach. > > > The current "best approach" can be found here here > > > < > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > > >, > > > where the new proposed algorithms are defined here > > > < > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > > > . > > > > > > In order to continue with the effort, a community consensus needs to be > > > reached about the preferred solution/solutions, once this is done, it > is > > > possible to go on and implement + stress test the proposed solution/s. > > > > > > I would appreciate a review from community members, moreover, I would > > also > > > appreciate any new propositions or improvements which can be done. > > > > > > Best regards > > > Natanel. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
