Some time ago I did a very quick and rough POC integrating scheduler with Fenzo just to evaluate how it may fit into our architecture. Surprisingly, it took just a few lines [1] to plug it in (without constraints support of course, which is a much bigger effort). There was also a big chunk of functionality missing that would allow us to support task preemption within Fenzo. I spoke with Sharma back then and he had intention to explore adding it. Not sure where it ended up though.
I still think we should explore Fenzo integration idea as it will unlock various rich bin packing algorithms we may never get to implement. I have heard other Mesos frameworks (besides internal Netflix proprietary schedulers) are actively exploring Fenzo [2]. It's interesting to see how Fenzo could help us reduce online fragmentation and support features like soft constraints out of the box. That said, I don't think Fenzo should be considered as a sole scheduling algorithm but rather as a supported alternative (at least in the first iteration). If anything, Fenzo onboarding exercise may help us make scheduling architecture more modular and pluggable. I agree with Bill though that adding Fenzo into Aurora should not be a self-goal but rather a well justified effort that clearly outlines the benefits. I'd suggest to whoever takes on exploring this route start with an Aurora list discussion first. Thanks, Maxim [1] - https://github.com/maxim111333/incubator-aurora/blob/fenzo_proto/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L210-L249 [2] - https://github.com/twosigma/Cook On Tue, Dec 29, 2015 at 9:43 AM, Dobromir Montauk <dmont...@twitter.com.invalid> wrote: > For AWS-based Aurora installations having 'tight bin packing' is critical > for shutting down unused machines. I believe Fenzo helps with that, right? > > On Tue, Dec 29, 2015 at 9:31 AM, Bill Farner <wfar...@apache.org> wrote: > >> It would also be helpful to capture the goals we hope to achieve with the >> integration. We should also assess the risk of bringing it in, >> specifically that we won't be left maintaining it (currently it's 100% >> owned by netflix, mostly 1 developer AFAICT). >> >> On Tue, Dec 29, 2015 at 4:01 AM, Erb, Stephan <stephan....@blue-yonder.com >> > >> wrote: >> >> > Someone also expressed interest in Fenzo by adding it to the >> > community-driven roadmap [1]. >> > >> > AFAIK nobody has looked at in in detail, yet. Or at least nobody has >> > posted about it on the mailinglist. Feel free to be that someone and >> take a >> > closer look at what would be necessary to leverage the power of Fenzo in >> > Aurora :-) >> > >> > Without checking, I would assume that the following areas might need the >> > most effort to address: >> > * the preemption mechanism of Aurora to make room for priority/production >> > jobs >> > * the work-in-progress feature using oversubscribed resources [2] >> > >> > Regards, >> > Stephan >> > >> > [1] >> > >> https://docs.google.com/document/d/1vyhTZSlEPeibQm2_7HK6JXOkydO0ZllZNQZ2O3cC4_0/edit?usp=sharing >> > [2] >> > >> https://docs.google.com/document/d/1r1WCHgmPJp5wbrqSZLsgtxPNj3sULfHrSFmxp2GyPTo/edit?pref=2&pli=1#heading=h.af56b6bntcao >> > >> > ________________________________________ >> > From: Mauricio Garavaglia <mauriciogaravag...@gmail.com> >> > Sent: Tuesday, December 29, 2015 6:02 AM >> > To: dev@aurora.apache.org >> > Subject: AURORA-1440 Evaluate Fenzo scheduling library >> > >> > Hello, >> > >> > I found the issue AURORA-1440 >> > <https://issues.apache.org/jira/browse/AURORA-1440> about evaluating >> fenzo >> > to replace the scheduling algorithm. Has there been any progress on it? >> is >> > the intention just replace the implementation or also expose part of the >> > configurations that fenzo allows? >> > >> > It would be handy to be able to select, for example, between cpu bin >> > packing and memory bin packing. But also take advantage of the rich >> > scheduling constraints api that it has. I assume a lot of discussion >> would >> > be needed regarding how to expose them though :) >> > Thanks >> > >> > >> > Mauricio >> > >>
