----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51027/#review146591 -----------------------------------------------------------
This patch really improved the performance a lot for allocator, I did some test with 6000 frameworks and 1000 agents (Add 6000 frameworks first then add 1000 agents), the result is quite promising! Please take a look at the result of adding 1000 agents, the time decreased from `4.6 mins` to `49 secs`! It would help a lot for master failover with thousands of frameworks and agents. Before fix: ``` [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.DeclineOffers/7 Using 1000 agents and 6000 frameworks Added 6000 frameworks in 702547us Added 1000 agents in 4.60214636666667mins ``` After fix: ``` [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.DeclineOffers/7 Using 1000 agents and 6000 frameworks Added 6000 frameworks in 89707us Added 1000 agents in 49.351196secs ``` - Guangya Liu On 八月 23, 2016, 8:49 a.m., Jacob Janco wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51027/ > ----------------------------------------------------------- > > (Updated 八月 23, 2016, 8:49 a.m.) > > > Review request for mesos, Benjamin Mahler, Guangya Liu, James Peach, Klaus > Ma, and Jiang Yan Xu. > > > Bugs: MESOS-3157 > https://issues.apache.org/jira/browse/MESOS-3157 > > > Repository: mesos > > > Description > ------- > > - Triggered allocations dispatch allocate() only > if there is no pending allocation in the queue. > - Allocation candidates are accumulated and only > cleared when enqueued allocations are processed. > - Batched allocations are handled synchronously. > > - Carrying over work from https://reviews.apache.org/r/41658/ and added the > previous reviewers > - Specifically, this patch introduces the boolean flag pendingAllocation, > which when set on event > triggered allocations, will prevent additional no-op allocations: the flag > is cleared when > the enqueued allocation is processed, subsequent event triggered > allocations will update a set > of allocation candidates rather than dispatching an additional allocate(). > > > Diffs > ----- > > src/master/allocator/mesos/hierarchical.hpp > bdbc6d3b5b959990538f4e3b7b1a3b031d9aea05 > src/master/allocator/mesos/hierarchical.cpp > 234ef98529964a0b6d3f132426a6c8ccbb1263ee > > Diff: https://reviews.apache.org/r/51027/diff/ > > > Testing > ------- > > make check > > note: check without filters depends on https://reviews.apache.org/r/51028 > > With new benchmark https://reviews.apache.org/r/49617: > Sample output without 51027: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 > Using 10000 agents and 3000 frameworks > Added 3000 frameworks in 57251us > Added 10000 agents in 3.21345353333333mins > allocator settled after 1.61236038333333mins > [ OK ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 > (290578 ms) > > Sample output with 51027: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 > Using 10000 agents and 3000 frameworks > Added 3000 frameworks in 39817us > Added 10000 agents in 3.22860541666667mins > allocator settled after 25.525654secs > [ OK ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 > (220137 ms) > > > Thanks, > > Jacob Janco > >
