----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6665/ -----------------------------------------------------------
(Updated March 11, 2013, 9:59 p.m.) Review request for mesos and Benjamin Hindman. Changes ------- Updated to Ben M's review. Description ------- Currently, every time we do an allocation we have to traverse the entire list of active slaves an check each one to see if its whitelisted and if it has resources to allocate. This patch keeps a set of all slaves that meet those requirements, and updates it when slaves are added/removed and when resources are allocated/recovered. Timing comparisons, using test_framework on a local cluster on my laptop (each number is an average over 10 tests): 100 slaves, 100 tasks without patch: 5.82s with patch: 5.773s improvement of about 1% 1000 slaves, 100 tasks without patch: 8.261s with patch: 8.07s improvement of about 2% Since this was a scalability issue, you'd presumably see a bigger improvement in larger clusters but its difficult to simulate more than 1000 slaves or so locally. If more convincing numbers are needed I can do some testing on EC2 (if nothing else, I'm hoping we'll have a Jenkins build with automated performance tests set up later this semester, so I can test this in the process of setting that up), but at the very least it seems clear that the patch improves the runtime complexity of the allocation algorithm and doesn't slow allocations down even in small clusters. Diffs (updated) ----- src/local/local.hpp 2633d25 src/local/local.cpp 3402029 src/master/allocator.hpp b68b67d src/master/drf_sorter.hpp 79566cc src/master/drf_sorter.cpp fe31a00 src/master/hierarchical_allocator_process.hpp c1d6f54 src/master/main.cpp ca0abec src/master/master.hpp c9b4b3f src/master/master.cpp 814a6e1 src/master/sorter.hpp 73db6b1 src/tests/allocator_tests.cpp b953cd1 src/tests/allocator_zookeeper_tests.cpp 3f70202 src/tests/fault_tolerance_tests.cpp 61509f1 src/tests/gc_tests.cpp fbdd6d6 src/tests/master_detector_tests.cpp 787ba19 src/tests/master_tests.cpp 2ba14fc src/tests/resource_offers_tests.cpp 44eaf0d src/tests/sorter_tests.cpp 61e6038 src/tests/utils.hpp d3efa58 third_party/libprocess/third_party/stout/include/stout/stringify.hpp 136316d Diff: https://reviews.apache.org/r/6665/diff/ Testing ------- make check Thanks, Thomas Marshall
