[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.
[ https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913777#comment-16913777 ] Benjamin Mahler commented on MESOS-9806: {noformat} commit de90b2b3078e06975ab2061db821cfe7dda8 Author: Benjamin Mahler Date: Thu Aug 22 17:41:28 2019 -0400 Optimized Resources::shrink. Master: *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 30.37 secs Made 0 allocation in 27.05 secs Master + this patch: *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 24.15 secs Made 0 allocation in 20.48 secs Review: https://reviews.apache.org/r/71353 {noformat} {noformat} commit 05e5ca4b3446e34447f632463efe9a34b4bace7f Author: Benjamin Mahler Date: Thu Aug 22 17:42:57 2019 -0400 Added ResourceQuantities::fromScalarResource. Master + previous patches: *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 24.15 secs Made 0 allocation in 20.48 secs Master + previous patches + this patch: Master + this patch: *HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 23.37 secs Made 0 allocation in 19.72 secs Review: https://reviews.apache.org/r/71354 {noformat} > Address allocator performance regression due to the removal of quota role > sorter. > - > > Key: MESOS-9806 > URL: https://issues.apache.org/jira/browse/MESOS-9806 > Project: Mesos > Issue Type: Improvement > Components: allocation >Reporter: Meng Zhu >Assignee: Meng Zhu >Priority: Critical > Labels: resource-management > > In MESOS-9802, we removed the quota role sorter which is tech debt. > However, this slows down the allocator. The problem is that in the first > stage, even though a cluster might have no active roles with non-default > quota, the allocator will now have to sort and go through each and every role > in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, > the allocator could experience ~50% performance degradation. > There are a couple of ways to address this issue. For example, we could make > the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return > all the roles with non-default quota. Alternatively, an even better approach > would be to deprecate the sorter concept and just have two standalone > functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree > structure (not yet exist in the allocator) and return the sorted roles. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.
[ https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913709#comment-16913709 ] Benjamin Mahler commented on MESOS-9806: {noformat} commit b6c87d7c44346b2497ace65b1d2060ee423aa772 Author: Benjamin Mahler Date: Wed Aug 21 20:10:42 2019 -0400 Eliminated double lookups in the allocator. Review: https://reviews.apache.org/r/71345 {noformat} {noformat} commit 790c4e72e1460035b13bf27f2cb8999709e9767e Author: Benjamin Mahler Date: Wed Aug 21 20:11:31 2019 -0400 Avoid duplicate allocatableTo call in the allocator. Review: https://reviews.apache.org/r/71346 {noformat} > Address allocator performance regression due to the removal of quota role > sorter. > - > > Key: MESOS-9806 > URL: https://issues.apache.org/jira/browse/MESOS-9806 > Project: Mesos > Issue Type: Improvement > Components: allocation >Reporter: Meng Zhu >Assignee: Meng Zhu >Priority: Critical > Labels: resource-management > > In MESOS-9802, we removed the quota role sorter which is tech debt. > However, this slows down the allocator. The problem is that in the first > stage, even though a cluster might have no active roles with non-default > quota, the allocator will now have to sort and go through each and every role > in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, > the allocator could experience ~50% performance degradation. > There are a couple of ways to address this issue. For example, we could make > the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return > all the roles with non-default quota. Alternatively, an even better approach > would be to deprecate the sorter concept and just have two standalone > functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree > structure (not yet exist in the allocator) and return the sorted roles. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.
[ https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912823#comment-16912823 ] Benjamin Mahler commented on MESOS-9806: https://reviews.apache.org/r/71345/ https://reviews.apache.org/r/71346/ https://reviews.apache.org/r/71347/ Master branch: {noformat} HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 36.1185929secs Made 0 allocation in 32.62218602secs {noformat} After all patches in review chain: {noformat} HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2 Made 3500 allocations in 21.389381617secs Made 0 allocation in 18.593000222secs {noformat} > Address allocator performance regression due to the removal of quota role > sorter. > - > > Key: MESOS-9806 > URL: https://issues.apache.org/jira/browse/MESOS-9806 > Project: Mesos > Issue Type: Improvement > Components: allocation >Reporter: Meng Zhu >Assignee: Meng Zhu >Priority: Critical > Labels: resource-management > > In MESOS-9802, we removed the quota role sorter which is tech debt. > However, this slows down the allocator. The problem is that in the first > stage, even though a cluster might have no active roles with non-default > quota, the allocator will now have to sort and go through each and every role > in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, > the allocator could experience ~50% performance degradation. > There are a couple of ways to address this issue. For example, we could make > the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return > all the roles with non-default quota. Alternatively, an even better approach > would be to deprecate the sorter concept and just have two standalone > functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree > structure (not yet exist in the allocator) and return the sorted roles. -- This message was sent by Atlassian Jira (v8.3.2#803003)