[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748766#comment-15748766 ]
Jason Lowe commented on YARN-5889: ---------------------------------- +1 for keeping the user limit behavior consistent with what it does today. Resources will still be offered to apps in priority order (typically FIFO unless overridden by the app's priority), and any user under the minimum limit should still be able to receive that resource. As such I'm not sure I agree with the proposal for active user limits. It appears that limit can go below the configured minimum if a lot of users are all asking for the available resources. That would change the behavior and affect app priorities beyond what the user limit does today. For purposes of unblocking the user limit preemption work, I'd like to preserve the current computation so adding the preemption feature doesn't affect the behavior of the scheduler even when the feature isn't used. Speaking of preemption, we will need to be careful on how the two limits are applied to prevent oscillations. If resources are available then the active user limit should be applied as it is today. However if resources are not available then the preemption limit should be applied not only to the user being targeted for preemption but also to the user who is requesting the resource to trigger the preemption. If we mix the two limits then preemption can oscillate between two users because the active limit can be larger than the preemption limit. For calculating the preemption all-user-limit, I think the equation can be simplified to absolute queue capacity * MULP. I personally do not want preemptions to give users resources beyond the minimum. For example if the MULP is configured to 25% and there are two users in the queue, A at 70% usage and B at 30%, I'd rather not lose work by shooting A's containers to give B resources beyond the configured minimum limit. Preemptions can be very costly to an application, so I don't think this should be completely fair (that's the job of FairScheduler). We should only preempt work to get users up to the minimum limit and only if others are above the minimum limit. Thoughts on this? > Improve user-limit calculation in capacity scheduler > ---------------------------------------------------- > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Reporter: Sunil G > Assignee: Sunil G > Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, > YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org