[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621659#comment-16621659
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 9/20/18 8:39 AM:
-----------------------------------------------------------------------

Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param maxResourceCapacityResource 
getNormalizedResource(Resource requestedResource, Resource 
maxResourceCapability);{code}
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 


was (Author: bsteinbach):
Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param 
maxResourceCapacityResource getNormalizedResource(Resource requestedResource, 
Resource maxResourceCapability);{code}
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 

> Limit container sizes per queue in FairScheduler
> ------------------------------------------------
>
>                 Key: YARN-8468
>                 URL: https://issues.apache.org/jira/browse/YARN-8468
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.1.0
>            Reporter: Antal Bálint Steinbach
>            Assignee: Antal Bálint Steinbach
>            Priority: Critical
>         Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, 
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch, 
> YARN-8468.014.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:
>  
> All the infrastructure is already in the code. We need to do the following:
>  * add the setting to the queue properties for all queue types (parent and 
> leaf), this will cover dynamically created queues.
>  * if we set it on the root we override the scheduler setting and we should 
> not allow that.
>  * make sure that queue resource cap can not be larger than scheduler max 
> resource cap in the config.
>  * implement getMaximumResourceCapability(String queueName) in the 
> FairScheduler
>  * implement getMaximumResourceCapability() in both FSParentQueue and 
> FSLeafQueue as follows
>  * expose the setting in the queue information in the RM web UI.
>  * expose the setting in the metrics etc for the queue.
>  * write JUnit tests.
>  * update the scheduler documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to