[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-07-10 Thread Szilard Nemeth (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538323#comment-16538323
 ] 

Szilard Nemeth edited comment on YARN-8468 at 7/10/18 9:41 AM:
---

Hi [~bsteinbach]!
Thanks for the patch. This is high quality code here.

I noticed a couple of things: 
- {{AllocationFileQueueParser: MAX_CONTAINER_RESOURCES}} could be 
package-private (without any modifier)
- {{QueueMaxContainerAllocationValidator.createExceptionText}}: please use 
{{String.format()}} instead of concatenating the parts of the string.
- {{QueueMaxContainerAllocationValidator}}: you used the method 
{{throwException}} 2 times, and you also used {{throw new 
YarnRuntimeException}} as is. I think you should either use the method for all 
3 invocations or just use {{throw new YarnRuntimeException()}} everywhere. I 
prefer the latter.
- {{QueueMaxContainerAllocationValidator.validate}}: I would use this kind of 
message instead: "Invalid queue resource allocation, it does not allowed to 
override " + MAX_CONTAINER_RESOURCES + " for the root queue!"
- {{QueueMaxContainerAllocationValidator.validate}}: Logging maxMem and 
maxCores on INFO level is unnecessary. I would not log these at all, even not 
on DEBUG level as it does not hold any meaningful information for the users 
like this.
- {{QueueMaxContainerAllocationValidator.checkContainerResources}}: Same as 
above, remove the logged queueMem and queueCores log statements.
- {{AllocationConfiguration.queueMaxContainerResourcesMap}}: Please add 
comments about what is this field for, as we have comments for other fields as 
well.
{{FSLeafQueue.getMaximumResourceCapability // 
FsParentQueue.getMaximumResourceCapability}}: I accidentally noticed there's a 
space missing between the "if" and the parentheses.
- {{TestQueueMaxContainerAllocationValidator}}: I think the convention is to 
use method names like {{testXXX}} so 
{{tooHighMemoryMaxContainerAllocationTest}} should change to 
{{testTooHighMemoryMaxContainerAllocation}}. In addition, I would change the 
name to {{testMaxContainerAllocationWithTooHighMemory}} and the rest of the 
methods similarly.
- {{TestQueueMaxContainerAllocationValidator}}: Please don't use 
{{QueueMaxContainerAllocationValidator.createExceptionText}} in the tests, as 
if the production code generates the text in a wrong format, then this test 
won't fail. I would simply use Strings here to assert the message.
- {{TestFairScheduler}}: Once again, the convention for method names is testXXX.
- In the {{FairScheduler.md}} documentation, I would replace "This property is 
invalid for root queue." with "This property cannot be defined for the root 
queue"

Please fix the lines longer than 80 chars, at least I saw one occurence in 
{{FairSchedulerTestBase}} and {{TestFairScheduler}}.

Thanks!


was (Author: snemeth):
Hi [~bsteinbach]!
Thanks for the patch. This is high quality code here.

I noticed a couple of things: 
- {{AllocationFileQueueParser: MAX_CONTAINER_RESOURCES}} could be 
package-private (without any modifier)
- {{QueueMaxContainerAllocationValidator.createExceptionText}}: please use 
{{String.format()}} instead of concatenating the parts of the string.
- {{QueueMaxContainerAllocationValidator}}: you used the method 
{{throwException}} 2 times, and you also used {{throw new 
YarnRuntimeException}} as is. I think you should either use the method for all 
3 invocations or just use {{throw new YarnRuntimeException()}} everywhere. I 
prefer the latter.
- {{QueueMaxContainerAllocationValidator.validate}}: I would use this kind of 
message instead: "Invalid queue resource allocation, it does not allowed to 
override " + MAX_CONTAINER_RESOURCES + " for the root queue!"
- {{QueueMaxContainerAllocationValidator.validate}}: Logging maxMem and 
maxCores on INFO level is unnecessary. I would not log these at all, even not 
on DEBUG level as it does not hold any meaningful information for the users 
like this.
- {{QueueMaxContainerAllocationValidator.checkContainerResources}}: Same as 
above, remove the logged queueMem and queueCores log statements.
- {{AllocationConfiguration.queueMaxContainerResourcesMap}}: Please add 
comments about what is this field for, as we have comments for other fields as 
well.
{{FSLeafQueue.getMaximumResourceCapability // 
FsParentQueue.getMaximumResourceCapability}}: I accidentally noticed there's a 
space missing between the "if" and the parentheses.
- {{TestQueueMaxContainerAllocationValidator}}: I think the convention is to 
use method names like {{testXXX}} so 
{{tooHighMemoryMaxContainerAllocationTest}} should change to 
{{testTooHighMemoryMaxContainerAllocation}}. In addition, I would change the 
name to {{testMaxContainerAllocationWithTooHighMemory}} and the rest of the 
methods similarly.
- {{TestQueueMaxContainerAllocationValidator}}: Please don't use 
{{QueueMaxContainerAllocationValidator

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-07-10 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538831#comment-16538831
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 7/10/18 3:54 PM:
---

Hi [~snemeth]

Thanks for the feedback. I applied all of your points except for removing 
_QueueMaxContainerAllocationValidator.createExceptionText_

from the test. I used it because I was testing if the parameters are correct 
for the exception not for validating the error message text.

 

Balint


was (Author: bsteinbach):
Hi [~snemeth]

Thanks for the feedback. I applied all of them except for removing 
_QueueMaxContainerAllocationValidator.createExceptionText_

from the test. I used it because I was testing if the parameters are correct 
for the exception not for validating the error message text.

 

Balint

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
>  Labels: patch
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:
>  
> All the infrastructure is already in the code. We need to do the following:
>  * add the setting to the queue properties for all queue types (parent and 
> leaf), this will cover dynamically created queues.
>  * if we set it on the root we override the scheduler setting and we should 
> not allow that.
>  * make sure that queue resource cap can not be larger than scheduler max 
> resource cap in the config.
>  * implement getMaximumResourceCapability(String queueName) in the 
> FairScheduler
>  * implement getMaximumResourceCapability() in both FSParentQueue and 
> FSLeafQueue as follows
>  * expose the setting in the queue information in the RM web UI.
>  * expose the setting in the metrics etc for the queue.
>  * write JUnit tests.
>  * update the scheduler documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-07-23 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552885#comment-16552885
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 7/23/18 1:47 PM:
---

Hi [~haibochen] !

I only commented "Thanks for the feedback [~wilfreds]", but I also fixed his 
suggestions. I am sorry for that, please find my responses inline.
 - a {{FSLeafQueue}} and {{FSParentQueue}} always have a parent doing a null 
check on the parent is unneeded. The only queue that does not have a parent is 
the root queue which you already have special cased. _(In some tests sub-queue 
does not have a parent)_
 - {{getMaximumResourceCapability}} must support resource types and not just 
memory and vcores, same as YARN-7556 for this setting (_It returns Resource I 
assume than it is ok with resource types_)
 - {{getMaxAllowedAllocation}} from the NodeTracker support more than just 
memory and vcores, needs to flow through (_It returns Resource I assume than it 
is ok with resource types_)
 - {{FairScheduler}}: Why change the static imports only for a part of the 
config values, either change all or none (none is preferred) (_Fixed_)
 - {{FairSchedulerPage}}: missing toString on the ResourceInfo (_added but I 
can't see why is it necessary_)
 - Testing must also use resource types not only the old configuration type 
like: "memory-mb=5120, test1=4, vcores=2" _(Test added)_
 - {{TestFairScheduler}} Testing must also include failure cases for sub queues 
not just the root queue: setting value on root queue should throw and should 
not be applied (_Fixed_)
 - If this TestQueueMaxContainerAllocationValidator is a new file make sure 
that you add the license etc (_license text added for the new files_)

Balint


was (Author: bsteinbach):
Hi [~haibochen] !

I only commented "Thanks for the feedback [~wilfreds]", but I also fixed his 
suggestions. I am sorry for that, please find my responses inline.
 - a {{FSLeafQueue}} and {{FSParentQueue}} always have a parent doing a null 
check on the parent is unneeded. The only queue that does not have a parent is 
the root queue which you already have special cased. _(In some tests sub-queue 
does not have a parent)_
 - {{getMaximumResourceCapability}} must support resource types and not just 
memory and vcores, same as YARN-7556 for this setting (_It supports Resource I 
assume than it is ok with resource types_)
 - {{getMaxAllowedAllocation}} from the NodeTracker support more than just 
memory and vcores, needs to flow through (_It supports Resource I assume than 
it is ok with resource types_)
 - {{FairScheduler}}: Why change the static imports only for a part of the 
config values, either change all or none (none is preferred) (_Fixed_)
 - {{FairSchedulerPage}}: missing toString on the ResourceInfo (_added but I 
can't see why is it necessary_)
 - Testing must also use resource types not only the old configuration type 
like: "memory-mb=5120, test1=4, vcores=2" _(Test added)_
 - {{TestFairScheduler}} Testing must also include failure cases for sub queues 
not just the root queue: setting value on root queue should throw and should 
not be applied (_Fixed_)
 - If this TestQueueMaxContainerAllocationValidator is a new file make sure 
that you add the license etc (_license text added for the new files_)

Balint

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
>  Labels: patch
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:
>  
> All the infrastructure is already in the code. We need to do the following:
>  * add the setting to the queue properties for all queue types (parent and 
> leaf), this will cover dynam

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-08-30 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596174#comment-16596174
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 8/30/18 8:23 AM:
---

Thank you [~leftnoteasy] for checking the patch. I uploaded a new one.

1) The ApplicationMaster gets the maximum allocation value here just like in 
the other case so the AM can handle it before submitting a request, but I think 
we still need the normalization/validation of the request just in case the 
application did something wrong. Furthermore, there was a 
validation/normalization step before, it was just using only the scheduler 
level maximums.

2) I reverted the formattings. I run an autoformatting unintentionally on 
SchedulerUtils.

Please let me know if I misunderstand something.


was (Author: bsteinbach):
Thank you [~leftnoteasy] for checking the patch. I uploaded a new one.

1) The ApplicationMaster gets the maximum allocation value here just like in 
the other case so the AM can handle it like before submitting a request, but I 
think we still need the normalization/validation of the request just in case 
the application did something wrong. Furthermore, there was a 
validation/normalization step before it was just using only the scheduler level 
maximums.

2) I reverted the formattings. I run an autoformatting unintentionally on 
SchedulerUtils.

Please let me know if I misunderstand something.

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:
>  
> All the infrastructure is already in the code. We need to do the following:
>  * add the setting to the queue properties for all queue types (parent and 
> leaf), this will cover dynamically created queues.
>  * if we set it on the root we override the scheduler setting and we should 
> not allow that.
>  * make sure that queue resource cap can not be larger than scheduler max 
> resource cap in the config.
>  * implement getMaximumResourceCapability(String queueName) in the 
> FairScheduler
>  * implement getMaximumResourceCapability() in both FSParentQueue and 
> FSLeafQueue as follows
>  * expose the setting in the queue information in the RM web UI.
>  * expose the setting in the metrics etc for the queue.
>  * write JUnit tests.
>  * update the scheduler documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-09-17 Thread Weiwei Yang (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617371#comment-16617371
 ] 

Weiwei Yang edited comment on YARN-8468 at 9/17/18 11:00 AM:
-

Generally took a look, it seems good. But it would be helpful if [~bsteinbach] 
can upload a up-to-date patch so I can apply to take a closer look. v12 patch 
doesn't apply anymore.

Some comments:

1) There are too many checkstyle issues need to be fixed

2) Suggest to add a NPE check in case app is null in following changes in 
PlacementConstraintProcessor
{code:java}
+ String queueName =

+        scheduler.getApplicationAttempt(appAttemptId).getQueueName();

+ Resource maxAllocation = scheduler.getMaximumResourceCapability(queueName);

{code}
Thanks


was (Author: cheersyang):
Generally took a look, it seems good. But it would be helpful if [~bsteinbach] 
can upload a up-to-date patch so I can apply to take a closer look. v12 patch 
doesn't apply anymore.

Some comments:

1) There are too many checkstyle issues need to be fixed

2) Suggest to add a NPE check in case app is null in following changes in 
PlacementConstraintProcessor

 

{code}

+ String queueName =

+        scheduler.getApplicationAttempt(appAttemptId).getQueueName();

+ Resource maxAllocation = scheduler.getMaximumResourceCapability(queueName);

{code}

Thanks

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, 
> YARN-8468.011.patch, YARN-8468.012.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:
>  
> All the infrastructure is already in the code. We need to do the following:
>  * add the setting to the queue properties for all queue types (parent and 
> leaf), this will cover dynamically created queues.
>  * if we set it on the root we override the scheduler setting and we should 
> not allow that.
>  * make sure that queue resource cap can not be larger than scheduler max 
> resource cap in the config.
>  * implement getMaximumResourceCapability(String queueName) in the 
> FairScheduler
>  * implement getMaximumResourceCapability() in both FSParentQueue and 
> FSLeafQueue as follows
>  * expose the setting in the queue information in the RM web UI.
>  * expose the setting in the metrics etc for the queue.
>  * write JUnit tests.
>  * update the scheduler documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-09-20 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621659#comment-16621659
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 9/20/18 8:38 AM:
---

Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param maxResourceCapacityResource getNormalizedResource(Resource 
requestedResource, Resource maxResourceCapability);{code}

I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 


was (Author: bsteinbach):
Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
// added param maxResourceCapacityResource getNormalizedResource(Resource 
requestedResource, Resource maxResourceCapability);
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, 
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch, 
> YARN-8468.014.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
> Suggested solution:

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-09-20 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621659#comment-16621659
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 9/20/18 8:39 AM:
---

Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param 
maxResourceCapacityResource getNormalizedResource(Resource requestedResource, 
Resource maxResourceCapability);{code}
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 


was (Author: bsteinbach):
Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param maxResourceCapacityResource getNormalizedResource(Resource 
requestedResource, Resource maxResourceCapability);{code}

I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, 
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch, 
> YARN-8468.014.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
>

[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler

2018-09-20 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621659#comment-16621659
 ] 

Antal Bálint Steinbach edited comment on YARN-8468 at 9/20/18 8:39 AM:
---

Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param maxResourceCapacityResource 
getNormalizedResource(Resource requestedResource, Resource 
maxResourceCapability);{code}
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 


was (Author: bsteinbach):
Hi [~cheersyang] ,

Some changes I made was committed in YARN-8720 . DefaultAMSProcessor, for 
example, could be reverted because of that.

I turned off the auto-formatting and the auto import organization already. 
Still, I have to format the changes I made or I will have checkstyle issues. I 
did some small changes like renaming variables from 3 letter abbreviations but 
only at that part of the code where I did some real work.

I understand why we should do "unnecessary changes" carefully, but it would be 
good to find a balance and have to possibility to fix some issues. Anyway, I 
will be much more careful about this thing in the future and try to reduce them 
once more in my patch. Somehow my IDE is against it so badly :). 

In SchedulerUtils normalizeAndValidateRequest method was overloaded 3 times. 
What I did there is reduced the number of normalizeAndValidateRequest methods 
from 4 to 2. I did that because I had to reduce the calls of 
scheduler.getMaximumResourceCapability. I did that during addressing the 
feedback of [~leftnoteasy] to reduce the calls of 
scheduler.getMaximumResourceCapability.

For the same reason, I had to change the YArnScheduler API to 
{code:java}
// added param 
maxResourceCapacityResource getNormalizedResource(Resource requestedResource, 
Resource maxResourceCapability);{code}
I have to pass around the maxResourceCapability everywhere to call 
scheduler.getMaximumResourceCapability only once per allocate/request.

I will upload a patch soon to remove more unnecessary changes. 

 

> Limit container sizes per queue in FairScheduler
> 
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch, 
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, 
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, 
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, 
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch, 
> YARN-8468.014.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" 
> to limit the overall size of a container. This applies globally to all 
> containers and cannot be limited by queue or and is not scheduler dependent.
>  
> The goal of this ticket is to allow this value to be set on a per queue basis.
>  
> The use case: User has two pools, one for ad hoc jobs and one for enterprise 
> apps. User wants to limit ad hoc jobs to small containers but allow 
> enterprise apps to request as many resources as needed. Setting 
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum 
> container size for all queues and setting maximum resources per queue with 
> “maxContainerResources” queue config value.
>  
>