[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322572#comment-16322572
]
Manikandan R edited comment on YARN-7159 at 1/11/18 5:18 PM:
-------------------------------------------------------------
[~sunilg] Fixed those 2 junit failures. Though
{{TestFairSchedulerConfiguration}} changes has fixed the junit issue, requires
you to check in detail because of following flow:
{code}
assertEquals(customResourceInformation(20000L, ""),
- calculator.normalize(customResource(10001L, ""), min, max, increment)
+ calculator.normalize(customResource(19999L, ""), min, max, increment)
.getResourceInformation(A_CUSTOM_RESOURCE));
{code}
In the above junit test case, {{increment}} has been derived from
{{FairSchedulerConfiguration#getIncrementAllocation}} which returns 10k as o/p.
{code} conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource" +
FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here,
value has been configured as 10 with "" units, but
{{ResourceUtils#createResourceTypesArray}} triggered from
{{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's
containing A_CUSTOM_RESOURCE with value as 10k, as it simply copies the object
and modifies only the value. Is this correct behaviour? Based on our earlier
discussion, conversion should happen if there is units difference and source
unit is higher than the unit at RM/server side. Because of this, old code in
{{DominantResourceCalculator#normalize}} was working properly for the above
junit test case.
was (Author: maniraj...@gmail.com):
[~sunilg] Fixed those 2 junit failures. Though
{{TestFairSchedulerConfiguration}} changes has fixed the junit issue, requires
you to check in detail because of following flow:
{code}
assertEquals(customResourceInformation(20000L, ""),
- calculator.normalize(customResource(10001L, ""), min, max, increment)
+ calculator.normalize(customResource(19999L, ""), min, max, increment)
.getResourceInformation(A_CUSTOM_RESOURCE));
{code}
In the above junit test case, {{increment}} has been derived from
{{FairSchedulerConfiguration#getIncrementAllocation}} which returns 10k as o/p.
{code} conf.set(YarnConfiguration.RESOURCE_TYPES + ".a-custom-resource" +
FairSchedulerConfiguration.INCREMENT_ALLOCATION, "10");{code}. Here,
value has been configured as 10 with "" units, but
{{ResourceUtils#createResourceTypesArray}} triggered from
{{FairSchedulerConfiguration#getIncrementAllocation}} returns array of RI's
containing A_CUSTOM_RESOURCE with value as 10k, as it simply copies the object
and modifies only the value. Is this correct behaviour? Based on our earlier
discussion, conversion should happen if there is units difference and source
unit is higher than the unit at RM/server side. Because of this, old code
DominantResourceCalculator#normalize was working properly for the above junit
test case.
> Normalize unit of resource objects in RM and avoid to do unit conversion in
> critical path
> -----------------------------------------------------------------------------------------
>
> Key: YARN-7159
> URL: https://issues.apache.org/jira/browse/YARN-7159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Wangda Tan
> Assignee: Manikandan R
> Priority: Critical
> Attachments: YARN-7159.001.patch, YARN-7159.002.patch,
> YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch,
> YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch,
> YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch,
> YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch,
> YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch,
> YARN-7159.019.patch, YARN-7159.020.patch, YARN-7159.021.patch
>
>
> Currently resource conversion could happen in critical code path when
> different unit is specified by client. This could impact performance and
> throughput of RM a lot. We should do unit normalization when resource passed
> to RM and avoid expensive unit conversion every time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
