[ https://issues.apache.org/jira/browse/YARN-10033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997701#comment-16997701 ]
Eric Payne commented on YARN-10033: ----------------------------------- Thanks a lot for the review, [~ebadger]. Yes, I agree that it's good to have these additional changes for completeness since future tests may need to use the dominant resource calculator. However, I don't think we need changes in {{parseResourceDetails}}. The if/else clause, already checks whether or not the unit test wants to use the default resource calculator (resource.length == 1) or the dominant resource calculator (resource.length != 1). > TestProportionalCapacityPreemptionPolicy not initializing vcores for > effective max resources > -------------------------------------------------------------------------------------------- > > Key: YARN-10033 > URL: https://issues.apache.org/jira/browse/YARN-10033 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, test > Affects Versions: 3.3.0, 3.2.1, 3.1.3 > Reporter: Eric Payne > Assignee: Eric Payne > Priority: Major > Attachments: YARN-10033.001.patch, YARN-10033.002.patch > > > TestProportionalCapacityPreemptionPolicy#testPreemptionWithVCoreResource is > preempting more containers than would happen on a real cluster. > This is because the process for mocking CS queues in > {{TestProportionalCapacityPreemptionPolicy}} fails to take into consideration > vcores when mocking effective max resources. > This causes miscalculations for how many vcores to preempt when the DRF is > being used in the test: > {code:title=TempQueuePerPartition#offer} > Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax( > Resources.subtract(getMax(), idealAssigned), > Resource.newInstance(0, 0)); > {code} > In the above code, the preemption policy is offering resources to an > underserved queue. {{getMax()}} will use the effective max resource if it > exists. Since this test is mocking effective max resources, it will return > that value. However, since the mock doesn't include vcores, the test treats > memory as the dominant resource and awards too many preempted containers to > the underserved queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org