[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269888#comment-17269888 ]
zhuqi commented on YARN-10590: ------------------------------ cc [~leftnoteasy] [~sunilg] [~bteke] When i deep into the loss of the caculation about {code:java} // TODO: Wangda, I think this is a wrong test, it doesn't consider rounding // loss of multiplication, the right value should be <10240, 2>, but the // test expects <10240, 1> {code} I think the test logic should change to : {code:java} // When in absolute mode. // We can only assertEquals in memory size, because the cap/absolute // is caculated by memory, we can only make sure this. if (leafQueue.getCapacityConfigType() == AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) { assertEquals(effMinCapacity.getMemorySize(), leafQueue.getEffectiveCapacity(label).getMemorySize()); assertTrue(effMinCapacity.getVirtualCores() <= leafQueue.getEffectiveCapacity(label).getVirtualCores()); } else { assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label)); } {code} Because in absolute mode with auto created leaf queue, we get the cap/absolute by : {code:java} private void updateQueueCapacities(QueueCapacities queueCapacities) { for (String label : queueCapacities.getExistingNodeLabels()) { queueCapacities.setCapacity(label, this.csContext.getResourceCalculator().divide( this.csContext.getClusterResource(), this.csContext.getConfiguration().getMinimumResourceRequirement( label, this.csContext.getConfiguration() .getAutoCreatedQueueTemplateConfPrefix(getQueuePath()), resourceTypes), getQueueResourceQuotas().getConfiguredMinResource(label))); .... queueCapacities.setAbsoluteCapacity( label, queueCapacities.getCapacity(label) * getQueueCapacities().getAbsoluteCapacity(label)); ... } } {code} In default resource caculator, we get the cap/absolute only by memory: {code:java} public float ratio(Resource a, Resource b) { return divideSafelyAsFloat(a.getMemorySize(), b.getMemorySize()); } {code} This cause the loss of multiply by cap/absolute, with the actual effective min resource. The actual effective min resource will be updated in : {code:java} // Update effective resource (min/max) to each child queue. if (getCapacityConfigType().equals( CapacityConfigType.ABSOLUTE_RESOURCE)) { queueResourceQuotas.setEffectiveMinResource(label, getMinResourceNormalized(queuePath, ((ParentQueue) parent).getEffectiveMinRatioPerResource(), minResource)); }{code} We will get the actual effective min resource, different with the get the cap/absolute only by memory. So in absolute auto created leaf queue , the effective min resource only can meet the memory caculated by cap/absolute with default resource caculator. The logic is wright, but the test we should fix. Thoughts? Thanks. > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > ------------------------------------------------------------------------------------- > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: zhuqi > Assignee: zhuqi > Priority: Major > Attachments: YARN-10590.001.patch > > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org