[ https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262562#comment-17262562 ]
zhuqi commented on YARN-10504: ------------------------------ [~wangda] [~bteke] When used the absolute resource in the autoCreatedLeafQueue, the logic is : 1.Initialize the update absolute template: {code:java} protected AutoCreatedLeafQueueConfig.Builder initializeLeafQueueConfigs() throws IOException { .... //Update the absolute template if (this.capacityConfigType.equals(CapacityConfigType.ABSOLUTE_RESOURCE)) { updateQueueCapacities(queueCapacities); } ... builder.capacities(queueCapacities); return builder; } //get resource value in private Resource internalGetLabeledResourceRequirementForQueue(String queue, String label, Set<String> resourceTypes, String suffix) { ... if (matcher.find()) { // Get the sub-group. String subGroup = matcher.group(0); if (subGroup.trim().isEmpty()) { return Resources.none(); } subGroup = subGroup.substring(1, subGroup.length() - 1); for (String kvPair : subGroup.trim().split(",")) { String[] splits = kvPair.split("="); // Ensure that each sub string is key value pair separated by '='. if (splits != null && splits.length > 1) { updateResourceValuesFromConfig(resourceTypes, resource, splits); } } } ... return resource; } // Update in ManagedParentQueue's updateQueueCapacities private void updateQueueCapacities(QueueCapacities queueCapacities) { for (String label : queueCapacities.getExistingNodeLabels()) { queueCapacities.setCapacity(label, this.csContext.getResourceCalculator().divide( this.csContext.getClusterResource(), this.csContext.getConfiguration().getMinimumResourceRequirement( label, this.csContext.getConfiguration() .getAutoCreatedQueueTemplateConfPrefix(getQueuePath()), resourceTypes), getQueueResourceQuotas().getConfiguredMinResource(label))); Resource childMaxResource = this.csContext.getConfiguration() .getMaximumResourceRequirement(label, this.csContext.getConfiguration() .getAutoCreatedQueueTemplateConfPrefix(getQueuePath()), resourceTypes); Resource parentMaxRes = getQueueResourceQuotas() .getConfiguredMaxResource(label); Resource effMaxResource = Resources.min( this.csContext.getResourceCalculator(), this.csContext.getClusterResource(), childMaxResource.equals(Resources.none()) ? parentMaxRes : childMaxResource, parentMaxRes); queueCapacities.setMaximumCapacity( label, this.csContext.getResourceCalculator().divide( this.csContext.getClusterResource(), effMaxResource, getQueueResourceQuotas().getConfiguredMaxResource(label))); queueCapacities.setAbsoluteCapacity( label, queueCapacities.getCapacity(label) * getQueueCapacities().getAbsoluteCapacity(label)); queueCapacities.setAbsoluteMaximumCapacity(label, queueCapacities.getMaximumCapacity(label) * getQueueCapacities().getAbsoluteMaximumCapacity(label)); } } {code} 2. Now, the capacity has been updated to absolute resource based value in addChildQueue: It back to the absolute resource in : setEffectiveMinResource {code:java} public void mergeCapacities(QueueCapacities capacities) { for ( String nodeLabel : capacities.getExistingNodeLabels()) { queueCapacities.setCapacity(nodeLabel, capacities.getCapacity(nodeLabel)); queueCapacities.setAbsoluteCapacity(nodeLabel, capacities .getAbsoluteCapacity(nodeLabel)); queueCapacities.setMaximumCapacity(nodeLabel, capacities .getMaximumCapacity(nodeLabel)); queueCapacities.setAbsoluteMaximumCapacity(nodeLabel, capacities .getAbsoluteMaximumCapacity(nodeLabel)); Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel, csContext.getClusterResource()); getQueueResourceQuotas().setEffectiveMinResource(nodeLabel, Resources.multiply(resourceByLabel, queueCapacities.getAbsoluteCapacity(nodeLabel))); getQueueResourceQuotas().setEffectiveMaxResource(nodeLabel, Resources.multiply(resourceByLabel, queueCapacities .getAbsoluteMaximumCapacity(nodeLabel))); } }{code} The effective resource have been updated to autoCreatedLeafQueue already. 3. Then, in LeafQueue's updateClusterResource: {code:java} public void updateClusterResource(Resource clusterResource, ResourceLimits currentResourceLimits) { writeLock.lock(); try { ... super.updateEffectiveResources(clusterResource); ...} }{code} We can change the updateEffectiveResources , to get consistent result to orgin: {code:java} void updateEffectiveResources(Resource clusterResource) { Set<String> configuredNodelabels = csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath()); for (String label : configuredNodelabels) { Resource resourceByLabel = labelManager.getResourceByLabel(label, clusterResource); Resource minResource = queueResourceQuotas.getConfiguredMinResource( label); // Update effective resource (min/max) to each child queue. if (getCapacityConfigType().equals( ... //The round loss caused by here CapacityConfigType.ABSOLUTE_RESOURCE) && !(this instanceof AutoCreatedLeafQueue)) { ... } else{ queueResourceQuotas.setEffectiveMinResource(label, Resources .multiply(resourceByLabel, queueCapacities.getAbsoluteCapacity(label))); queueResourceQuotas.setEffectiveMaxResource(label, Resources .multiply(resourceByLabel, queueCapacities.getAbsoluteMaximumCapacity(label))); } } } {code} If you any advice for this. Thanks. > Implement weight mode in Capacity Scheduler > ------------------------------------------- > > Key: YARN-10504 > URL: https://issues.apache.org/jira/browse/YARN-10504 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Benjamin Teke > Assignee: Benjamin Teke > Priority: Major > Attachments: YARN-10504.001.patch, YARN-10504.002.patch, > YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, > YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, > YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, > YARN-10504.ver-2.patch, YARN-10504.ver-3.patch > > > To allow the possibility to flexibly create queues in Capacity Scheduler a > weight mode should be introduced. The existing \{{capacity }}property should > be used with a different syntax, i.e: > root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0 > root.users.capacity = 1.0w > root.users.capacity = w:1.0 > Weight support should not impact the existing functionality. > > The new functionality should: > * accept and validate the new weight values > * enforce a singular mode on the whole queue tree > * (re)calculate the relative (percentage-based) capacities based on the > weights during launch and every time the queue structure changes -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org