[ https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067012#comment-16067012 ]
Robert Kanter commented on MAPREDUCE-6647: ------------------------------------------ I think [~haibo.chen]'s example supports your use case then. If the minimum allocation is 2GB but the user requests 1GB, Yarn will give them 2GB but prior to MR-6647, they'd only be charged for 1GB, which is incorrect. > MR usage counters use the resources requested instead of the resources > allocated > -------------------------------------------------------------------------------- > > Key: MAPREDUCE-6647 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Haibo Chen > Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, > mapreduce6647.003.patch, mapreduce6647.004.patch > > > As can be seen in the following snippet, the MR counters for usage use the > resources requested instead of the resources allocated. The scheduler > increment-allocation-mb configs could lead to these values not being the > same. We could change the counters to use the allocated resources in order to > account for this. > {code} > private static void updateMillisCounters(JobCounterUpdateEvent jce, > TaskAttemptImpl taskAttempt) { > /***omitted**/ > long duration = (taskAttempt.getFinishTime() - > taskAttempt.getLaunchTime()); > int mbRequired = > taskAttempt.getMemoryRequired(taskAttempt.conf, taskType); > int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, > taskType); > int minSlotMemSize = taskAttempt.conf.getInt( > YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, > YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); > int simSlotsRequired = > minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired > / minSlotMemSize); > if (taskType == TaskType.MAP) { > jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * > duration); > jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired); > jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * > vcoresRequired); > jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration); > } else { > jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired > * duration); > jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * > mbRequired); > jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * > vcoresRequired); > jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration); > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org