[ https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated MAPREDUCE-6647: ------------------------------------- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Thanks Haibo. Committed to trunk! > MR usage counters use the resources requested instead of the resources > allocated > -------------------------------------------------------------------------------- > > Key: MAPREDUCE-6647 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Haibo Chen > Assignee: Haibo Chen > Fix For: 2.9.0 > > Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, > mapreduce6647.003.patch, mapreduce6647.004.patch > > > As can be seen in the following snippet, the MR counters for usage use the > resources requested instead of the resources allocated. The scheduler > increment-allocation-mb configs could lead to these values not being the > same. We could change the counters to use the allocated resources in order to > account for this. > {code} > private static void updateMillisCounters(JobCounterUpdateEvent jce, > TaskAttemptImpl taskAttempt) { > /***omitted**/ > long duration = (taskAttempt.getFinishTime() - > taskAttempt.getLaunchTime()); > int mbRequired = > taskAttempt.getMemoryRequired(taskAttempt.conf, taskType); > int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, > taskType); > int minSlotMemSize = taskAttempt.conf.getInt( > YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, > YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); > int simSlotsRequired = > minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired > / minSlotMemSize); > if (taskType == TaskType.MAP) { > jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * > duration); > jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired); > jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * > vcoresRequired); > jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration); > } else { > jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired > * duration); > jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * > mbRequired); > jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * > vcoresRequired); > jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)