[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-04-06 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated MAPREDUCE-6647:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
   Status: Resolved  (was: Patch Available)

Thanks Haibo.  Committed to trunk!

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, 
> mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-28 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.004.patch

updated per Robert's comments.

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, 
> mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-03 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.003.patch

updated to take care of checkstyle issue

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, 
> mapreduce6647.003.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-03 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.002.patch

updated patch to take care of test failures in TestRecovery

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-02 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Status: Patch Available  (was: Open)

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-02 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.001.patch

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-02 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: (was: mapreduce6647.001.patch)

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-02 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.001.patch

> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>   YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
> duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
>   jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired 
> * duration);
>   jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * 
> mbRequired);
>   jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
> vcoresRequired);
>   jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated

2016-03-02 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6647:
--
Description: 
As can be seen in the following snippet, the MR counters for usage use the 
resources requested instead of the resources allocated. The scheduler 
increment-allocation-mb configs could lead to these values not being the same. 
We could change the counters to use the allocated resources in order to account 
for this.

{code}
  private static void updateMillisCounters(JobCounterUpdateEvent jce,
  TaskAttemptImpl taskAttempt) {
 /***omitted**/
long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
int mbRequired =
taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);

int minSlotMemSize = taskAttempt.conf.getInt(
  YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
  YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);

int simSlotsRequired =
minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
/ minSlotMemSize);

if (taskType == TaskType.MAP) {
  jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
duration);
  jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
  jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
vcoresRequired);
  jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
} else {
  jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * 
duration);
  jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
  jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
vcoresRequired);
  jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
}
{code}


  was:
As can be seen in the following snippet, the MR counters for usage use the 
resources requested instead of the resources allocated. The scheduler 
increment-allocation-mb configs could lead to these values not being the same. 
We could change the counters to use the allocated resources in order to account 
for this.

{code}
  private static void updateMillisCounters(JobCounterUpdateEvent jce,
  TaskAttemptImpl taskAttempt) {
   /***omitted**/
  long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
int mbRequired =
taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);

int minSlotMemSize = taskAttempt.conf.getInt(
  YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
  YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);

int simSlotsRequired =
minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
/ minSlotMemSize);

if (taskType == TaskType.MAP) {
  jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * 
duration);
  jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
  jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * 
vcoresRequired);
  jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
} else {
  jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * 
duration);
  jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
  jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * 
vcoresRequired);
  jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
}
{code}



> MR usage counters use the resources requested instead of the resources 
> allocated
> 
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> As can be seen in the following snippet, the MR counters for usage use the 
> resources requested instead of the resources allocated. The scheduler 
> increment-allocation-mb configs could lead to these values not being the 
> same. We could change the counters to use the allocated resources in order to 
> account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>   TaskAttemptImpl taskAttempt) {
>  /***omitted**/
> long duration = (taskAttempt.getFinishTime() - 
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, 
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
>   YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>