[
https://issues.apache.org/jira/browse/MAPREDUCE-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224974#comment-14224974
]
Zhijie Shen commented on MAPREDUCE-6168:
----------------------------------------
Now I can recall the context before: MR Client 2.2 and 2.3 is not able to talk
to MR 2.4+ Runtime (including the rencently released 2.6), because more
counters are introduced into MR 2.4. In MAPREDUCE-5831, we finally choose to
fix the client, such that in even later version, if we introduce more counters,
MR Client 2.6+ is able to be compatible to them. However, for MR 2.2 and 2.3,
the problem is not fixed unless we back port the fix to these two versions, or
we fix the runtime not to emit the new counters to the old client. As to the
latter option, it's going to be hard because we don't have the client version
information in the communication protocol. I'm Okay if we resolve this issue as
"won't fix".
> Old MR client is still broken when receiving new counters from MR job
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-6168
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6168
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Zhijie Shen
> Assignee: Junping Du
> Priority: Blocker
>
> In the following scenarios:
> 1. Either insecure or secure;
> 2. MR 2.2 with new shuffle on NM;
> 3. Submitting via old client.
> We will see the following console exception:
> {code}
> 14/11/17 14:56:19 INFO mapreduce.Job: Job job_1416264695865_0003 completed
> successfully
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
> at java.lang.Enum.valueOf(Enum.java:236)
> at
> org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
> at
> org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
> at
> org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> The problem is supposed to be fixed by MAPREDUCE-5831, however, it seems that
> we haven't cover all the problematic code path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
