[ https://issues.apache.org/jira/browse/MAPREDUCE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980455#comment-13980455 ]
Hudson commented on MAPREDUCE-5837: ----------------------------------- SUCCESS: Integrated in Hadoop-trunk-Commit #5565 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5565/]) MAPREDUCE-5837. MRAppMaster fails when checking on uber mode. Contributed by Haohui Mai. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1589903) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java > MRAppMaster fails when checking on uber mode > -------------------------------------------- > > Key: MAPREDUCE-5837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5837 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Haohui Mai > Assignee: Haohui Mai > Priority: Critical > Fix For: 2.5.0 > > Attachments: MAPREDUCE-5837.000.patch > > > When the MRAppMaster determines whether the job should run in the uber mode, > it call {{Class.forName()}} to check whether the class is derived from > {{ChainMapper}}: > {code} > try { > String mapClassName = conf.get(MRJobConfig.MAP_CLASS_ATTR); > if (mapClassName != null) { > Class<?> mapClass = Class.forName(mapClassName); > if (ChainMapper.class.isAssignableFrom(mapClass)) > isChainJob = true; > } > } catch (ClassNotFoundException cnfe) { > // don't care; assume it's not derived from ChainMapper > } > {code} > The problem here is that {{Class.forName()}} can also throw > {{NoClassDefError}}. It happens when the additional dependent jar is > unavailable to the MRAppMaster. For example, the MRAppMaster complains about > a MR job on Scala: > {noformat} > 2014-04-15 11:52:55,877 FATAL [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.NoClassDefFoundError: scala/Function1 > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:190) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.java:1282) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(JobImpl.java:1224) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3700(JobImpl.java:136) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1425) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1363) > at > org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:976) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:135) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1263) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1063) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1480) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1606) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1476) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1409) > Caused by: java.lang.ClassNotFoundException: scala.Function1 > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 22 more > {noformat} > > The proposed fix is to catch {{NoClassDefError}} at the corresponding places. -- This message was sent by Atlassian JIRA (v6.2#6252)