Ran into an interesting situation relating to framework app in hdfs using
mapreduce.application.framework.path.
Had mapreduce.application.framework.path configured with the
mapreduce.tar.gz#mr-framework and mapreduce.application.classpath referring
to the mr-framework directories.
Later when attempted to withdraw this in the job client (where
mapreduce.application.framework.path was set to empty and
mapreduce.application.classpath set to jars in hadoop installation on the
hosts) the MRAppMaster was throwing this
java.lang.IllegalArgumentException: Could not locate MapReduce framework
name 'mr-framework' in mapreduce.application.classpath
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:228)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:248)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.getInitialClasspath(TaskAttemptImpl.java:620)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:755)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:812)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1527)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1504)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1069)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:145)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1311)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1303)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:745)
Turns out that mapreduce.application.classpath was being sourced via
job.xml while the mapreduce.application.framework.path was being sourced
from node local mapred-site.xml (given the mapreduce.framework.path wasn't
present in the job.xml). This meant that it wasn't possible to withdraw
localizing framework app from hdfs without fixing the config in all the
nodes.
Question to this group is, Aren't these configs app specific ones ? Why is
it necessary to load the default config in MRAppMaster instead of just
working with job.xml? Specific section of code of interest:
MRAppMaster appMaster =
new MRAppMaster(applicationAttemptId, containerId, nodeHostString,
Integer.parseInt(nodePortString),
Integer.parseInt(nodeHttpPortString), appSubmitTime);
ShutdownHookManager.get().addShutdownHook(
new MRAppMasterShutdownHook(appMaster), SHUTDOWN_HOOK_PRIORITY);
JobConf conf = new JobConf(new YarnConfiguration());
conf.addResource(new Path(MRJobConfig.JOB_CONF_FILE));
Another observation is that the job.xml visible via the UI keeps showing
the job.xml as provided by the user, however the conf.xml in job-staging
directory under user's home had the merged config from
node::mapred-site.xml + job.xml
FWIW, adding a :/mr-framework/* to mapreduce.application.classpath allows
you to wriggle out of this situation.
Regards
Srikanth Sundarrajan
