Hi.
I'm trying yarn + security but still cannot make a mapred example
runing. Can anyone help me to take a look?
My env:
- 3-slave cluster on ec2. Centos 5.5
- nn, dn, rm, nm all started, with security enabled.
- i saw java.lang.NoClassDefFoundError from LinuxContainerExecutor eror
log:
./application_1330545370212_0004/container_1330545370212_0004_01_000001/stderr
- If i disable security, still saw this issue.
Any hint?
I followed the instructions from
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html
Steps:
I started a mapred sample from nn/rm:
$ usr/lib/hadoop/bin/yarn --config ./conf jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar
randomwriter 10 10
Logs are from nn, nm,
------------------------------------------
[yarn@ip-10-176-231-35 hadoop]$ /usr/lib/hadoop/bin/yarn --config ./conf
jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar
randomwriter 10 10
Running 30 maps.
Job started: Wed Feb 29 20:33:48 UTC 2012
12/02/29 20:33:48 WARN conf.Configuration:
mapred.used.genericoptionsparser is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used
12/02/29 20:33:49 INFO mapreduce.JobSubmitter: number of splits:30
12/02/29 20:33:49 INFO mapred.ResourceMgrDelegate: Submitted application
application_1330545370212_0005 to ResourceManager at
ip-10-176-231-35.us-west-1.compute.internal/10.176.231.35:7090
12/02/29 20:33:49 INFO mapreduce.Job: The url to track the job:
http://ip-10-176-231-35.us-west-1.compute.internal:7050/proxy/application_1330545370212_0005/
12/02/29 20:33:49 INFO mapreduce.Job: Running job: job_1330545370212_0005
12/02/29 20:33:53 INFO mapreduce.Job: Job job_1330545370212_0005 running
in uber mode : false
12/02/29 20:33:53 INFO mapreduce.Job: map 0% reduce 0%
12/02/29 20:33:53 INFO mapreduce.Job: Job job_1330545370212_0005 failed
with state FAILED due to: Application application_1330545370212_0005
failed 1 times due to AM Container for
appattempt_1330545370212_0005_000001 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:261)
at org.apache.hadoop.util.Shell.run(Shell.java:188)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
main : command provided 1
main : user is yarn
.Failing this attempt.. Failing the application.
12/02/29 20:33:53 INFO mapreduce.Job: Counters: 0
Job ended: Wed Feb 29 20:33:53 UTC 2012
The job took 5 seconds.
------------------------------------------
LinuxContainer error:
[root@ip-10-176-203-45 yarn]# more
./application_1330545370212_0004/container_1330545370212_0004_01_000001/stderr
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/yar
n/service/CompositeService
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14
1)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.service.Comp
ositeService
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 12 more
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
Program will exit.
----------------------------------------
