Dear All,
I like to run a python mapreduce sample program. I use the below command
and I get an exception as provided below:
I googled a few solutions they suggest to update yarn-site.xml as below but
I still get the same exception message.
Please let me know what you think.
Thank you
hadoop-2.8.5/etc/hadoop/yarn-site.xml
################################
as below:
<configuration>
<property>
<name>mapreduceyarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
Command and Output
####################
hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh
19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated,
please use generic option -files instead.
packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py,
/tmp/kmeans/line_cust/lineCustReduceGroup.py,
/tmp/hadoop-unjar6079850917434748483/] []
/tmp/streamjob3876527625248696490.jar tmpDir=null
19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at /
0.0.0.0:8032
19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at /
0.0.0.0:8032
19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to process
: 1
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1560691739419_0001
19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application
application_1560691739419_0001
19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job:
http://localhost:8088/proxy/application_1560691739419_0001/
19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001
19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running in
uber mode : false
19/06/16 14:32:08 INFO mapreduce.Job: map 0% reduce 0%
19/06/16 14:32:09 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000000_0, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000002 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:09 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000001_0, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000003 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:11 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000000_1, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000004 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:12 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000001_1, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000005 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:14 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000000_2, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000006 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:15 INFO mapreduce.Job: Task Id :
attempt_1560691739419_0001_m_000001_2, Status : FAILED
Container launch failed for container_1560691739419_0001_01_000007 :
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
at
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/06/16 14:32:17 INFO mapreduce.Job: map 100% reduce 100%
19/06/16 14:32:17 INFO mapreduce.Job: Job job_1560691739419_0001 failed
with state FAILED due to: Task failed task_1560691739419_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
19/06/16 14:32:17 INFO mapreduce.Job: Counters: 14
Job Counters
Failed map tasks=7
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=7
Other local map tasks=5
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=5
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5
Total time spent by all reduce tasks (ms)=0
Total vcore-milliseconds taken by all map tasks=5
Total vcore-milliseconds taken by all reduce tasks=0
Total megabyte-milliseconds taken by all map tasks=5120
Total megabyte-milliseconds taken by all reduce tasks=0
19/06/16 14:32:17 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
