Looks there is typo in configuration..? should be <name>yarn.nodemanager.aux-services</name>.. Upload yarn-site.xml and nodemanager logs
<property> <name>mapreduceyarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> From: Mike IT Expert [mailto:mikeitexp...@gmail.com] Sent: Sunday, June 16, 2019 8:30 PM To: Wanqiang Ji <wanqiang...@gmail.com> Cc: user <user@hadoop.apache.org> Subject: Re: hadoop-2.8.5 Container launch failed for container_1560691739419_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist I am not sure how to do that so I did buy stop and start scripts as below: Please let me know how, I am new to this.... even a link reference wold be very helpful. Thanks [Selection_054.png] hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager localhost: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 no proxyserver to stop hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-dfs.sh Stopping namenodes on [localhost] localhost: stopping namenode localhost: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0<http://0.0.0.0>: stopping secondarynamenode hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-dfs.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-namenode-mike-laptop.out localhost: starting datanode, logging to /home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-datanode-mike-laptop.out Starting secondary namenodes [0.0.0.0] 0.0.0.0<http://0.0.0.0>: starting secondarynamenode, logging to /home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-secondarynamenode-mike-laptop.out hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-resourcemanager-mike-laptop.out localhost: starting nodemanager, logging to /home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-nodemanager-mike-laptop.out hadoop@mike-laptop:/tmp/kmeans/kmeans2$ On Sun, Jun 16, 2019 at 10:42 AM Wanqiang Ji <wanqiang...@gmail.com<mailto:wanqiang...@gmail.com>> wrote: Do you forgot restart the NodeManager service when updated the yarn-site.xml? On Sun, Jun 16, 2019 at 5:17 PM Mike IT Expert <mikeitexp...@gmail.com<mailto:mikeitexp...@gmail.com>> wrote: Dear All, I like to run a python mapreduce sample program. I use the below command and I get an exception as provided below: I googled a few solutions they suggest to update yarn-site.xml as below but I still get the same exception message. Please let me know what you think. Thank you hadoop-2.8.5/etc/hadoop/yarn-site.xml ################################ as below: <configuration> <property> <name>mapreduceyarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration> Command and Output #################### hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh 19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead. packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py, /tmp/kmeans/line_cust/lineCustReduceGroup.py, /tmp/hadoop-unjar6079850917434748483/] [] /tmp/streamjob3876527625248696490.jar tmpDir=null 19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032> 19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032> 19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to process : 1 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560691739419_0001 19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application application_1560691739419_0001 19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1560691739419_0001/ 19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001 19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running in uber mode : false 19/06/16 14:32:08 INFO mapreduce.Job: map 0% reduce 0% 19/06/16 14:32:09 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000000_0, Status : FAILED Container launch failed for container_1560691739419_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:09 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000001_0, Status : FAILED Container launch failed for container_1560691739419_0001_01_000003 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:11 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000000_1, Status : FAILED Container launch failed for container_1560691739419_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:12 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000001_1, Status : FAILED Container launch failed for container_1560691739419_0001_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:14 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000000_2, Status : FAILED Container launch failed for container_1560691739419_0001_01_000006 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:15 INFO mapreduce.Job: Task Id : attempt_1560691739419_0001_m_000001_2, Status : FAILED Container launch failed for container_1560691739419_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/06/16 14:32:17 INFO mapreduce.Job: map 100% reduce 100% 19/06/16 14:32:17 INFO mapreduce.Job: Job job_1560691739419_0001 failed with state FAILED due to: Task failed task_1560691739419_0001_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 19/06/16 14:32:17 INFO mapreduce.Job: Counters: 14 Job Counters Failed map tasks=7 Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=7 Other local map tasks=5 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=5 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5 Total time spent by all reduce tasks (ms)=0 Total vcore-milliseconds taken by all map tasks=5 Total vcore-milliseconds taken by all reduce tasks=0 Total megabyte-milliseconds taken by all map tasks=5120 Total megabyte-milliseconds taken by all reduce tasks=0 19/06/16 14:32:17 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!
