RE: hadoop-2.8.5 Container launch failed for container_1560691739419_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

2019-06-20 Thread Brahma Reddy Battula 
Looks there is typo in configuration..?  should be 
yarn.nodemanager.aux-services.. Upload yarn-site.xml and 
nodemanager logs


mapreduceyarn.nodemanager.aux-services
mapreduce_shuffle



From: Mike IT Expert [mailto:mikeitexp...@gmail.com]
Sent: Sunday, June 16, 2019 8:30 PM
To: Wanqiang Ji 
Cc: user 
Subject: Re: hadoop-2.8.5 Container launch failed for 
container_1560691739419_0001_01_02 : 
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The 
auxService:mapreduce_shuffle does not exist

I am not sure how to do that so I did buy stop and start scripts as below:

Please let me know how, I am new to this even a link reference wold be very 
helpful.

Thanks

[Selection_054.png]

hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
localhost: nodemanager did not stop gracefully after 5 seconds: killing with 
kill -9
no proxyserver to stop
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-dfs.sh
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-namenode-mike-laptop.out
localhost: starting datanode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-datanode-mike-laptop.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-secondarynamenode-mike-laptop.out
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to 
/home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-resourcemanager-mike-laptop.out
localhost: starting nodemanager, logging to 
/home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-nodemanager-mike-laptop.out
hadoop@mike-laptop:/tmp/kmeans/kmeans2$



On Sun, Jun 16, 2019 at 10:42 AM Wanqiang Ji 
mailto:wanqiang...@gmail.com>> wrote:
Do you forgot restart the NodeManager service when updated the yarn-site.xml?

On Sun, Jun 16, 2019 at 5:17 PM Mike IT Expert 
mailto:mikeitexp...@gmail.com>> wrote:
Dear All,

I like to run a python mapreduce sample program. I use the below command and I 
get an exception as provided below:

I googled a few solutions they suggest to update yarn-site.xml as below but I 
still get the same exception message.

Please let me know what you think.

Thank you


hadoop-2.8.5/etc/hadoop/yarn-site.xml

as below:



mapreduceyarn.nodemanager.aux-services
mapreduce_shuffle

  
yarn.nodemanager.aux-services.mapreduce_shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
  

Command and Output

hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh
19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated, please 
use generic option -files instead.
packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py, 
/tmp/kmeans/line_cust/lineCustReduceGroup.py, 
/tmp/hadoop-unjar6079850917434748483/] [] /tmp/streamjob3876527625248696490.jar 
tmpDir=null
19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at 
/0.0.0.0:8032
19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at 
/0.0.0.0:8032
19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to process : 1
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
job_1560691739419_0001
19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application 
application_1560691739419_0001
19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job: 
http://localhost:8088/proxy/application_1560691739419_0001/
19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001
19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running in 
uber mode : false
19/06/16 14:32:08 INFO mapreduce.Job:  map 0% reduce 0%
19/06/16 14:32:09 INFO mapreduce.Job: Task Id : 
attempt_1560691739419_0001_m_00_0, Status : FAILED
Container launch failed for container_1560691739419_0001_01_02 : 
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The 
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at

Re: hadoop-2.8.5 Container launch failed for container_1560691739419_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

2019-06-16 Thread Wanqiang Ji 
Do you forgot restart the NodeManager service when updated the
yarn-site.xml?

On Sun, Jun 16, 2019 at 5:17 PM Mike IT Expert 
wrote:

> Dear All,
>
> I like to run a python mapreduce sample program. I use the below command
> and I get an exception as provided below:
>
> I googled a few solutions they suggest to update yarn-site.xml as below
> but I still get the same exception message.
>
> Please let me know what you think.
>
> Thank you
>
>
> hadoop-2.8.5/etc/hadoop/yarn-site.xml
> 
> as below:
>
> 
> 
> mapreduceyarn.nodemanager.aux-services
> mapreduce_shuffle
> 
>   
> yarn.nodemanager.aux-services.mapreduce_shuffle.class
> org.apache.hadoop.mapred.ShuffleHandler
>   
> 
>
> Command and Output
> 
> hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh
> 19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated,
> please use generic option -files instead.
> packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py,
> /tmp/kmeans/line_cust/lineCustReduceGroup.py,
> /tmp/hadoop-unjar6079850917434748483/] []
> /tmp/streamjob3876527625248696490.jar tmpDir=null
> 19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to
> process : 1
> 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2
> 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1560691739419_0001
> 19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application
> application_1560691739419_0001
> 19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job:
> http://localhost:8088/proxy/application_1560691739419_0001/
> 19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001
> 19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running
> in uber mode : false
> 19/06/16 14:32:08 INFO mapreduce.Job:  map 0% reduce 0%
> 19/06/16 14:32:09 INFO mapreduce.Job: Task Id :
> attempt_1560691739419_0001_m_00_0, Status : FAILED
> Container launch failed for container_1560691739419_0001_01_02 :
> org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
> auxService:mapreduce_shuffle does not exist
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 19/06/16 14:32:09 INFO mapreduce.Job: Task Id :
> attempt_1560691739419_0001_m_01_0, Status : FAILED
> Container launch failed for container_1560691739419_0001_01_03 :
> org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The
> auxService:mapreduce_shuffle does not exist
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>