Re: video stream as input to sequence files
Thanks. Is there some example of this process. Regards, On Sat, Feb 28, 2015 at 7:11 AM, daemeon reiydelle daeme...@gmail.com wrote: My thinking ... in your map step take each frame and tag it with an appropriate unique key. Your reducers (if used) then do the frame analysis, If doing frame sequences, then you need to decide the granularity vs. time each node spends executing. Same sort of process that is done for e.g. satellite images undergoing feature recognition analysis. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Feb 25, 2015 at 11:54 PM, tesm...@gmail.com tesm...@gmail.com wrote: Dear Daemeon, Thanks for your rpely. Here is my flow. I am processing video frames using MapReduce. Presently, I convert the video files to individual framess, make a sequence file out of them and transfer the sequence file to HDFS. This flow is not optimized and I need to optimize it. On Thu, Feb 26, 2015 at 3:00 AM, daemeon reiydelle daeme...@gmail.com wrote: Can you explain your use case? *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198 %28%2B1%29%20415.501.0198London (+44) (0) 20 8144 9872 %28%2B44%29%20%280%29%2020%208144%209872* On Wed, Feb 25, 2015 at 4:01 PM, tesm...@gmail.com tesm...@gmail.com wrote: Hi, How can I make my video data files as input for sequence file or to HDFS directly. Regards, Tariq
Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?
Dear Jonathan, Would you please describe the process of running EMR based Hadoop for $15.00, I tried and my cost were rocketing like $60 for one hour. Regards On 05/03/2015 23:57, Jonathan Aquilina wrote: krish EMR wont cost you much with all the testing and data we ran through the test systems as well as the large amont of data when everythign was read we paid about 15.00 USD. I honestly do not think that the specs there would be enough as java can be pretty ram hungry. --- Regards, Jonathan Aquilina Founder Eagle Eye T On 2015-03-06 00:41, Krish Donald wrote: Hi, I am new to AWS and would like to setup Hadoop cluster using cloudera manager for 6-7 nodes. t2.micro on AWS; Is it enough for setting up Hadoop cluster ? I would like to use free service as of now. Please advise. Thanks Krish
Re: How to resolve--- Unauthorized request to start container. This token is expired.
Dear Jan, I changed the data of the node by sudo date *newdatetimestring* Thanks for your help Regards, On Thu, Feb 26, 2015 at 6:31 PM, Jan van Bemmelen j...@tokyoeye.net wrote: Hi Tariq, You seem to be using debian or ubuntu. The documentation here will guide you through setting up ntp: http://www.cyberciti.biz/faq/debian-ubuntu-linux-install-ntpd/ . When you have finished these steps you can check the system’s clocks using the ‘date’ command’. The differences between the servers should be minimal. Regards, Jan On 26 Feb 2015, at 19:19, tesm...@gmail.com wrote: Thanks Jan. I did the follwoing: 1) Manually set the timezone of all the nodes using sudo dpkg-reconfigure tzdata 2) Re-booted the nodes Still having the same exception. How can I configure NTP? Regards, Tariq On Thu, Feb 26, 2015 at 5:33 PM, Jan van Bemmelen j...@tokyoeye.net wrote: Could you check for any time differences between your servers? If so, please install and run NTP, and retry your job. Regards, Jan On 26 Feb 2015, at 17:57, tesm...@gmail.com wrote: I am getting Unauthorized request to start container. This token is expired. How to resovle it. The problem is reported on different forums, but I could not find an solution to it. Below is the execution log 15/02/26 16:41:02 INFO impl.YarnClientImpl: Submitted application application_1424968835929_0001 15/02/26 16:41:02 INFO mapreduce.Job: The url to track the job: http://101-master15:8088/proxy/application_1424968835929_0001/ 15/02/26 16:41:02 INFO mapreduce.Job: Running job: job_1424968835929_0001 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 running in uber mode : false 15/02/26 16:41:04 INFO mapreduce.Job: map 0% reduce 0% 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 failed with state FAILED due to: Application application_1424968835929_0001 failed 2 times due to Error launching appattempt_1424968835929_0001_02. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1424969604829 found 1424969463686 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) . Failing the application. 15/02/26 16:41:04 INFO mapreduce.Job: Counters: 0 Time taken: 0 days, 0 hours, 0 minutes, 9 seconds.
Re: How to resolve--- Unauthorized request to start container. This token is expired.
Thanks Jan. I did the follwoing: 1) Manually set the timezone of all the nodes using sudo dpkg-reconfigure tzdata 2) Re-booted the nodes Still having the same exception. How can I configure NTP? Regards, Tariq On Thu, Feb 26, 2015 at 5:33 PM, Jan van Bemmelen j...@tokyoeye.net wrote: Could you check for any time differences between your servers? If so, please install and run NTP, and retry your job. Regards, Jan On 26 Feb 2015, at 17:57, tesm...@gmail.com wrote: I am getting Unauthorized request to start container. This token is expired. How to resovle it. The problem is reported on different forums, but I could not find an solution to it. Below is the execution log 15/02/26 16:41:02 INFO impl.YarnClientImpl: Submitted application application_1424968835929_0001 15/02/26 16:41:02 INFO mapreduce.Job: The url to track the job: http://101-master15:8088/proxy/application_1424968835929_0001/ 15/02/26 16:41:02 INFO mapreduce.Job: Running job: job_1424968835929_0001 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 running in uber mode : false 15/02/26 16:41:04 INFO mapreduce.Job: map 0% reduce 0% 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 failed with state FAILED due to: Application application_1424968835929_0001 failed 2 times due to Error launching appattempt_1424968835929_0001_02. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1424969604829 found 1424969463686 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) . Failing the application. 15/02/26 16:41:04 INFO mapreduce.Job: Counters: 0 Time taken: 0 days, 0 hours, 0 minutes, 9 seconds.
Re: How to resolve--- Unauthorized request to start container. This token is expired.
Thanks Jan, I followed the link and re-booted the node. Still no success. Time on this node is about 13 minutes behind the other nodes. Any otehr suggestion please This node is workig as my namenode On Thu, Feb 26, 2015 at 6:31 PM, Jan van Bemmelen j...@tokyoeye.net wrote: Hi Tariq, You seem to be using debian or ubuntu. The documentation here will guide you through setting up ntp: http://www.cyberciti.biz/faq/debian-ubuntu-linux-install-ntpd/ . When you have finished these steps you can check the system’s clocks using the ‘date’ command’. The differences between the servers should be minimal. Regards, Jan On 26 Feb 2015, at 19:19, tesm...@gmail.com wrote: Thanks Jan. I did the follwoing: 1) Manually set the timezone of all the nodes using sudo dpkg-reconfigure tzdata 2) Re-booted the nodes Still having the same exception. How can I configure NTP? Regards, Tariq On Thu, Feb 26, 2015 at 5:33 PM, Jan van Bemmelen j...@tokyoeye.net wrote: Could you check for any time differences between your servers? If so, please install and run NTP, and retry your job. Regards, Jan On 26 Feb 2015, at 17:57, tesm...@gmail.com wrote: I am getting Unauthorized request to start container. This token is expired. How to resovle it. The problem is reported on different forums, but I could not find an solution to it. Below is the execution log 15/02/26 16:41:02 INFO impl.YarnClientImpl: Submitted application application_1424968835929_0001 15/02/26 16:41:02 INFO mapreduce.Job: The url to track the job: http://101-master15:8088/proxy/application_1424968835929_0001/ 15/02/26 16:41:02 INFO mapreduce.Job: Running job: job_1424968835929_0001 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 running in uber mode : false 15/02/26 16:41:04 INFO mapreduce.Job: map 0% reduce 0% 15/02/26 16:41:04 INFO mapreduce.Job: Job job_1424968835929_0001 failed with state FAILED due to: Application application_1424968835929_0001 failed 2 times due to Error launching appattempt_1424968835929_0001_02. Got exception: org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1424969604829 found 1424969463686 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) . Failing the application. 15/02/26 16:41:04 INFO mapreduce.Job: Counters: 0 Time taken: 0 days, 0 hours, 0 minutes, 9 seconds.
Re: java.net.UnknownHostException on one node only
Thanks Varun, Where shall I check to resolve it? Regards, Tariq On Mon, Feb 23, 2015 at 4:07 AM, Varun Kumar varun@gmail.com wrote: Hi Tariq, Issues looks like DNS configuration issue. On Sun, Feb 22, 2015 at 3:51 PM, tesm...@gmail.com tesm...@gmail.com wrote: I am getting java.net.UnknownHost exception continuously on one node Hadoop MApReduce execution. That node is accessible via SSH. This node is shown in yarn node -list and hadfs dfsadmin -report queries. Below is the log from execution 15/02/22 20:17:42 INFO mapreduce.Job: Task Id : attempt_1424622614381_0008_m_43_0, Status : FAILED Container launch failed for container_1424622614381_0008_01_16 : java.lang.IllegalArgumentException: *java.net.UnknownHostException: 101-master10* at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:352) at org.apache.hadoop.yarn.util.ConverterUtils.convertFromYarn(ConverterUtils.java:237) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:218) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.init(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) *Caused by: java.net.UnknownHostException: 101-master10* ... 12 more 15/02/22 20:17:44 INFO Regards, Tariq -- Regards, Varun Kumar.P
HDFS data after nodes become unavailable?
Dear all, I have transferred the data from local storage to HDFS in my 10 nodes Hadoop cluster. The relication facotr is 3. Some nodes, say 3, are not available after some time. I can't use those nodes for computation or storage of data. What will happen to the data stored on HDFS of those nodes? Do I need to remvoe all the data from HDFS and copy it again? Regards,
video stream as input to sequence files
Hi, How can I make my video data files as input for sequence file or to HDFS directly. Regards, Tariq
Re: video stream as input to sequence files
Dear Daemeon, Thanks for your rpely. Here is my flow. I am processing video frames using MapReduce. Presently, I convert the video files to individual framess, make a sequence file out of them and transfer the sequence file to HDFS. This flow is not optimized and I need to optimize it. On Thu, Feb 26, 2015 at 3:00 AM, daemeon reiydelle daeme...@gmail.com wrote: Can you explain your use case? *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Feb 25, 2015 at 4:01 PM, tesm...@gmail.com tesm...@gmail.com wrote: Hi, How can I make my video data files as input for sequence file or to HDFS directly. Regards, Tariq
java.net.UnknownHostException on one node only
I am getting java.net.UnknownHost exception continuously on one node Hadoop MApReduce execution. That node is accessible via SSH. This node is shown in yarn node -list and hadfs dfsadmin -report queries. Below is the log from execution 15/02/22 20:17:42 INFO mapreduce.Job: Task Id : attempt_1424622614381_0008_m_43_0, Status : FAILED Container launch failed for container_1424622614381_0008_01_16 : java.lang.IllegalArgumentException: *java.net.UnknownHostException: 101-master10* at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:352) at org.apache.hadoop.yarn.util.ConverterUtils.convertFromYarn(ConverterUtils.java:237) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:218) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.init(ContainerManagementProtocolProxy.java:196) at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) *Caused by: java.net.UnknownHostException: 101-master10* ... 12 more 15/02/22 20:17:44 INFO Regards, Tariq
Running MapReduce jobs in batch mode on different data sets
Hi, Is it possible to run jobs on Hadoop in batch mode? I have 5 different datasets in HDFS and need to run the same MapReduce application on these datasets sets one after the other. Right now I am doing it manually How can I automate this? How can I save the log of each execution in text files for later processing? Regards, Tariq
Scheduling in YARN according to available resources
I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. I followed the link from Hortonwroks [ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html ] and made calculation according to the hardware configruation on my nodes. Added the update mapred-site and yarn-site.xml files in my question. Still my application is crashing with the same exection My mapreduce application has 34 input splits with a block size of 128MB. **mapred-site.xml** has the following properties: mapreduce.framework.name = yarn mapred.child.java.opts= -Xmx2048m mapreduce.map.memory.mb = 4096 mapreduce.map.java.opts = -Xmx2048m **yarn-site.xml** has the following properties: yarn.resourcemanager.hostname= hadoop-master yarn.nodemanager.aux-services= mapreduce_shuffle yarn.nodemanager.resource.memory-mb = 6144 yarn.scheduler.minimum-allocation-mb = 2048 yarn.scheduler.maximum-allocation-mb = 6144 Exception from container-launch: ExitCodeException exitCode=134: /bin/bash: line 1: 3876 Aborted (core dumped) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx8192m -Djava.io.tmpdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1424264025191_0002/container_1424264025191_0002_01_11/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.0.12 50842 attempt_1424264025191_0002_m_05_0 11 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stdout 2 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stderr How can avoid this?any help is appreciated It looks to me that YAN is trying to launch all the container simultaneously and anot according to the available resources. Is there an option to restrict number of containers on hadoop ndoes? Regards, Tariq
YARN container lauch failed exception and mapred-site.xml configuration
I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. **EDIT-1@ARNON:** I followed the link, mad calculation according to the hardware configruation on my nodes and have added the update mapred-site and yarn-site.xml files in my question. Still my application is crashing with the same exection My mapreduce application has 34 input splits with a block size of 128MB. **mapred-site.xml** has the following properties: mapreduce.framework.name = yarn mapred.child.java.opts= -Xmx2048m mapreduce.map.memory.mb = 4096 mapreduce.map.java.opts = -Xmx2048m **yarn-site.xml** has the following properties: yarn.resourcemanager.hostname= hadoop-master yarn.nodemanager.aux-services= mapreduce_shuffle yarn.nodemanager.resource.memory-mb = 6144 yarn.scheduler.minimum-allocation-mb = 2048 yarn.scheduler.maximum-allocation-mb = 6144 Exception from container-launch: ExitCodeException exitCode=134: /bin/bash: line 1: 3876 Aborted (core dumped) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx8192m -Djava.io.tmpdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1424264025191_0002/container_1424264025191_0002_01_11/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.0.12 50842 attempt_1424264025191_0002_m_05_0 11 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stdout 2 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stderr How can avoid this?any help is appreciated Is there an option to restrict number of containers on hadoop ndoes?
Re: Scheduling in YARN according to available resources
Thanks for your answer Nair, Is installing Oracle JDK on Ubuntu is that complicated as described in this link http://askubuntu.com/questions/56104/how-can-i-install-sun-oracles-proprietary-java-jdk-6-7-8-or-jre Is there an alternate? Regards On Sat, Feb 21, 2015 at 6:50 AM, R Nair ravishankar.n...@gmail.com wrote: I had an issue very similar, I changed and used Oracle JDK. There is nothing I see wrong with your configuration in my first look, thanks Regards, Nair On Sat, Feb 21, 2015 at 1:42 AM, tesm...@gmail.com tesm...@gmail.com wrote: I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. I followed the link from Hortonwroks [ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html ] and made calculation according to the hardware configruation on my nodes. Added the update mapred-site and yarn-site.xml files in my question. Still my application is crashing with the same exection My mapreduce application has 34 input splits with a block size of 128MB. **mapred-site.xml** has the following properties: mapreduce.framework.name = yarn mapred.child.java.opts= -Xmx2048m mapreduce.map.memory.mb = 4096 mapreduce.map.java.opts = -Xmx2048m **yarn-site.xml** has the following properties: yarn.resourcemanager.hostname= hadoop-master yarn.nodemanager.aux-services= mapreduce_shuffle yarn.nodemanager.resource.memory-mb = 6144 yarn.scheduler.minimum-allocation-mb = 2048 yarn.scheduler.maximum-allocation-mb = 6144 Exception from container-launch: ExitCodeException exitCode=134: /bin/bash: line 1: 3876 Aborted (core dumped) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx8192m -Djava.io.tmpdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1424264025191_0002/container_1424264025191_0002_01_11/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.0.12 50842 attempt_1424264025191_0002_m_05_0 11 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stdout 2 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stderr How can avoid this?any help is appreciated It looks to me that YAN is trying to launch all the container simultaneously and anot according to the available resources. Is there an option to restrict number of containers on hadoop ndoes? Regards, Tariq -- Warmest Regards, Ravi Shankar
Fwd: YARN container lauch failed exception and mapred-site.xml configuration
I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. I followed the link o horton works [ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html] and made calculation according to the hardware configruation on my nodes and have added the update mapred-site and yarn-site.xml files in my question. Still my application is crashing with the same exection My mapreduce application has 34 input splits with a block size of 128MB. **mapred-site.xml** has the following properties: mapreduce.framework.name = yarn mapred.child.java.opts= -Xmx2048m mapreduce.map.memory.mb = 4096 mapreduce.map.java.opts = -Xmx2048m **yarn-site.xml** has the following properties: yarn.resourcemanager.hostname= hadoop-master yarn.nodemanager.aux-services= mapreduce_shuffle yarn.nodemanager.resource.memory-mb = 6144 yarn.scheduler.minimum-allocation-mb = 2048 yarn.scheduler.maximum-allocation-mb = 6144 Exception from container-launch: ExitCodeException exitCode=134: /bin/bash: line 1: 3876 Aborted (core dumped) /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx8192m -Djava.io.tmpdir=/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1424264025191_0002/container_1424264025191_0002_01_11/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.0.12 50842 attempt_1424264025191_0002_m_05_0 11 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stdout 2 /home/ubuntu/hadoop/logs/userlogs/application_1424264025191_0002/container_1424264025191_0002_01_11/stderr How can avoid this?any help is appreciated Is there an option to restrict number of containers on hadoop ndoes?
