Hadoop 2.2 HttpCleint Jar Mismatch
Hello experts, I am consistently running into this excepition , googling tells me that this is because of the Jar mismatch between Hadoop and the Nutch Job , with Hadoop2.2 using older version of the HttpClient. I am not able to figure out how I could make Hadoop 2.2 pickup the jars from the sumitted job (which is using the old map reduce API) , can some one who has gone thru this before let me know ? Thanks. 2013-12-10 23:00:09,396 INFO [FetcherThread] org.apache.nutch.fetcher.Fetcher: fetch of http://www.xyz.com/ failed with: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105) at org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:154) at org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:127)
Hadoop 2.2 HttpCleint Jar Mismatch
Resending my message not sure if it went thru the first time... any help would be greatly appreciated. I am consistently running into this excepition , googling tells me that this is because of the Jar mismatch between Hadoop and the Nutch Job , with Hadoop2.2 using older version of the HttpClient. I am not able to figure out how I could make Hadoop 2.2 pickup the jars from the sumitted job (which is using the old map reduce API) , can some one who has gone thru this before let me know ? Thanks. 2013-12-10 23:00:09,396 INFO [FetcherThread] org.apache.nutch.fetcher.Fetcher: fetch of http://www.xyz.com/ failed with: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105) at org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:154) at org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:127)
How to set the environment variables in Hadoop 2.2.0
The hadoop document below suggests that the following variables be set inorder for Hadoop to prioritize the client jars over the Hadoop jars , however , I am not sure how to set them can someone please tell me how to set these . *HADOOP_USER_CLASSPATH_FIRST=true* and *HADOOP_CLASSPATH* =...:hadoop-examples-1.x.x.jar to run their target examples jar, and add the following configuration in mapred-site.xml to make the processes in YARN containers pick this jar as well. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html Thanks.
Hadoop 2.2 Psuedo- Distributed Mode
Hi folks, I am unning Haddop 2.2 in pseudo-distributed mode and I have a i5 processor with 8 GB RAM running on CentOS 6.4 . However my Nutch job fails after a few minutes into execution withan OOM exception , I have increased the HADOOP_HEAPSIZE from the 1000MB to 4GB, but I still face the issue. I do not get this issues if the job is run in a local mode. I want to know how many slave nodes does Hadoop 2.2 create in a psuedo-distributed mode and how can I control it ? Thanks.
Unable to change the virtual memory to be more than the default 2.1 GB
Hello Folks, I am running hadoop 2.2 in a pseudo-distributed mode on a laptop with 8GB RAM. Whenever I submit a job I get an error that says that the that the virtual memory usage exceeded , like below. I have changed the ratio yarn.nodenamager.vmem-pmem-ratio in yarn-site.xml to 10 , however the virtual memory is not getting increased more than 2.1 GB , as can been seen in the error message below and the container is being killed. Can some one please let me know if there is any other setting that needs to be changed ? Thanks in advance! *Error Message :* INFO mapreduce.Job: Task Id : attempt_1388632710048_0009_m_00_2, Status : FAILED Container [pid=12013,containerID=container_1388632710048_0009_01_04] is running beyond virtual memory limits. Current usage: 544.9 MB of 1 GB physical memory used; 14.5 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1388632710048_0009_01_04 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 12077 12018 12013 12013 (phantomjs) 16 2 1641000960 6728 /usr/local/bin/phantomjs --webdriver=15358 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12013 882 12013 12013 (bash) 1 0 108650496 305 /bin/bash -c /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 attempt_1388632710048_0009_m_00_2 4 1>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stdout 2>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stderr |- 12075 12018 12013 12013 (phantomjs) 17 1 1615687680 6539 /usr/local/bin/phantomjs --webdriver=29062 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12074 12018 12013 12013 (phantomjs) 16 2 1641000960 6727 /usr/local/bin/phantomjs --webdriver=5958 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12073 12018 12013 12013 (phantomjs) 17 2 1641000960 6732 /usr/local/bin/phantomjs --webdriver=31836 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12090 12018 12013 12013 (phantomjs) 16 2 1615687680 6538 /usr/local/bin/phantomjs --webdriver=24519 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12072 12018 12013 12013 (phantomjs) 16 1 1641000960 6216 /usr/local/bin/phantomjs --webdriver=10175 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12091 12018 12013 12013 (phantomjs) 17 1 1615687680 6036 /usr/local/bin/phantomjs --webdriver=5043 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12018 12013 12013 12013 (java) 996 41 820924416 79595 /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 attempt_1388632710048_0009_m_00_2 4 |- 12078 12018 12013 12013 (phantomjs) 16 3 1615687680 6545 /usr/local/bin/phantomjs --webdriver=12650 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log |- 12079 12018 12013 12013 (phantomjs) 17 2 1642020864 7542 /usr/local/bin/phantomjs --webdriver=18444 --webdriver-log
Re: Unable to change the virtual memory to be more than the default 2.1 GB
Hi German, Thanks for your reply! a) Yes setting the property yarn.nodemanager.vmem-check-enabled to false seems to have avoid the problem. b) I woud want to set the pmem/vmem ratio to a higher value and keep the virtual memory with in certain limits but , changing this value is not having any effect on the Hadoop2.2 YARN . c) Why would virtual memory increase and the physical memory stay the same , what might be the causes that would make this happen in YARN ? Thanks. On Thu, Jan 2, 2014 at 11:18 AM, German Florez-Larrahondo < german...@samsung.com> wrote: > A few things you can try > > > > a) If you don’t care about virtual memory controls at all you can > bypass it by doing the following change in the XML and restarting YARN. > Only you know if this is OK for the application you are trying (IMO the > virtual memory being used is huge!) > > > > yarn.nodemanager.vmem-check-enabled > > false > > > > b) If you still want to control the pmem/vmem, do you restart YARN > after doing the chage in the XML file? > > > > > > Regards./g > > > > *From:* S.L [mailto:simpleliving...@gmail.com] > *Sent:* Wednesday, January 01, 2014 9:51 PM > *To:* user@hadoop.apache.org > *Subject:* Unable to change the virtual memory to be more than the > default 2.1 GB > > > > Hello Folks, > > I am running hadoop 2.2 in a pseudo-distributed mode on a laptop with 8GB > RAM. > > Whenever I submit a job I get an error that says that the that the virtual > memory usage exceeded , like below. > > I have changed the ratio > > in yarn-site.xml to 10 , however the virtual memory is not getting > increased more than 2.1 GB , as can been seen in the error message below > and the container is being killed. > > Can some one please let me know if there is any other setting that needs > to be changed ? Thanks in advance! > > *Error Message :* > > INFO mapreduce.Job: Task Id : attempt_1388632710048_0009_m_00_2, > Status : FAILED > Container [pid=12013,containerID=container_1388632710048_0009_01_04] > is running beyond virtual memory limits. Current usage: 544.9 MB of 1 GB > physical memory used; 14.5 GB of 2.1 GB virtual memory used. Killing > container. > Dump of the process-tree for container_1388632710048_0009_01_04 : > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 12077 12018 12013 12013 (phantomjs) 16 2 1641000960 6728 > /usr/local/bin/phantomjs --webdriver=15358 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12013 882 12013 12013 (bash) 1 0 108650496 305 /bin/bash -c > /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx200m > -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/tmp > -Dlog4j.configuration=container-log4j.properties > -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04 > -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA > org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 > attempt_1388632710048_0009_m_00_2 4 > 1>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stdout > 2>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stderr > > |- 12075 12018 12013 12013 (phantomjs) 17 1 1615687680 6539 > /usr/local/bin/phantomjs --webdriver=29062 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12074 12018 12013 12013 (phantomjs) 16 2 1641000960 6727 > /usr/local/bin/phantomjs --webdriver=5958 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12073 12018 12013 12013 (phantomjs) 17 2 1641000960 6732 > /usr/local/bin/phantomjs --webdriver=31836 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12090 12018 12013 12013 (phantomjs) 16 2 1615687680 6538 > /usr/local/bin/phantomjs --webdriver=24519 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_138863
Re: Unable to change the virtual memory to be more than the default 2.1 GB
Vinod, Thanks for your reply. 1. If I understand you correct you are asking me to change the memory allocation for each map and reduce tasks , isnt this related to the physical memory which is not an issue(with in limits) in my application ? The problem I am facing is with the virtual memory. 2. You are right I am spawning shells , but immediately closing them after each request in the map task. Why would virtual memory increase and the physical memory stay the same , what might be the causes that would make this happen in YARN ? How can I keep it with in manageable limits ? If I run the same task as a stand alone program it works fine , obviously because its not a memory leak kind of a scenario affecting the physical memory. On Thu, Jan 2, 2014 at 1:14 PM, Vinod Kumar Vavilapalli < vino...@hortonworks.com> wrote: > You need to change the application configuration itself to tell YARN that > each task needs more than the default. I see that this is a mapreduce app, > so you have to change the per-application configuration: > mapreduce.map.memory.mb and mapreduce.reduce.memory.mb in either > mapred-site.xml or via the command line. > > Side notes: Seems like you are spawning lots of shells under your mapper > and YARN's NodeManager is detecting that the total virtual memory usage is > 14.5GB. You may want to reduce that number of shells, lest the OS itself > might kill your tasks depend on the system configuration. > > Thanks, > +Vinod > > On Jan 1, 2014, at 7:50 PM, S.L wrote: > > Hello Folks, > > I am running hadoop 2.2 in a pseudo-distributed mode on a laptop with 8GB > RAM. > > Whenever I submit a job I get an error that says that the that the virtual > memory usage exceeded , like below. > > I have changed the ratio yarn.nodenamager.vmem-pmem-ratio in yarn-site.xml > to 10 , however the virtual memory is not getting increased more than 2.1 > GB , as can been seen in the error message below and the container is being > killed. > > Can some one please let me know if there is any other setting that needs > to be changed ? Thanks in advance! > > *Error Message :* > > INFO mapreduce.Job: Task Id : attempt_1388632710048_0009_m_00_2, > Status : FAILED > Container [pid=12013,containerID=container_1388632710048_0009_01_04] > is running beyond virtual memory limits. Current usage: 544.9 MB of 1 GB > physical memory used; 14.5 GB of 2.1 GB virtual memory used. Killing > container. > Dump of the process-tree for container_1388632710048_0009_01_04 : > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 12077 12018 12013 12013 (phantomjs) 16 2 1641000960 6728 > /usr/local/bin/phantomjs --webdriver=15358 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12013 882 12013 12013 (bash) 1 0 108650496 305 /bin/bash -c > /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx200m > -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/tmp > -Dlog4j.configuration=container-log4j.properties > -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04 > -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA > org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 > attempt_1388632710048_0009_m_00_2 4 > 1>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stdout > 2>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_04/stderr > > |- 12075 12018 12013 12013 (phantomjs) 17 1 1615687680 6539 > /usr/local/bin/phantomjs --webdriver=29062 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12074 12018 12013 12013 (phantomjs) 16 2 1641000960 6727 > /usr/local/bin/phantomjs --webdriver=5958 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12073 12018 12013 12013 (phantomjs) 17 2 1641000960 6732 > /usr/local/bin/phantomjs --webdriver=31836 > --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_04/phantomjsdriver.log > > |- 12090 12018 12013 12013 (phantomjs) 16 2
Hadoop "Spill Failed" Exception in an ec2 instance with 420 GB of instance storage
Hi All, I am using Hadoop2.3.0 and have installed it as single node cluster (psuedo-distributed mode) on CentOS 6.4 Amazon ec2 instance with an instance storage of 420GB and 7.5GB of RAM , my understanding is that the " Spill Failed " exception only occurs when the node runs out of the disk space however , after running map/reduce tasks for only a short amount of time (no where near to 420 GB of data ) I get the following exception. I would like to mention that I moved the Hadoop installation on the same node from an EBS volume of 8GB (where I had installed it originally) to an instance store volume of 420GB on the same node and changed the $HADOOP_HOME environment variable and other properties to point to the instance store volume accordingly and the Hadoop2.3.0 is now completely contained in the 420 GB drive. However I still see the following exception , can you please let me know if there is anything besides Diskspace that can cause the Spill Failed exception ? 2014-02-28 15:35:07,630 ERROR [IPC Server handler 12 on 58189] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1393591821307_0013_m_00_0 - exited : java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1533) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1442) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1393591821307_0013_m_00_0_spill_26.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1564) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:853) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1503) 2014-02-28 15:35:07,604 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: Spill failed 2014-02-28 15:35:07,605 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1533) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1442) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1393591821307_0013_m_00_0_spill_26.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1564) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:853) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1503)
Hadoop Spill Exception
Hi All, I am running Nutch on Hadoop2.3.0 and I get Hadoop Spill Exception even though the disk space that is being utilized is only 30% of the available. I am looking at the syslog file on the userlogs directory however , I do not see much information there except the following line. How do I know whats causing this spill exception as disk space does not seem to be the problem. I see the following entry in the syslog file. 2014-04-17 12:17:28,517 INFO [FetcherThread] org.apache.hadoop.mapred.MapTask: kvstart = 6159120(24636480); kvend = 6152164(24608656); length = 6957/6553600 2014-04-17 12:17:28,528 INFO [FetcherThread] org.apache.hadoop.mapred.MapTask: (EQUATOR) 3661729 kvi 915428(3661712) 2014-04-17 12:17:28,529 ERROR [FetcherThread] org.apache.nutch.fetcher.Fetcher: fetcher caught:java.io.IOException: Spill failed Thanks for your help.
hadoop.tmp.dir directory size
Hi Folks, I am running a map only (no reduce) job on Hadoop 2.3.0 and the /tmp/hadoop-df/nm-local-dir (df being my user name) directory is growing exponentially causing me run out of my 80GB disk space. I would like to know if there is any mechanism(automatic or manual) by which I can free up the space in /tmp/hadoop-df/nm-local-dir while the job is still running. Thanks.
Re: hadoop.tmp.dir directory size
Can I do this while the job is still running ? I know I cant delete the directory , but I just want confirmation that the data Hadoop writes into /tmp/hadoop-df/nm-local-dir (df being my user name) can be discarded while the job is being executed. On Wed, Apr 30, 2014 at 6:40 AM, unmesha sreeveni wrote: > > Try > > *hadoop fs -rmr /tmp* > > -- > *Thanks & Regards * > > > *Unmesha Sreeveni U.B* > *Hadoop, Bigdata Developer* > *Center for Cyber Security | Amrita Vishwa Vidyapeetham* > http://www.unmeshasreeveni.blogspot.in/ > > >
Random Exception
Hi All, I get this exception after af resubmit my failed MapReduce jon, can one please let me know what this exception means ? 14/05/02 01:28:25 INFO mapreduce.Job: Task Id : attempt_1398989569957_0021_m_00_0, Status : FAILED Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1398989569957_0021_m_00_0/intermediate.26 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:711) at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:579) at org.apache.hadoop.mapred.Merger.merge(Merger.java:150) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1870) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
Re: Random Exception
On Fri, May 2, 2014 at 12:20 PM, S.L wrote: > I am using Hadoop 2.3 , the problem is that my disk runs out of space > (80GB) and then I reboot my machine , which causes my /tmp data to be > deleted and frees up space , I then resubmit the job assuming that since > the namenode and datanode data are not stored in /tmp everything should be > ok.( I have set the namenode and datanode to be stored at a dieffrent > location than /tmp). > > By the way this is a single node cluster(psuedo distributed) setup. > > > On Fri, May 2, 2014 at 9:02 AM, Marcos Ortiz wrote: > >> It seems that your Hadoop data directory is broken or your disk has >> problems. >> Which version of Hadoop are you using? >> >> On Friday, May 02, 2014 08:43:44 AM S.L wrote: >> > Hi All, >> > >> > I get this exception after af resubmit my failed MapReduce jon, can one >> > please let me know what this exception means ? >> > >> > 14/05/02 01:28:25 INFO mapreduce.Job: Task Id : >> > attempt_1398989569957_0021_m_00_0, Status : FAILED >> > Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not >> > find any valid local directory for >> > attempt_1398989569957_0021_m_00_0/intermediate.26 >> > at >> > >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr >> > ite(LocalDirAllocator.java:402) at >> > >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocato >> > r.java:150) at >> > >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocato >> > r.java:131) at >> > org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:711) at >> > org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:579) at >> > org.apache.hadoop.mapred.Merger.merge(Merger.java:150) >> > at >> > >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:187 >> > 0) at >> > >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) >> > at >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Unknown Source) >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja >> > va:1548) >> >> >> I Conferencia Científica Internacional UCIENCIA 2014 en la UCI del 24 al >> 26 de abril de 2014, La Habana, Cuba. Ver http://uciencia.uci.cu >> > >
Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.
Hi All, I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master and three slave nodes , the slave nodes are listed in the $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the Master Name node on port 9000, however when I start the datanode on any of the slaves I get the following exception . 2014-08-03 08:04:27,952 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode Uuid null) service to server1.dealyaft.com/170.75.152.162:9000 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode because hostname cannot be resolved . The following are the contents of my core-site.xml. fs.default.name hdfs://server1.mydomain.com:9000 Also in my hdfs-site.xml I am not setting any value for dfs.hosts or dfs.hosts.exclude properties. Thanks.
Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.
The contents are 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh < riteshoneinamill...@gmail.com> wrote: > check the contents of '/etc/hosts' file > > > On Mon, Aug 4, 2014 at 3:27 AM, S.L wrote: > >> Hi All, >> >> I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master and >> three slave nodes , the slave nodes are listed in the >> $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the >> Master Name node on port 9000, however when I start the datanode on any of >> the slaves I get the following exception . >> >> 2014-08-03 08:04:27,952 FATAL >> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for >> block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode >> Uuid null) service to server1.dealyaft.com/170.75.152.162:9000 >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): >> Datanode denied communication with namenode because hostname cannot be >> resolved . >> >> The following are the contents of my core-site.xml. >> >> >> >> fs.default.name >> hdfs://server1.mydomain.com:9000 >> >> >> >> Also in my hdfs-site.xml I am not setting any value for dfs.hosts or >> dfs.hosts.exclude properties. >> >> Thanks. >> > >
Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.
when you say /etc/hosts/ file , you mean only on the master of on both the master and slaves? On Tue, Aug 5, 2014 at 1:20 AM, Satyam Singh wrote: > You have not given namenode uri in /etc/hosts file , thus it can't > resolve it to ipaddress and your namenode would also be not started. > Preferable practice is to start your cluster through start-dfs.sh command, > it implicitly starts first namenode and then all its datanodes. > > Also make sure you have given ipaddress in salve file, if not then also > make entry for hostnames in /etc/hosts file > > > BR, > Satyam > > On 08/05/2014 12:21 AM, S.L wrote: > > > The contents are > > 127.0.0.1 localhost localhost.localdomain localhost4 > localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 > localhost6.localdomain6 > > > > On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh < > riteshoneinamill...@gmail.com> wrote: > >> check the contents of '/etc/hosts' file >> >> >> On Mon, Aug 4, 2014 at 3:27 AM, S.L wrote: >> >>> Hi All, >>> >>> I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master >>> and three slave nodes , the slave nodes are listed in the >>> $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the >>> Master Name node on port 9000, however when I start the datanode on any of >>> the slaves I get the following exception . >>> >>> 2014-08-03 08:04:27,952 FATAL >>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for >>> block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode >>> Uuid null) service to server1.dealyaft.com/170.75.152.162:9000 >>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): >>> Datanode denied communication with namenode because hostname cannot be >>> resolved . >>> >>> The following are the contents of my core-site.xml. >>> >>> >>> >>> fs.default.name >>> hdfs://server1.mydomain.com:9000 >>> >>> >>> >>> Also in my hdfs-site.xml I am not setting any value for dfs.hosts or >>> dfs.hosts.exclude properties. >>> >>> Thanks. >>> >> >> > >
Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.
There is no entry there. Sent from my HTC - Reply message - From: "hadoop hive" To: Subject: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster. Date: Tue, Aug 5, 2014 6:36 AM Remove the entry from dfs.exclude if there is any On Aug 4, 2014 3:28 AM, "S.L" wrote: > Hi All, > > I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master and > three slave nodes , the slave nodes are listed in the > $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the > Master Name node on port 9000, however when I start the datanode on any of > the slaves I get the following exception . > > 2014-08-03 08:04:27,952 FATAL > org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for > block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode > Uuid null) service to server1.dealyaft.com/170.75.152.162:9000 > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): > Datanode denied communication with namenode because hostname cannot be > resolved . > > The following are the contents of my core-site.xml. > > > > fs.default.name > hdfs://server1.mydomain.com:9000 > > > > Also in my hdfs-site.xml I am not setting any value for dfs.hosts or > dfs.hosts.exclude properties. > > Thanks. >
Passing VM arguments while submitting a job in Hadoop YARN.
Hi All, All we submit a job using bin/hadoop script on teh resource manager node, what if we need our job to be passed in a vm argument like in my case 'target-env' , how o I do that , will this argument be passed to all the node managers in different nodes ?
Nutch 1.7 failing on Hadoop YARN after running for a while.
Hi All, I have a failure in one of the applications consistently after my Nutch job runs for like an hour, can some please suggest why this error is occurring looking at the exception message below. Diagnostics: Application application_1408512952691_0017 failed 2 times due to AM Container for appattempt_1408512952691_0017_02 exited with exitCode: -1000 due to: File does not exist: hdfs:// server1.mydomain.com:9000/tmp/hadoop-yarn/staging/df/.staging/job_1408512952691_0017/job.jar .Failing this attempt.. Failing the application.
Hadoop YARM Cluster Setup Questions
Hi Folks, I was not able to find a clear answer to this , I know that on the master node we need to have a slaves file listing all the slaves , but do we need to have the slave nodes have a master file listing the single name node( I am not using a secondary name node). I only have the slaves file on the master node. I was not able to find a clear answer to this ,the reason I ask this is because when I submit a hadoop job , even though the input is being split into 2 parts , only one data node is assigned applications , the other two ( I have three) are no tbeing assigned any applications. Thanks in advance!
Re: Hadoop YARM Cluster Setup Questions
Ok, Ill copy the slaves file to the other slave nodes as well. What about the masters file though? Sent from my HTC - Reply message - From: "rab ra" To: "user@hadoop.apache.org" Subject: Hadoop YARM Cluster Setup Questions Date: Sat, Aug 23, 2014 5:03 AM Hi, 1. Typically,we used to copy the slaves file all the participating nodes though I do not have concrete theory to back up this. Atleast, this is what I was doing in hadoop 1.2 and I am doing the same in hadoop 2x 2. I think, you should investigate the yarn GUI and see how many maps it has spanned. There is a high possibility that both the maps are running in the same node in parallel. Since there are two splits, there would be two map processes, and one node is capable of handling more than one map. 3. There could be no replica of input file stored and it is small, and hence stored in a single block in one node itself. These could be few hints which might help you regards rab On Sat, Aug 23, 2014 at 12:26 PM, S.L wrote: > Hi Folks, > > I was not able to find a clear answer to this , I know that on the master > node we need to have a slaves file listing all the slaves , but do we need > to have the slave nodes have a master file listing the single name node( I > am not using a secondary name node). I only have the slaves file on the > master node. > > I was not able to find a clear answer to this ,the reason I ask this is > because when I submit a hadoop job , even though the input is being split > into 2 parts , only one data node is assigned applications , the other two > ( I have three) are no tbeing assigned any applications. > > Thanks in advance! >
Re: Hadoop YARM Cluster Setup Questions
Thats what I thought too, but please check the Answer #2 here in this question , I am facing a similar problem. http://stackoverflow.com/questions/12135949/why-map-task-always-running-on-a-single-node On Sat, Aug 23, 2014 at 12:44 PM, Chris MacKenzie < stu...@chrismackenziephotography.co.uk> wrote: > Hi, > > The requirement is simply to have the slaves and masters files on the > resource manager it's used by the shell script that starts the demons :-) > > Sent from my iPhone > > On 23 Aug 2014, at 16:02, "S.L" wrote: > > Ok, Ill copy the slaves file to the other slave nodes as well. > > What about the masters file though? > > Sent from my HTC > > - Reply message - > From: "rab ra" > To: "user@hadoop.apache.org" > Subject: Hadoop YARM Cluster Setup Questions > Date: Sat, Aug 23, 2014 5:03 AM > > Hi, > > 1. Typically,we used to copy the slaves file all the participating nodes > though I do not have concrete theory to back up this. Atleast, this is what > I was doing in hadoop 1.2 and I am doing the same in hadoop 2x > > 2. I think, you should investigate the yarn GUI and see how many maps it > has spanned. There is a high possibility that both the maps are running in > the same node in parallel. Since there are two splits, there would be two > map processes, and one node is capable of handling more than one map. > > 3. There could be no replica of input file stored and it is small, and > hence stored in a single block in one node itself. > > These could be few hints which might help you > > regards > rab > > > > On Sat, Aug 23, 2014 at 12:26 PM, S.L wrote: > > > Hi Folks, > > > > I was not able to find a clear answer to this , I know that on the master > > node we need to have a slaves file listing all the slaves , but do we need > > to have the slave nodes have a master file listing the single name node( I > > am not using a secondary name node). I only have the slaves file on the > > master node. > > > > I was not able to find a clear answer to this ,the reason I ask this is > > because when I submit a hadoop job , even though the input is being split > > into 2 parts , only one data node is assigned applications , the other two > > ( I have three) are no tbeing assigned any applications. > > > > Thanks in advance! > > > >
Job is reported as complete on history server while on console it shows as only half way thru
Hi All, I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when I submit this job YARN created multiple applications for that submitted job , and the last application that is running in YARN is marked as complete even as on console its reported as only 58% complete . I have confirmed that its also not printing the log statements that its supposed to print when the job is actually complete . Please see the output from the job submission console below. It just stops at 58% and job history server and YARN cluster UI reports that this job has already succeeded. 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks.
Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
Hi All, I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when I submit this job YARN the application that is running in YARN is marked as complete even as on console its reported as only 58% complete . I have confirmed that its also not printing the log statements that its supposed to print when the job is actually complete . Please see the output from the job submission console below. It just stops at 58% and job history server and YARN cluster UI reports that this job has already succeeded. Can someone let me know why this is happening ? 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks.
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
I am not sure, I running a sequence of MRV1 jobs using a bash script , this seems to happen in the 4th iteration consistently, how do I confirm this possibility? On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli wrote: > Is it possible that the client JVM is somehow getting killed while the > YARN application finishes as usual on the cluster in the background? > > +Vinod > > On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: > >> >>Hi All, >> >> I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is >> when I submit this job YARN the application that is running in YARN is >> marked as complete even as on console its reported as only 58% complete . I >> have confirmed that its also not printing the log statements that its >> supposed to print when the job is actually complete . >> >> Please see the output from the job submission console below. It just >> stops at 58% and job history server and YARN cluster UI reports that this >> job has already succeeded. Can someone let me know why this is happening ? >> >> 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% >> 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% >> 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% >> 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% >> 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% >> >> Thanks. >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
I don't see any hs_err file in the current directory , so I don't think that is the case. On Wed, Sep 17, 2014 at 2:21 PM, S.L wrote: > > I am not sure, I running a sequence of MRV1 jobs using a bash script , > this seems to happen in the 4th iteration consistently, how do I confirm > this possibility? > > On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> Is it possible that the client JVM is somehow getting killed while the >> YARN application finishes as usual on the cluster in the background? >> >> +Vinod >> >> On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: >> >>> >>>Hi All, >>> >>> I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is >>> when I submit this job YARN the application that is running in YARN is >>> marked as complete even as on console its reported as only 58% complete . I >>> have confirmed that its also not printing the log statements that its >>> supposed to print when the job is actually complete . >>> >>> Please see the output from the job submission console below. It just >>> stops at 58% and job history server and YARN cluster UI reports that this >>> job has already succeeded. Can someone let me know why this is happening ? >>> >>> 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% >>> 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% >>> 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% >>> 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% >>> 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% >>> >>> Thanks. >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > >
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
Does anyone know what the issue could be , is there any property setting that is causing this behavior ? On Wed, Sep 17, 2014 at 3:11 PM, S.L wrote: > > I don't see any hs_err file in the current directory , so I don't think > that is the case. > > On Wed, Sep 17, 2014 at 2:21 PM, S.L wrote: > >> >> I am not sure, I running a sequence of MRV1 jobs using a bash script , >> this seems to happen in the 4th iteration consistently, how do I confirm >> this possibility? >> >> On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli < >> vino...@apache.org> wrote: >> >>> Is it possible that the client JVM is somehow getting killed while the >>> YARN application finishes as usual on the cluster in the background? >>> >>> +Vinod >>> >>> On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: >>> >>>> >>>>Hi All, >>>> >>>> I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is >>>> when I submit this job YARN the application that is running in YARN is >>>> marked as complete even as on console its reported as only 58% complete . I >>>> have confirmed that its also not printing the log statements that its >>>> supposed to print when the job is actually complete . >>>> >>>> Please see the output from the job submission console below. It just >>>> stops at 58% and job history server and YARN cluster UI reports that this >>>> job has already succeeded. Can someone let me know why this is happening ? >>>> >>>> 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% >>>> 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% >>>> 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% >>>> 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% >>>> 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% >>>> >>>> Thanks. >>>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >> >
Hadoop UI - Unable to connect to the application master from the Hadoop UI.
Hi All, I am running a 3 node Apache Hadoop YARN 2.3.0 cluster , after a job is submitted , when I ascess the application master from the UI I get the following exception and I am unable to connect to the Application Master from the UI, can someone let me know , what I need to look at? HTTP ERROR 500 Problem accessing /proxy/application_1411841629814_0032/. Reason: Connection refused Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.(Socket.java:425) at java.net.Socket.(Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
Re: Hadoop UI - Unable to connect to the application master from the Hadoop UI.
The host name is fully qualified , meaning there is nothing more that I can add , it just seems the ports might be messed up , but I don't know which ones On Mon, Sep 29, 2014 at 12:44 AM, Susheel Kumar Gadalay wrote: > I also faced some issue like this. > > It shows the URL in : > > Copy paste the link in browser and expand the host name. > > I set up the host names in windows user etc/hosts file but still it > could not resolve. > > On 9/29/14, S.L wrote: > > Hi All, > > > > I am running a 3 node Apache Hadoop YARN 2.3.0 cluster , after a job is > > submitted , when I ascess the application master from the UI I get the > > following exception and I am unable to connect to the Application Master > > from the UI, can someone let me know , what I need to look at? > > > > HTTP ERROR 500 > > > > Problem accessing /proxy/application_1411841629814_0032/. Reason: > > > > Connection refused > > > > Caused by: > > > > java.net.ConnectException: Connection refused > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > at > > > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > > at > > > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > > at > > > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > at java.net.Socket.connect(Socket.java:579) > > at java.net.Socket.connect(Socket.java:528) > > at java.net.Socket.(Socket.java:425) > > at java.net.Socket.(Socket.java:280) > > at > > > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) > > at > > > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) > > at > > > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > > at > > > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > > at > > > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > > at > > > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > > at > > > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) > > at > > > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) > > at > > > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > > at > > > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > > at > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > > at > > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > > at > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > > > org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > > > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183) > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > org.apache.hadoop.http.NoCacheFilte
Re: Hadoop UI - Unable to connect to the application master from the Hadoop UI.
I did that, still a no go, there is the /proxy in my url ,that seems to be causing the issue. - Reply message - From: "Susheel Kumar Gadalay" To: Subject: Hadoop UI - Unable to connect to the application master from the Hadoop UI. Date: Tue, Sep 30, 2014 2:08 AM What I meant is expand the hostname by IP address. It worked for me. On 9/30/14, S.L wrote: > The host name is fully qualified , meaning there is nothing more that I can > add , it just seems the ports might be messed up , but I don't know which > ones > > On Mon, Sep 29, 2014 at 12:44 AM, Susheel Kumar Gadalay > > wrote: > >> I also faced some issue like this. >> >> It shows the URL in : >> >> Copy paste the link in browser and expand the host name. >> >> I set up the host names in windows user etc/hosts file but still it >> could not resolve. >> >> On 9/29/14, S.L wrote: >> > Hi All, >> > >> > I am running a 3 node Apache Hadoop YARN 2.3.0 cluster , after a job >> > is >> > submitted , when I ascess the application master from the UI I get the >> > following exception and I am unable to connect to the Application >> > Master >> > from the UI, can someone let me know , what I need to look at? >> > >> > HTTP ERROR 500 >> > >> > Problem accessing /proxy/application_1411841629814_0032/. Reason: >> > >> > Connection refused >> > >> > Caused by: >> > >> > java.net.ConnectException: Connection refused >> > at java.net.PlainSocketImpl.socketConnect(Native Method) >> > at >> > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) >> > at >> > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) >> > at >> > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) >> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> > at java.net.Socket.connect(Socket.java:579) >> > at java.net.Socket.connect(Socket.java:528) >> > at java.net.Socket.(Socket.java:425) >> > at java.net.Socket.(Socket.java:280) >> > at >> > >> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) >> > at >> > >> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) >> > at >> > >> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) >> > at >> > >> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) >> > at >> > >> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) >> > at >> > >> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >> > at >> > >> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) >> > at >> > >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) >> > at >> > >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> > at >> > >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) >> > at >> > >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) >> > at >> > >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) >> > at >> > >> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) >> > at >> > >> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) >> > at >> > >> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) >> > at >> > >> com.googl
DataNode Timeout exceptions.
Hi All, I am on Apache Yarn 2.3.0 and lately I have been seeing this exceptions happening frequently.Can someone tell me the root cause of this issue. I have set the the property in mapred-site.xml as follows , is there any other property that I need to set also? mapreduce.task.timeout 180 The time out value for taks, I set this because the JVMs might be busy in GC and this is causing timeout in Hadoop Tasks. 15/05/26 02:06:53 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1751673171-112.123.123.123-1431824104307:blk_1073749395_8571 java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/112.123.123.123:35398 remote=/112.123.123.123:50010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1881) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:726) 15/05/26 02:06:53 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/df/.staging/job_1431824165463_0221 15/05/26 02:06:54 WARN security.UserGroupInformation: PriviledgedActionException as:df (auth:SIMPLE) cause:java.io.IOException: All datanodes 112.123.123.123:50010 are bad. Aborting... 15/05/26 02:06:54 WARN security.UserGroupInformation: PriviledgedActionException as:df (auth:SIMPLE) cause:java.io.IOException: All datanodes 112.123.123.123:50010 are bad. Aborting... Exception in thread "main" java.io.IOException: All datanodes 112.123.123.123:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1023) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483)
Re: DataNode Timeout exceptions.
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:712) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:340) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:101) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221) On Tue, May 26, 2015 at 8:29 PM, Ted Yu wrote: > bq. All datanodes 112.123.123.123:50010 are bad. Aborting... > > How many datanodes do you have ? > > Can you check datanode namenode log ? > > Cheers > > On Tue, May 26, 2015 at 5:00 PM, S.L wrote: > >> Hi All, >> >> I am on Apache Yarn 2.3.0 and lately I have been seeing this exceptions >> happening frequently.Can someone tell me the root cause of this issue. >> >> I have set the the property in mapred-site.xml as follows , is there any >> other property that I need to set also? >> >> >> mapreduce.task.timeout >> 180 >> >> The time out value for taks, I set this because the JVMs might be >> busy in GC and this is causing timeout in Hadoop Tasks. >> >> >> >> >> >> 15/05/26 02:06:53 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor >> exception for block >> BP-1751673171-112.123.123.123-1431824104307:blk_1073749395_8571 >> java.net.SocketTimeoutException: 65000 millis timeout while waiting for >> channel to be ready for read. ch : >> java.nio.channels.SocketChannel[connected local=/112.123.123.123:35398 >> remote=/112.123.123.123:50010] >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) >> at java.io.FilterInputStream.read(FilterInputStream.java:83) >> at java.io.FilterInputStream.read(FilterInputStream.java:83) >> at >> org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1881) >> at >> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:116) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:726) >> 15/05/26 02:06:53 INFO mapreduce.JobSubmitter: Cleaning up the staging >> area /tmp/hadoop-yarn/staging/df/.staging/job_1431824165463_0221 >> 15/05/26 02:06:54 WARN security.UserGroupInformation: >> PriviledgedActionException as:df (auth:SIMPLE) cause:java.io.IOException: >> All datanodes 112.123.123.123:50010 are bad. Aborting... >> 15/05/26 02:06:54 WARN security.UserGroupInformation: >> PriviledgedActionException as:df (auth:SIMPLE) cause:java.io.IOException: >> All datanodes 112.123.123.123:50010 are bad. Aborting... >> Exception in thread "main" java.io.IOException: All datanodes >> 112.123.123.123:50010 are bad. Aborting... >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1023) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838) >> at >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483) >> >> >> >> >
Hadoop YARN Map Reduce Job failing with a Null Pointer Exception.
Can someone please let me know the root cause here. 15/06/05 21:21:31 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 15/06/05 21:21:40 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 15/06/05 21:21:41 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 15/06/05 21:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:df (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:272) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956) Exception in thread "main" java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:272) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956) at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334) at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:386) at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:531) at org.apache.hadoop.mapreduce.Job$5.run(Job.java:668) at org.apache.hadoop.mapreduce.Job$5.run(Job.java:665) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665) at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1366) at org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407) at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1351) at org.apache.nutch.crawl.Crawl.run(Crawl.java:161) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.nutch.crawl.Crawl.main(Crawl.java:55) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:272) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:
