Hi all, I finally solved the problem. It was due to the cloud middleware I used to run the Hadoop VMs. The domain type in the libvirt xm file was incorrectly set to 'qemu'. Once I fixed this and changed to 'kvm' everything started to work properly. Thanks for the support.
Matteo On Jun 4, 2013, at 5:43 PM, Alexander Alten-Lorenz <wget.n...@gmail.com> wrote: > Hi Matteo, > > Are you able to add more space to your test machines? Also, what says the pi > example (hadoop jar hadoop-examples pi 10 10 ? > > - Alex > > On Jun 4, 2013, at 4:34 PM, "Lanati, Matteo" <matteo.lan...@lrz.de> wrote: > >> Hi again, >> >> unfortunately my problem is not solved. >> I downloaded Hadoop v. 1.1.2a and made a basic configuration as suggested in >> [1]. >> No security, no ACLs, default scheduler ... The files are attached. >> I still have the same error message. I also tried another Java version (6u45 >> instead of 7u21). >> How can I increase the debug level to have a deeper look? >> Thanks, >> >> Matteo >> >> >> [1] >> http://hadoop.apache.org/docs/r1.1.2/cluster_setup.html#Cluster+Restartability >> On Jun 4, 2013, at 3:52 AM, Azuryy Yu <azury...@gmail.com> wrote: >> >>> Hi Harsh, >>> >>> I need to take care my eyes recently, I mis-read 1.2.0 to 1.0.2, so I said >>> upgrade. Sorry. >>> >>> >>> On Tue, Jun 4, 2013 at 9:46 AM, Harsh J <ha...@cloudera.com> wrote: >>> Azuryy, >>> >>> 1.1.2 < 1.2.0. Its not an upgrade you're suggesting there. If you feel >>> there's been a regression, can you comment that on the JIRA? >>> >>> On Tue, Jun 4, 2013 at 6:57 AM, Azuryy Yu <azury...@gmail.com> wrote: >>>> yes. hadoop-1.1.2 was released on Jan. 31st. just download it. >>>> >>>> >>>> On Tue, Jun 4, 2013 at 6:33 AM, Lanati, Matteo <matteo.lan...@lrz.de> >>>> wrote: >>>>> >>>>> Hi Azuryy, >>>>> >>>>> thanks for the update. Sorry for the silly question, but where can I >>>>> download the patched version? >>>>> If I look into the closest mirror (i.e. >>>>> http://mirror.netcologne.de/apache.org/hadoop/common/), I can see that the >>>>> Hadoop 1.1.2 version was last updated on Jan. 31st. >>>>> Thanks in advance, >>>>> >>>>> Matteo >>>>> >>>>> PS: just to confirm that I tried a minimal Hadoop 1.2.0 setup, so without >>>>> any security, and the problem is there. >>>>> >>>>> On Jun 3, 2013, at 3:02 PM, Azuryy Yu <azury...@gmail.com> wrote: >>>>> >>>>>> can you upgrade to 1.1.2, which is also a stable release, and fixed the >>>>>> bug you facing now. >>>>>> >>>>>> --Send from my Sony mobile. >>>>>> >>>>>> On Jun 2, 2013 3:23 AM, "Shahab Yunus" <shahab.yu...@gmail.com> wrote: >>>>>> Thanks Harsh for the reply. I was confused too that why security is >>>>>> causing this. >>>>>> >>>>>> Regards, >>>>>> Shahab >>>>>> >>>>>> >>>>>> On Sat, Jun 1, 2013 at 12:43 PM, Harsh J <ha...@cloudera.com> wrote: >>>>>> Shahab - I see he has mentioned generally that security is enabled >>>>>> (but not that it happens iff security is enabled), and the issue here >>>>>> doesn't have anything to do with security really. >>>>>> >>>>>> Azurry - Lets discuss the code issues on the JIRA (instead of here) or >>>>>> on the mapreduce-dev lists. >>>>>> >>>>>> On Sat, Jun 1, 2013 at 10:05 PM, Shahab Yunus <shahab.yu...@gmail.com> >>>>>> wrote: >>>>>>> HI Harsh, >>>>>>> >>>>>>> Quick question though: why do you think it only happens if the OP >>>>>>> 'uses >>>>>>> security' as he mentioned? >>>>>>> >>>>>>> Regards, >>>>>>> Shahab >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 1, 2013 at 11:49 AM, Harsh J <ha...@cloudera.com> wrote: >>>>>>>> >>>>>>>> Does smell like a bug as that number you get is simply >>>>>>>> Long.MAX_VALUE, >>>>>>>> or 8 exbibytes. >>>>>>>> >>>>>>>> Looking at the sources, this turns out to be a rather funny Java >>>>>>>> issue >>>>>>>> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE >>>>>>>> return in such a case). I've logged a bug report for this at >>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a >>>>>>>> reproducible case. >>>>>>>> >>>>>>>> Does this happen consistently for you? >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double) >>>>>>>> >>>>>>>> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <matteo.lan...@lrz.de> >>>>>>>> wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I stumbled upon this problem as well while trying to run the >>>>>>>>> default >>>>>>>>> wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 >>>>>>>>> virtual >>>>>>>>> machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One >>>>>>>>> node is >>>>>>>>> used as JT+NN, the other as TT+DN. Security is enabled. The input >>>>>>>>> file is >>>>>>>>> about 600 kB and the error is >>>>>>>>> >>>>>>>>> 2013-06-01 12:22:51,999 WARN >>>>>>>>> org.apache.hadoop.mapred.JobInProgress: No >>>>>>>>> room for map task. Node 10.156.120.49 has 22854692864 bytes free; >>>>>>>>> but we >>>>>>>>> expect map to take 9223372036854775807 >>>>>>>>> >>>>>>>>> The logfile is attached, together with the configuration files. The >>>>>>>>> version I'm using is >>>>>>>>> >>>>>>>>> Hadoop 1.2.0 >>>>>>>>> Subversion >>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 >>>>>>>>> -r >>>>>>>>> 1479473 >>>>>>>>> Compiled by hortonfo on Mon May 6 06:59:37 UTC 2013 >>>>>>>>> From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405 >>>>>>>>> This command was run using >>>>>>>>> /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar >>>>>>>>> >>>>>>>>> If I run the default configuration (i.e. no securty), then the job >>>>>>>>> succeeds. >>>>>>>>> >>>>>>>>> Is there something missing in how I set up my nodes? How is it >>>>>>>>> possible >>>>>>>>> that the envisaged value for the needed space is so big? >>>>>>>>> >>>>>>>>> Thanks in advance. >>>>>>>>> >>>>>>>>> Matteo >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Which version of Hadoop are you using. A quick search shows me a >>>>>>>>>> bug >>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-5241 that seems to >>>>>>>>>> show >>>>>>>>>> similar symptoms. However, that was fixed a long while ago. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui < >>>>>>>>>> reduno1...@googlemail.com> wrote: >>>>>>>>>> >>>>>>>>>>> This the content of the jobtracker log file : >>>>>>>>>>> 2013-03-23 12:06:48,912 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> Input >>>>>>>>>>> size for job job_201303231139_0001 = 6950001. Number of splits = >>>>>>>>>>> 7 >>>>>>>>>>> 2013-03-23 12:06:48,925 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000000 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,927 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000001 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,930 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000002 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,931 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000003 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,933 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000004 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,934 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000005 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,939 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> tip:task_201303231139_0001_m_000006 has split on >>>>>>>>>>> node:/default-rack/hadoop0.novalocal >>>>>>>>>>> 2013-03-23 12:06:48,950 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 >>>>>>>>>>> 2013-03-23 12:06:48,978 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> Job >>>>>>>>>>> job_201303231139_0001 initialized successfully with 7 map tasks >>>>>>>>>>> and 1 >>>>>>>>>>> reduce tasks. >>>>>>>>>>> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: >>>>>>>>>>> Adding >>>>>>>>>>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip >>>>>>>>>>> task_201303231139_0001_m_000008, for tracker >>>>>>>>>>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' >>>>>>>>>>> 2013-03-23 12:08:00,340 INFO >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> Task >>>>>>>>>>> 'attempt_201303231139_0001_m_000008_0' has completed >>>>>>>>>>> task_201303231139_0001_m_000008 successfully. >>>>>>>>>>> 2013-03-23 12:08:00,538 WARN >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> No >>>>>>>>>>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >>>>>>>>>>> free; >>>>>>>>>>> but we >>>>>>>>>>> expect map to take 1317624576693539401 >>>>>>>>>>> 2013-03-23 12:08:00,543 WARN >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> No >>>>>>>>>>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >>>>>>>>>>> free; >>>>>>>>>>> but we >>>>>>>>>>> expect map to take 1317624576693539401 >>>>>>>>>>> 2013-03-23 12:08:00,544 WARN >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> No >>>>>>>>>>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >>>>>>>>>>> free; >>>>>>>>>>> but we >>>>>>>>>>> expect map to take 1317624576693539401 >>>>>>>>>>> 2013-03-23 12:08:00,544 WARN >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> No >>>>>>>>>>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >>>>>>>>>>> free; >>>>>>>>>>> but we >>>>>>>>>>> expect map to take 1317624576693539401 >>>>>>>>>>> 2013-03-23 12:08:01,264 WARN >>>>>>>>>>> org.apache.hadoop.mapred.JobInProgress: >>>>>>>>>>> No >>>>>>>>>>> room for map task. Node hadoop1.novalocal has 8807518208 bytes >>>>>>>>>>> free; >>>>>>>>>>> but we >>>>>>>>>>> expect map to take 1317624576693539401 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The value in we excpect map to take is too huge >>>>>>>>>>> 1317624576693539401 >>>>>>>>>>> bytes !!!!!!! >>>>>>>>>>> >>>>>>>>>>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < >>>>>>>>>>> reduno1...@googlemail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> The estimated value that the hadoop compute is too huge for the >>>>>>>>>>>> simple >>>>>>>>>>>> example that i am running . >>>>>>>>>>>> >>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >>>>>>>>>>>> Date: Sat, Mar 23, 2013 at 11:32 AM >>>>>>>>>>>> Subject: Re: About running a simple wordcount mapreduce >>>>>>>>>>>> To: Abdelrahman Shettia <ashet...@hortonworks.com> >>>>>>>>>>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1...@gmail.com> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This the output that I get I am running two machines as you can >>>>>>>>>>>> see >>>>>>>>>>>> do >>>>>>>>>>>> u see anything suspicious ? >>>>>>>>>>>> Configured Capacity: 21145698304 (19.69 GB) >>>>>>>>>>>> Present Capacity: 17615499264 (16.41 GB) >>>>>>>>>>>> DFS Remaining: 17615441920 (16.41 GB) >>>>>>>>>>>> DFS Used: 57344 (56 KB) >>>>>>>>>>>> DFS Used%: 0% >>>>>>>>>>>> Under replicated blocks: 0 >>>>>>>>>>>> Blocks with corrupt replicas: 0 >>>>>>>>>>>> Missing blocks: 0 >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------- >>>>>>>>>>>> Datanodes available: 2 (2 total, 0 dead) >>>>>>>>>>>> >>>>>>>>>>>> Name: 11.1.0.6:50010 >>>>>>>>>>>> Decommission Status : Normal >>>>>>>>>>>> Configured Capacity: 10572849152 (9.85 GB) >>>>>>>>>>>> DFS Used: 28672 (28 KB) >>>>>>>>>>>> Non DFS Used: 1765019648 (1.64 GB) >>>>>>>>>>>> DFS Remaining: 8807800832(8.2 GB) >>>>>>>>>>>> DFS Used%: 0% >>>>>>>>>>>> DFS Remaining%: 83.31% >>>>>>>>>>>> Last contact: Sat Mar 23 11:30:10 CET 2013 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Name: 11.1.0.3:50010 >>>>>>>>>>>> Decommission Status : Normal >>>>>>>>>>>> Configured Capacity: 10572849152 (9.85 GB) >>>>>>>>>>>> DFS Used: 28672 (28 KB) >>>>>>>>>>>> Non DFS Used: 1765179392 (1.64 GB) >>>>>>>>>>>> DFS Remaining: 8807641088(8.2 GB) >>>>>>>>>>>> DFS Used%: 0% >>>>>>>>>>>> DFS Remaining%: 83.3% >>>>>>>>>>>> Last contact: Sat Mar 23 11:30:08 CET 2013 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < >>>>>>>>>>>> ashet...@hortonworks.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Redwane, >>>>>>>>>>>>> >>>>>>>>>>>>> Please run the following command as hdfs user on any datanode. >>>>>>>>>>>>> The >>>>>>>>>>>>> output will be something like this. Hope this helps >>>>>>>>>>>>> >>>>>>>>>>>>> hadoop dfsadmin -report >>>>>>>>>>>>> Configured Capacity: 81075068925 (75.51 GB) >>>>>>>>>>>>> Present Capacity: 70375292928 (65.54 GB) >>>>>>>>>>>>> DFS Remaining: 69895163904 (65.09 GB) >>>>>>>>>>>>> DFS Used: 480129024 (457.89 MB) >>>>>>>>>>>>> DFS Used%: 0.68% >>>>>>>>>>>>> Under replicated blocks: 0 >>>>>>>>>>>>> Blocks with corrupt replicas: 0 >>>>>>>>>>>>> Missing blocks: 0 >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> -Abdelrahman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 >>>>>>>>>>>>> <reduno1...@googlemail.com>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have my hosts running on openstack virtual machine instances >>>>>>>>>>>>>> each >>>>>>>>>>>>>> instance has 10gb hard disc . Is there a way too see how much >>>>>>>>>>>>>> space >>>>>>>>>>>>>> is in >>>>>>>>>>>>>> the hdfs without web ui . >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from Samsung Mobile >>>>>>>>>>>>>> >>>>>>>>>>>>>> Serge Blazhievsky <hadoop...@gmail.com> wrote: >>>>>>>>>>>>>> Check web ui how much space you have on hdfs??? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < >>>>>>>>>>>>>> ashet...@hortonworks.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Redwane , >>>>>>>>>>>>>> >>>>>>>>>>>>>> It is possible that the hosts which are running tasks are do >>>>>>>>>>>>>> not >>>>>>>>>>>>>> have >>>>>>>>>>>>>> enough space. Those dirs are confiugred in mapred-site.xml >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui < >>>>>>>>>>>>>> reduno1...@googlemail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >>>>>>>>>>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM >>>>>>>>>>>>>>> Subject: About running a simple wordcount mapreduce >>>>>>>>>>>>>>> To: mapreduce-iss...@hadoop.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>> I am trying to run a wordcount mapreduce job on several >>>>>>>>>>>>>>> files >>>>>>>>>>>>>>> (<20 >>>>>>>>>>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce. >>>>>>>>>>>>>>> The jobtracker log file shows the following warning: >>>>>>>>>>>>>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for map >>>>>>>>>>>>>>> task. >>>>>>>>>>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we >>>>>>>>>>>>>>> expect >>>>>>>>>>>>>>> map to >>>>>>>>>> take >>>>>>>>>>>>>>> 1317624576693539401 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please help me , >>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Matteo Lanati >>>>>>>>> Distributed Resources Group >>>>>>>>> Leibniz-Rechenzentrum (LRZ) >>>>>>>>> Boltzmannstrasse 1 >>>>>>>>> 85748 Garching b. München (Germany) >>>>>>>>> Phone: +49 89 35831 8724 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Harsh J >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Harsh J >>>>>> >>>>> >>>>> Matteo Lanati >>>>> Distributed Resources Group >>>>> Leibniz-Rechenzentrum (LRZ) >>>>> Boltzmannstrasse 1 >>>>> 85748 Garching b. München (Germany) >>>>> Phone: +49 89 35831 8724 >>>>> >>>> >>> >>> >>> >>> -- >>> Harsh J >>> >> >> Matteo Lanati >> Distributed Resources Group >> Leibniz-Rechenzentrum (LRZ) >> Boltzmannstrasse 1 >> 85748 Garching b. München (Germany) >> Phone: +49 89 35831 8724 >> <core-site.xml><hdfs-site.xml><mapred-site.xml> > Matteo Lanati Distributed Resources Group Leibniz-Rechenzentrum (LRZ) Boltzmannstrasse 1 85748 Garching b. München (Germany) Phone: +49 89 35831 8724
