Does smell like a bug as that number you get is simply Long.MAX_VALUE, or 8 exbibytes.
Looking at the sources, this turns out to be a rather funny Java issue (there's a divide by zero happening and [1] suggests Long.MAX_VALUE return in such a case). I've logged a bug report for this at https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a reproducible case. Does this happen consistently for you? [1] http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double) On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <matteo.lan...@lrz.de> wrote: > Hi all, > > I stumbled upon this problem as well while trying to run the default > wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 virtual > machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One node is > used as JT+NN, the other as TT+DN. Security is enabled. The input file is > about 600 kB and the error is > > 2013-06-01 12:22:51,999 WARN org.apache.hadoop.mapred.JobInProgress: No room > for map task. Node 10.156.120.49 has 22854692864 bytes free; but we expect > map to take 9223372036854775807 > > The logfile is attached, together with the configuration files. The version > I'm using is > > Hadoop 1.2.0 > Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 > -r 1479473 > Compiled by hortonfo on Mon May 6 06:59:37 UTC 2013 > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405 > This command was run using > /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar > > If I run the default configuration (i.e. no securty), then the job succeeds. > > Is there something missing in how I set up my nodes? How is it possible that > the envisaged value for the needed space is so big? > > Thanks in advance. > > Matteo > > > >>Which version of Hadoop are you using. A quick search shows me a bug >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show >>similar symptoms. However, that was fixed a long while ago. >> >> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui < >>reduno1...@googlemail.com> wrote: >> >>> This the content of the jobtracker log file : >>> 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgress: Input >>> size for job job_201303231139_0001 = 6950001. Number of splits = 7 >>> 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000000 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000001 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000002 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000003 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000004 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000005 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.JobInProgress: >>> tip:task_201303231139_0001_m_000006 has split on >>> node:/default-rack/hadoop0.novalocal >>> 2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgress: >>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 >>> 2013-03-23 12:06:48,978 INFO org.apache.hadoop.mapred.JobInProgress: Job >>> job_201303231139_0001 initialized successfully with 7 map tasks and 1 >>> reduce tasks. >>> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: Adding >>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip >>> task_201303231139_0001_m_000008, for tracker >>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' >>> 2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgress: Task >>> 'attempt_201303231139_0001_m_000008_0' has completed >>> task_201303231139_0001_m_000008 successfully. >>> 2013-03-23 12:08:00,538 WARN org.apache.hadoop.mapred.JobInProgress: No >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we >>> expect map to take 1317624576693539401 >>> 2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgress: No >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we >>> expect map to take 1317624576693539401 >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we >>> expect map to take 1317624576693539401 >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: No >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; but we >>> expect map to take 1317624576693539401 >>> 2013-03-23 12:08:01,264 WARN org.apache.hadoop.mapred.JobInProgress: No >>> room for map task. Node hadoop1.novalocal has 8807518208 bytes free; but we >>> expect map to take 1317624576693539401 >>> >>> >>> The value in we excpect map to take is too huge 1317624576693539401 >>> bytes !!!!!!! >>> >>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < >>> reduno1...@googlemail.com> wrote: >>> >>>> The estimated value that the hadoop compute is too huge for the simple >>>> example that i am running . >>>> >>>> ---------- Forwarded message ---------- >>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >>>> Date: Sat, Mar 23, 2013 at 11:32 AM >>>> Subject: Re: About running a simple wordcount mapreduce >>>> To: Abdelrahman Shettia <ashet...@hortonworks.com> >>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1...@gmail.com> >>>> >>>> >>>> This the output that I get I am running two machines as you can see do >>>> u see anything suspicious ? >>>> Configured Capacity: 21145698304 (19.69 GB) >>>> Present Capacity: 17615499264 (16.41 GB) >>>> DFS Remaining: 17615441920 (16.41 GB) >>>> DFS Used: 57344 (56 KB) >>>> DFS Used%: 0% >>>> Under replicated blocks: 0 >>>> Blocks with corrupt replicas: 0 >>>> Missing blocks: 0 >>>> >>>> ------------------------------------------------- >>>> Datanodes available: 2 (2 total, 0 dead) >>>> >>>> Name: 11.1.0.6:50010 >>>> Decommission Status : Normal >>>> Configured Capacity: 10572849152 (9.85 GB) >>>> DFS Used: 28672 (28 KB) >>>> Non DFS Used: 1765019648 (1.64 GB) >>>> DFS Remaining: 8807800832(8.2 GB) >>>> DFS Used%: 0% >>>> DFS Remaining%: 83.31% >>>> Last contact: Sat Mar 23 11:30:10 CET 2013 >>>> >>>> >>>> Name: 11.1.0.3:50010 >>>> Decommission Status : Normal >>>> Configured Capacity: 10572849152 (9.85 GB) >>>> DFS Used: 28672 (28 KB) >>>> Non DFS Used: 1765179392 (1.64 GB) >>>> DFS Remaining: 8807641088(8.2 GB) >>>> DFS Used%: 0% >>>> DFS Remaining%: 83.3% >>>> Last contact: Sat Mar 23 11:30:08 CET 2013 >>>> >>>> >>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < >>>> ashet...@hortonworks.com> wrote: >>>> >>>>> Hi Redwane, >>>>> >>>>> Please run the following command as hdfs user on any datanode. The >>>>> output will be something like this. Hope this helps >>>>> >>>>> hadoop dfsadmin -report >>>>> Configured Capacity: 81075068925 (75.51 GB) >>>>> Present Capacity: 70375292928 (65.54 GB) >>>>> DFS Remaining: 69895163904 (65.09 GB) >>>>> DFS Used: 480129024 (457.89 MB) >>>>> DFS Used%: 0.68% >>>>> Under replicated blocks: 0 >>>>> Blocks with corrupt replicas: 0 >>>>> Missing blocks: 0 >>>>> >>>>> Thanks >>>>> -Abdelrahman >>>>> >>>>> >>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 >>>>> <reduno1...@googlemail.com>wrote: >>>>> >>>>>> >>>>>> I have my hosts running on openstack virtual machine instances each >>>>>> instance has 10gb hard disc . Is there a way too see how much space is in >>>>>> the hdfs without web ui . >>>>>> >>>>>> >>>>>> Sent from Samsung Mobile >>>>>> >>>>>> Serge Blazhievsky <hadoop...@gmail.com> wrote: >>>>>> Check web ui how much space you have on hdfs??? >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < >>>>>> ashet...@hortonworks.com> wrote: >>>>>> >>>>>> Hi Redwane , >>>>>> >>>>>> It is possible that the hosts which are running tasks are do not have >>>>>> enough space. Those dirs are confiugred in mapred-site.xml >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui < >>>>>> reduno1...@googlemail.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> ---------- Forwarded message ---------- >>>>>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM >>>>>>> Subject: About running a simple wordcount mapreduce >>>>>>> To: mapreduce-iss...@hadoop.apache.org >>>>>>> >>>>>>> >>>>>>> Hi >>>>>>> I am trying to run a wordcount mapreduce job on several files (<20 >>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce. >>>>>>> The jobtracker log file shows the following warning: >>>>>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for map task. >>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expect map to >>take >>>>>>> 1317624576693539401 >>>>>>> >>>>>>> Please help me , >>>>>>> Best Regards, >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> > > > Matteo Lanati > Distributed Resources Group > Leibniz-Rechenzentrum (LRZ) > Boltzmannstrasse 1 > 85748 Garching b. München (Germany) > Phone: +49 89 35831 8724 -- Harsh J
