can you upgrade to 1.1.2, which is also a stable release, and fixed the bug you facing now.
--Send from my Sony mobile. On Jun 2, 2013 3:23 AM, "Shahab Yunus" <shahab.yu...@gmail.com> wrote: > Thanks Harsh for the reply. I was confused too that why security is > causing this. > > Regards, > Shahab > > > On Sat, Jun 1, 2013 at 12:43 PM, Harsh J <ha...@cloudera.com> wrote: > >> Shahab - I see he has mentioned generally that security is enabled >> (but not that it happens iff security is enabled), and the issue here >> doesn't have anything to do with security really. >> >> Azurry - Lets discuss the code issues on the JIRA (instead of here) or >> on the mapreduce-dev lists. >> >> On Sat, Jun 1, 2013 at 10:05 PM, Shahab Yunus <shahab.yu...@gmail.com> >> wrote: >> > HI Harsh, >> > >> > Quick question though: why do you think it only happens if the OP 'uses >> > security' as he mentioned? >> > >> > Regards, >> > Shahab >> > >> > >> > On Sat, Jun 1, 2013 at 11:49 AM, Harsh J <ha...@cloudera.com> wrote: >> >> >> >> Does smell like a bug as that number you get is simply Long.MAX_VALUE, >> >> or 8 exbibytes. >> >> >> >> Looking at the sources, this turns out to be a rather funny Java issue >> >> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE >> >> return in such a case). I've logged a bug report for this at >> >> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a >> >> reproducible case. >> >> >> >> Does this happen consistently for you? >> >> >> >> [1] >> >> >> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double) >> >> >> >> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <matteo.lan...@lrz.de> >> >> wrote: >> >> > Hi all, >> >> > >> >> > I stumbled upon this problem as well while trying to run the default >> >> > wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 >> virtual >> >> > machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One >> node is >> >> > used as JT+NN, the other as TT+DN. Security is enabled. The input >> file is >> >> > about 600 kB and the error is >> >> > >> >> > 2013-06-01 12:22:51,999 WARN org.apache.hadoop.mapred.JobInProgress: >> No >> >> > room for map task. Node 10.156.120.49 has 22854692864 bytes free; >> but we >> >> > expect map to take 9223372036854775807 >> >> > >> >> > The logfile is attached, together with the configuration files. The >> >> > version I'm using is >> >> > >> >> > Hadoop 1.2.0 >> >> > Subversion >> >> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2-r >> >> > 1479473 >> >> > Compiled by hortonfo on Mon May 6 06:59:37 UTC 2013 >> >> > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405 >> >> > This command was run using >> >> > /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar >> >> > >> >> > If I run the default configuration (i.e. no securty), then the job >> >> > succeeds. >> >> > >> >> > Is there something missing in how I set up my nodes? How is it >> possible >> >> > that the envisaged value for the needed space is so big? >> >> > >> >> > Thanks in advance. >> >> > >> >> > Matteo >> >> > >> >> > >> >> > >> >> >>Which version of Hadoop are you using. A quick search shows me a bug >> >> >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show >> >> >>similar symptoms. However, that was fixed a long while ago. >> >> >> >> >> >> >> >> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui < >> >> >>reduno1...@googlemail.com> wrote: >> >> >> >> >> >>> This the content of the jobtracker log file : >> >> >>> 2013-03-23 12:06:48,912 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> Input >> >> >>> size for job job_201303231139_0001 = 6950001. Number of splits = 7 >> >> >>> 2013-03-23 12:06:48,925 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000000 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,927 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000001 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,930 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000002 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,931 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000003 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,933 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000004 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,934 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000005 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,939 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> tip:task_201303231139_0001_m_000006 has split on >> >> >>> node:/default-rack/hadoop0.novalocal >> >> >>> 2013-03-23 12:06:48,950 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 >> >> >>> 2013-03-23 12:06:48,978 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> Job >> >> >>> job_201303231139_0001 initialized successfully with 7 map tasks >> and 1 >> >> >>> reduce tasks. >> >> >>> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: >> >> >>> Adding >> >> >>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip >> >> >>> task_201303231139_0001_m_000008, for tracker >> >> >>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' >> >> >>> 2013-03-23 12:08:00,340 INFO >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> Task >> >> >>> 'attempt_201303231139_0001_m_000008_0' has completed >> >> >>> task_201303231139_0001_m_000008 successfully. >> >> >>> 2013-03-23 12:08:00,538 WARN >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> No >> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >> free; >> >> >>> but we >> >> >>> expect map to take 1317624576693539401 >> >> >>> 2013-03-23 12:08:00,543 WARN >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> No >> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >> free; >> >> >>> but we >> >> >>> expect map to take 1317624576693539401 >> >> >>> 2013-03-23 12:08:00,544 WARN >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> No >> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >> free; >> >> >>> but we >> >> >>> expect map to take 1317624576693539401 >> >> >>> 2013-03-23 12:08:00,544 WARN >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> No >> >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes >> free; >> >> >>> but we >> >> >>> expect map to take 1317624576693539401 >> >> >>> 2013-03-23 12:08:01,264 WARN >> org.apache.hadoop.mapred.JobInProgress: >> >> >>> No >> >> >>> room for map task. Node hadoop1.novalocal has 8807518208 bytes >> free; >> >> >>> but we >> >> >>> expect map to take 1317624576693539401 >> >> >>> >> >> >>> >> >> >>> The value in we excpect map to take is too huge >> 1317624576693539401 >> >> >>> bytes !!!!!!! >> >> >>> >> >> >>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < >> >> >>> reduno1...@googlemail.com> wrote: >> >> >>> >> >> >>>> The estimated value that the hadoop compute is too huge for the >> >> >>>> simple >> >> >>>> example that i am running . >> >> >>>> >> >> >>>> ---------- Forwarded message ---------- >> >> >>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >> >> >>>> Date: Sat, Mar 23, 2013 at 11:32 AM >> >> >>>> Subject: Re: About running a simple wordcount mapreduce >> >> >>>> To: Abdelrahman Shettia <ashet...@hortonworks.com> >> >> >>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1...@gmail.com> >> >> >>>> >> >> >>>> >> >> >>>> This the output that I get I am running two machines as you can >> see >> >> >>>> do >> >> >>>> u see anything suspicious ? >> >> >>>> Configured Capacity: 21145698304 (19.69 GB) >> >> >>>> Present Capacity: 17615499264 (16.41 GB) >> >> >>>> DFS Remaining: 17615441920 (16.41 GB) >> >> >>>> DFS Used: 57344 (56 KB) >> >> >>>> DFS Used%: 0% >> >> >>>> Under replicated blocks: 0 >> >> >>>> Blocks with corrupt replicas: 0 >> >> >>>> Missing blocks: 0 >> >> >>>> >> >> >>>> ------------------------------------------------- >> >> >>>> Datanodes available: 2 (2 total, 0 dead) >> >> >>>> >> >> >>>> Name: 11.1.0.6:50010 >> >> >>>> Decommission Status : Normal >> >> >>>> Configured Capacity: 10572849152 (9.85 GB) >> >> >>>> DFS Used: 28672 (28 KB) >> >> >>>> Non DFS Used: 1765019648 (1.64 GB) >> >> >>>> DFS Remaining: 8807800832(8.2 GB) >> >> >>>> DFS Used%: 0% >> >> >>>> DFS Remaining%: 83.31% >> >> >>>> Last contact: Sat Mar 23 11:30:10 CET 2013 >> >> >>>> >> >> >>>> >> >> >>>> Name: 11.1.0.3:50010 >> >> >>>> Decommission Status : Normal >> >> >>>> Configured Capacity: 10572849152 (9.85 GB) >> >> >>>> DFS Used: 28672 (28 KB) >> >> >>>> Non DFS Used: 1765179392 (1.64 GB) >> >> >>>> DFS Remaining: 8807641088(8.2 GB) >> >> >>>> DFS Used%: 0% >> >> >>>> DFS Remaining%: 83.3% >> >> >>>> Last contact: Sat Mar 23 11:30:08 CET 2013 >> >> >>>> >> >> >>>> >> >> >>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < >> >> >>>> ashet...@hortonworks.com> wrote: >> >> >>>> >> >> >>>>> Hi Redwane, >> >> >>>>> >> >> >>>>> Please run the following command as hdfs user on any datanode. >> The >> >> >>>>> output will be something like this. Hope this helps >> >> >>>>> >> >> >>>>> hadoop dfsadmin -report >> >> >>>>> Configured Capacity: 81075068925 (75.51 GB) >> >> >>>>> Present Capacity: 70375292928 (65.54 GB) >> >> >>>>> DFS Remaining: 69895163904 (65.09 GB) >> >> >>>>> DFS Used: 480129024 (457.89 MB) >> >> >>>>> DFS Used%: 0.68% >> >> >>>>> Under replicated blocks: 0 >> >> >>>>> Blocks with corrupt replicas: 0 >> >> >>>>> Missing blocks: 0 >> >> >>>>> >> >> >>>>> Thanks >> >> >>>>> -Abdelrahman >> >> >>>>> >> >> >>>>> >> >> >>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 >> >> >>>>> <reduno1...@googlemail.com>wrote: >> >> >>>>> >> >> >>>>>> >> >> >>>>>> I have my hosts running on openstack virtual machine instances >> each >> >> >>>>>> instance has 10gb hard disc . Is there a way too see how much >> space >> >> >>>>>> is in >> >> >>>>>> the hdfs without web ui . >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> Sent from Samsung Mobile >> >> >>>>>> >> >> >>>>>> Serge Blazhievsky <hadoop...@gmail.com> wrote: >> >> >>>>>> Check web ui how much space you have on hdfs??? >> >> >>>>>> >> >> >>>>>> Sent from my iPhone >> >> >>>>>> >> >> >>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < >> >> >>>>>> ashet...@hortonworks.com> wrote: >> >> >>>>>> >> >> >>>>>> Hi Redwane , >> >> >>>>>> >> >> >>>>>> It is possible that the hosts which are running tasks are do not >> >> >>>>>> have >> >> >>>>>> enough space. Those dirs are confiugred in mapred-site.xml >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui < >> >> >>>>>> reduno1...@googlemail.com> wrote: >> >> >>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> ---------- Forwarded message ---------- >> >> >>>>>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >> >> >>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM >> >> >>>>>>> Subject: About running a simple wordcount mapreduce >> >> >>>>>>> To: mapreduce-iss...@hadoop.apache.org >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Hi >> >> >>>>>>> I am trying to run a wordcount mapreduce job on several files >> >> >>>>>>> (<20 >> >> >>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce. >> >> >>>>>>> The jobtracker log file shows the following warning: >> >> >>>>>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for map >> >> >>>>>>> task. >> >> >>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expect >> >> >>>>>>> map to >> >> >>take >> >> >>>>>>> 1317624576693539401 >> >> >>>>>>> >> >> >>>>>>> Please help me , >> >> >>>>>>> Best Regards, >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>> >> >> >>>>> >> >> >>>> >> >> >>>> >> >> >>> >> >> > >> >> > >> >> > Matteo Lanati >> >> > Distributed Resources Group >> >> > Leibniz-Rechenzentrum (LRZ) >> >> > Boltzmannstrasse 1 >> >> > 85748 Garching b. München (Germany) >> >> > Phone: +49 89 35831 8724 >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J >> > >
