Shahab - I see he has mentioned generally that security is enabled (but not that it happens iff security is enabled), and the issue here doesn't have anything to do with security really.
Azurry - Lets discuss the code issues on the JIRA (instead of here) or on the mapreduce-dev lists. On Sat, Jun 1, 2013 at 10:05 PM, Shahab Yunus <shahab.yu...@gmail.com> wrote: > HI Harsh, > > Quick question though: why do you think it only happens if the OP 'uses > security' as he mentioned? > > Regards, > Shahab > > > On Sat, Jun 1, 2013 at 11:49 AM, Harsh J <ha...@cloudera.com> wrote: >> >> Does smell like a bug as that number you get is simply Long.MAX_VALUE, >> or 8 exbibytes. >> >> Looking at the sources, this turns out to be a rather funny Java issue >> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE >> return in such a case). I've logged a bug report for this at >> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a >> reproducible case. >> >> Does this happen consistently for you? >> >> [1] >> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double) >> >> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <matteo.lan...@lrz.de> >> wrote: >> > Hi all, >> > >> > I stumbled upon this problem as well while trying to run the default >> > wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 virtual >> > machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One node is >> > used as JT+NN, the other as TT+DN. Security is enabled. The input file is >> > about 600 kB and the error is >> > >> > 2013-06-01 12:22:51,999 WARN org.apache.hadoop.mapred.JobInProgress: No >> > room for map task. Node 10.156.120.49 has 22854692864 bytes free; but we >> > expect map to take 9223372036854775807 >> > >> > The logfile is attached, together with the configuration files. The >> > version I'm using is >> > >> > Hadoop 1.2.0 >> > Subversion >> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r >> > 1479473 >> > Compiled by hortonfo on Mon May 6 06:59:37 UTC 2013 >> > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405 >> > This command was run using >> > /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar >> > >> > If I run the default configuration (i.e. no securty), then the job >> > succeeds. >> > >> > Is there something missing in how I set up my nodes? How is it possible >> > that the envisaged value for the needed space is so big? >> > >> > Thanks in advance. >> > >> > Matteo >> > >> > >> > >> >>Which version of Hadoop are you using. A quick search shows me a bug >> >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show >> >>similar symptoms. However, that was fixed a long while ago. >> >> >> >> >> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui < >> >>reduno1...@googlemail.com> wrote: >> >> >> >>> This the content of the jobtracker log file : >> >>> 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> Input >> >>> size for job job_201303231139_0001 = 6950001. Number of splits = 7 >> >>> 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000000 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000001 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,930 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000002 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,931 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000003 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,933 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000004 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,934 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000005 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,939 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> tip:task_201303231139_0001_m_000006 has split on >> >>> node:/default-rack/hadoop0.novalocal >> >>> 2013-03-23 12:06:48,950 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 >> >>> 2013-03-23 12:06:48,978 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> Job >> >>> job_201303231139_0001 initialized successfully with 7 map tasks and 1 >> >>> reduce tasks. >> >>> 2013-03-23 12:06:50,855 INFO org.apache.hadoop.mapred.JobTracker: >> >>> Adding >> >>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip >> >>> task_201303231139_0001_m_000008, for tracker >> >>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' >> >>> 2013-03-23 12:08:00,340 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> Task >> >>> 'attempt_201303231139_0001_m_000008_0' has completed >> >>> task_201303231139_0001_m_000008 successfully. >> >>> 2013-03-23 12:08:00,538 WARN org.apache.hadoop.mapred.JobInProgress: >> >>> No >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; >> >>> but we >> >>> expect map to take 1317624576693539401 >> >>> 2013-03-23 12:08:00,543 WARN org.apache.hadoop.mapred.JobInProgress: >> >>> No >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; >> >>> but we >> >>> expect map to take 1317624576693539401 >> >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: >> >>> No >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; >> >>> but we >> >>> expect map to take 1317624576693539401 >> >>> 2013-03-23 12:08:00,544 WARN org.apache.hadoop.mapred.JobInProgress: >> >>> No >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes free; >> >>> but we >> >>> expect map to take 1317624576693539401 >> >>> 2013-03-23 12:08:01,264 WARN org.apache.hadoop.mapred.JobInProgress: >> >>> No >> >>> room for map task. Node hadoop1.novalocal has 8807518208 bytes free; >> >>> but we >> >>> expect map to take 1317624576693539401 >> >>> >> >>> >> >>> The value in we excpect map to take is too huge 1317624576693539401 >> >>> bytes !!!!!!! >> >>> >> >>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < >> >>> reduno1...@googlemail.com> wrote: >> >>> >> >>>> The estimated value that the hadoop compute is too huge for the >> >>>> simple >> >>>> example that i am running . >> >>>> >> >>>> ---------- Forwarded message ---------- >> >>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >> >>>> Date: Sat, Mar 23, 2013 at 11:32 AM >> >>>> Subject: Re: About running a simple wordcount mapreduce >> >>>> To: Abdelrahman Shettia <ashet...@hortonworks.com> >> >>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1...@gmail.com> >> >>>> >> >>>> >> >>>> This the output that I get I am running two machines as you can see >> >>>> do >> >>>> u see anything suspicious ? >> >>>> Configured Capacity: 21145698304 (19.69 GB) >> >>>> Present Capacity: 17615499264 (16.41 GB) >> >>>> DFS Remaining: 17615441920 (16.41 GB) >> >>>> DFS Used: 57344 (56 KB) >> >>>> DFS Used%: 0% >> >>>> Under replicated blocks: 0 >> >>>> Blocks with corrupt replicas: 0 >> >>>> Missing blocks: 0 >> >>>> >> >>>> ------------------------------------------------- >> >>>> Datanodes available: 2 (2 total, 0 dead) >> >>>> >> >>>> Name: 11.1.0.6:50010 >> >>>> Decommission Status : Normal >> >>>> Configured Capacity: 10572849152 (9.85 GB) >> >>>> DFS Used: 28672 (28 KB) >> >>>> Non DFS Used: 1765019648 (1.64 GB) >> >>>> DFS Remaining: 8807800832(8.2 GB) >> >>>> DFS Used%: 0% >> >>>> DFS Remaining%: 83.31% >> >>>> Last contact: Sat Mar 23 11:30:10 CET 2013 >> >>>> >> >>>> >> >>>> Name: 11.1.0.3:50010 >> >>>> Decommission Status : Normal >> >>>> Configured Capacity: 10572849152 (9.85 GB) >> >>>> DFS Used: 28672 (28 KB) >> >>>> Non DFS Used: 1765179392 (1.64 GB) >> >>>> DFS Remaining: 8807641088(8.2 GB) >> >>>> DFS Used%: 0% >> >>>> DFS Remaining%: 83.3% >> >>>> Last contact: Sat Mar 23 11:30:08 CET 2013 >> >>>> >> >>>> >> >>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < >> >>>> ashet...@hortonworks.com> wrote: >> >>>> >> >>>>> Hi Redwane, >> >>>>> >> >>>>> Please run the following command as hdfs user on any datanode. The >> >>>>> output will be something like this. Hope this helps >> >>>>> >> >>>>> hadoop dfsadmin -report >> >>>>> Configured Capacity: 81075068925 (75.51 GB) >> >>>>> Present Capacity: 70375292928 (65.54 GB) >> >>>>> DFS Remaining: 69895163904 (65.09 GB) >> >>>>> DFS Used: 480129024 (457.89 MB) >> >>>>> DFS Used%: 0.68% >> >>>>> Under replicated blocks: 0 >> >>>>> Blocks with corrupt replicas: 0 >> >>>>> Missing blocks: 0 >> >>>>> >> >>>>> Thanks >> >>>>> -Abdelrahman >> >>>>> >> >>>>> >> >>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 >> >>>>> <reduno1...@googlemail.com>wrote: >> >>>>> >> >>>>>> >> >>>>>> I have my hosts running on openstack virtual machine instances each >> >>>>>> instance has 10gb hard disc . Is there a way too see how much space >> >>>>>> is in >> >>>>>> the hdfs without web ui . >> >>>>>> >> >>>>>> >> >>>>>> Sent from Samsung Mobile >> >>>>>> >> >>>>>> Serge Blazhievsky <hadoop...@gmail.com> wrote: >> >>>>>> Check web ui how much space you have on hdfs??? >> >>>>>> >> >>>>>> Sent from my iPhone >> >>>>>> >> >>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < >> >>>>>> ashet...@hortonworks.com> wrote: >> >>>>>> >> >>>>>> Hi Redwane , >> >>>>>> >> >>>>>> It is possible that the hosts which are running tasks are do not >> >>>>>> have >> >>>>>> enough space. Those dirs are confiugred in mapred-site.xml >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui < >> >>>>>> reduno1...@googlemail.com> wrote: >> >>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> ---------- Forwarded message ---------- >> >>>>>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> >> >>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM >> >>>>>>> Subject: About running a simple wordcount mapreduce >> >>>>>>> To: mapreduce-iss...@hadoop.apache.org >> >>>>>>> >> >>>>>>> >> >>>>>>> Hi >> >>>>>>> I am trying to run a wordcount mapreduce job on several files >> >>>>>>> (<20 >> >>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce. >> >>>>>>> The jobtracker log file shows the following warning: >> >>>>>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for map >> >>>>>>> task. >> >>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we expect >> >>>>>>> map to >> >>take >> >>>>>>> 1317624576693539401 >> >>>>>>> >> >>>>>>> Please help me , >> >>>>>>> Best Regards, >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> > >> > >> > Matteo Lanati >> > Distributed Resources Group >> > Leibniz-Rechenzentrum (LRZ) >> > Boltzmannstrasse 1 >> > 85748 Garching b. München (Germany) >> > Phone: +49 89 35831 8724 >> >> >> >> -- >> Harsh J > > -- Harsh J
