Hi Harsh, I need to take care my eyes recently, I mis-read 1.2.0 to 1.0.2, so I said upgrade. Sorry.
On Tue, Jun 4, 2013 at 9:46 AM, Harsh J <ha...@cloudera.com> wrote: > Azuryy, > > 1.1.2 < 1.2.0. Its not an upgrade you're suggesting there. If you feel > there's been a regression, can you comment that on the JIRA? > > On Tue, Jun 4, 2013 at 6:57 AM, Azuryy Yu <azury...@gmail.com> wrote: > > yes. hadoop-1.1.2 was released on Jan. 31st. just download it. > > > > > > On Tue, Jun 4, 2013 at 6:33 AM, Lanati, Matteo <matteo.lan...@lrz.de> > wrote: > >> > >> Hi Azuryy, > >> > >> thanks for the update. Sorry for the silly question, but where can I > >> download the patched version? > >> If I look into the closest mirror (i.e. > >> http://mirror.netcologne.de/apache.org/hadoop/common/), I can see that > the > >> Hadoop 1.1.2 version was last updated on Jan. 31st. > >> Thanks in advance, > >> > >> Matteo > >> > >> PS: just to confirm that I tried a minimal Hadoop 1.2.0 setup, so > without > >> any security, and the problem is there. > >> > >> On Jun 3, 2013, at 3:02 PM, Azuryy Yu <azury...@gmail.com> wrote: > >> > >> > can you upgrade to 1.1.2, which is also a stable release, and fixed > the > >> > bug you facing now. > >> > > >> > --Send from my Sony mobile. > >> > > >> > On Jun 2, 2013 3:23 AM, "Shahab Yunus" <shahab.yu...@gmail.com> > wrote: > >> > Thanks Harsh for the reply. I was confused too that why security is > >> > causing this. > >> > > >> > Regards, > >> > Shahab > >> > > >> > > >> > On Sat, Jun 1, 2013 at 12:43 PM, Harsh J <ha...@cloudera.com> wrote: > >> > Shahab - I see he has mentioned generally that security is enabled > >> > (but not that it happens iff security is enabled), and the issue here > >> > doesn't have anything to do with security really. > >> > > >> > Azurry - Lets discuss the code issues on the JIRA (instead of here) or > >> > on the mapreduce-dev lists. > >> > > >> > On Sat, Jun 1, 2013 at 10:05 PM, Shahab Yunus <shahab.yu...@gmail.com > > > >> > wrote: > >> > > HI Harsh, > >> > > > >> > > Quick question though: why do you think it only happens if the OP > >> > > 'uses > >> > > security' as he mentioned? > >> > > > >> > > Regards, > >> > > Shahab > >> > > > >> > > > >> > > On Sat, Jun 1, 2013 at 11:49 AM, Harsh J <ha...@cloudera.com> > wrote: > >> > >> > >> > >> Does smell like a bug as that number you get is simply > >> > >> Long.MAX_VALUE, > >> > >> or 8 exbibytes. > >> > >> > >> > >> Looking at the sources, this turns out to be a rather funny Java > >> > >> issue > >> > >> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE > >> > >> return in such a case). I've logged a bug report for this at > >> > >> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a > >> > >> reproducible case. > >> > >> > >> > >> Does this happen consistently for you? > >> > >> > >> > >> [1] > >> > >> > >> > >> > http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double) > >> > >> > >> > >> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo < > matteo.lan...@lrz.de> > >> > >> wrote: > >> > >> > Hi all, > >> > >> > > >> > >> > I stumbled upon this problem as well while trying to run the > >> > >> > default > >> > >> > wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 > >> > >> > virtual > >> > >> > machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One > >> > >> > node is > >> > >> > used as JT+NN, the other as TT+DN. Security is enabled. The input > >> > >> > file is > >> > >> > about 600 kB and the error is > >> > >> > > >> > >> > 2013-06-01 12:22:51,999 WARN > >> > >> > org.apache.hadoop.mapred.JobInProgress: No > >> > >> > room for map task. Node 10.156.120.49 has 22854692864 bytes free; > >> > >> > but we > >> > >> > expect map to take 9223372036854775807 > >> > >> > > >> > >> > The logfile is attached, together with the configuration files. > The > >> > >> > version I'm using is > >> > >> > > >> > >> > Hadoop 1.2.0 > >> > >> > Subversion > >> > >> > > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 > >> > >> > -r > >> > >> > 1479473 > >> > >> > Compiled by hortonfo on Mon May 6 06:59:37 UTC 2013 > >> > >> > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405 > >> > >> > This command was run using > >> > >> > /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar > >> > >> > > >> > >> > If I run the default configuration (i.e. no securty), then the > job > >> > >> > succeeds. > >> > >> > > >> > >> > Is there something missing in how I set up my nodes? How is it > >> > >> > possible > >> > >> > that the envisaged value for the needed space is so big? > >> > >> > > >> > >> > Thanks in advance. > >> > >> > > >> > >> > Matteo > >> > >> > > >> > >> > > >> > >> > > >> > >> >>Which version of Hadoop are you using. A quick search shows me a > >> > >> >> bug > >> > >> >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to > >> > >> >> show > >> > >> >>similar symptoms. However, that was fixed a long while ago. > >> > >> >> > >> > >> >> > >> > >> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui < > >> > >> >>reduno1...@googlemail.com> wrote: > >> > >> >> > >> > >> >>> This the content of the jobtracker log file : > >> > >> >>> 2013-03-23 12:06:48,912 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> Input > >> > >> >>> size for job job_201303231139_0001 = 6950001. Number of splits > = > >> > >> >>> 7 > >> > >> >>> 2013-03-23 12:06:48,925 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000000 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,927 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000001 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,930 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000002 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,931 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000003 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,933 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000004 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,934 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000005 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,939 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> tip:task_201303231139_0001_m_000006 has split on > >> > >> >>> node:/default-rack/hadoop0.novalocal > >> > >> >>> 2013-03-23 12:06:48,950 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> job_201303231139_0001 LOCALITY_WAIT_FACTOR=0.5 > >> > >> >>> 2013-03-23 12:06:48,978 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> Job > >> > >> >>> job_201303231139_0001 initialized successfully with 7 map tasks > >> > >> >>> and 1 > >> > >> >>> reduce tasks. > >> > >> >>> 2013-03-23 12:06:50,855 INFO > org.apache.hadoop.mapred.JobTracker: > >> > >> >>> Adding > >> > >> >>> task (JOB_SETUP) 'attempt_201303231139_0001_m_000008_0' to tip > >> > >> >>> task_201303231139_0001_m_000008, for tracker > >> > >> >>> 'tracker_hadoop0.novalocal:hadoop0.novalocal/127.0.0.1:44879' > >> > >> >>> 2013-03-23 12:08:00,340 INFO > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> Task > >> > >> >>> 'attempt_201303231139_0001_m_000008_0' has completed > >> > >> >>> task_201303231139_0001_m_000008 successfully. > >> > >> >>> 2013-03-23 12:08:00,538 WARN > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> No > >> > >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes > >> > >> >>> free; > >> > >> >>> but we > >> > >> >>> expect map to take 1317624576693539401 > >> > >> >>> 2013-03-23 12:08:00,543 WARN > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> No > >> > >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes > >> > >> >>> free; > >> > >> >>> but we > >> > >> >>> expect map to take 1317624576693539401 > >> > >> >>> 2013-03-23 12:08:00,544 WARN > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> No > >> > >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes > >> > >> >>> free; > >> > >> >>> but we > >> > >> >>> expect map to take 1317624576693539401 > >> > >> >>> 2013-03-23 12:08:00,544 WARN > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> No > >> > >> >>> room for map task. Node hadoop0.novalocal has 8791543808 bytes > >> > >> >>> free; > >> > >> >>> but we > >> > >> >>> expect map to take 1317624576693539401 > >> > >> >>> 2013-03-23 12:08:01,264 WARN > >> > >> >>> org.apache.hadoop.mapred.JobInProgress: > >> > >> >>> No > >> > >> >>> room for map task. Node hadoop1.novalocal has 8807518208 bytes > >> > >> >>> free; > >> > >> >>> but we > >> > >> >>> expect map to take 1317624576693539401 > >> > >> >>> > >> > >> >>> > >> > >> >>> The value in we excpect map to take is too huge > >> > >> >>> 1317624576693539401 > >> > >> >>> bytes !!!!!!! > >> > >> >>> > >> > >> >>> On Sat, Mar 23, 2013 at 11:37 AM, Redwane belmaati cherkaoui < > >> > >> >>> reduno1...@googlemail.com> wrote: > >> > >> >>> > >> > >> >>>> The estimated value that the hadoop compute is too huge for > the > >> > >> >>>> simple > >> > >> >>>> example that i am running . > >> > >> >>>> > >> > >> >>>> ---------- Forwarded message ---------- > >> > >> >>>> From: Redwane belmaati cherkaoui <reduno1...@googlemail.com> > >> > >> >>>> Date: Sat, Mar 23, 2013 at 11:32 AM > >> > >> >>>> Subject: Re: About running a simple wordcount mapreduce > >> > >> >>>> To: Abdelrahman Shettia <ashet...@hortonworks.com> > >> > >> >>>> Cc: user@hadoop.apache.org, reduno1985 <reduno1...@gmail.com> > >> > >> >>>> > >> > >> >>>> > >> > >> >>>> This the output that I get I am running two machines as you > can > >> > >> >>>> see > >> > >> >>>> do > >> > >> >>>> u see anything suspicious ? > >> > >> >>>> Configured Capacity: 21145698304 (19.69 GB) > >> > >> >>>> Present Capacity: 17615499264 (16.41 GB) > >> > >> >>>> DFS Remaining: 17615441920 (16.41 GB) > >> > >> >>>> DFS Used: 57344 (56 KB) > >> > >> >>>> DFS Used%: 0% > >> > >> >>>> Under replicated blocks: 0 > >> > >> >>>> Blocks with corrupt replicas: 0 > >> > >> >>>> Missing blocks: 0 > >> > >> >>>> > >> > >> >>>> ------------------------------------------------- > >> > >> >>>> Datanodes available: 2 (2 total, 0 dead) > >> > >> >>>> > >> > >> >>>> Name: 11.1.0.6:50010 > >> > >> >>>> Decommission Status : Normal > >> > >> >>>> Configured Capacity: 10572849152 (9.85 GB) > >> > >> >>>> DFS Used: 28672 (28 KB) > >> > >> >>>> Non DFS Used: 1765019648 (1.64 GB) > >> > >> >>>> DFS Remaining: 8807800832(8.2 GB) > >> > >> >>>> DFS Used%: 0% > >> > >> >>>> DFS Remaining%: 83.31% > >> > >> >>>> Last contact: Sat Mar 23 11:30:10 CET 2013 > >> > >> >>>> > >> > >> >>>> > >> > >> >>>> Name: 11.1.0.3:50010 > >> > >> >>>> Decommission Status : Normal > >> > >> >>>> Configured Capacity: 10572849152 (9.85 GB) > >> > >> >>>> DFS Used: 28672 (28 KB) > >> > >> >>>> Non DFS Used: 1765179392 (1.64 GB) > >> > >> >>>> DFS Remaining: 8807641088(8.2 GB) > >> > >> >>>> DFS Used%: 0% > >> > >> >>>> DFS Remaining%: 83.3% > >> > >> >>>> Last contact: Sat Mar 23 11:30:08 CET 2013 > >> > >> >>>> > >> > >> >>>> > >> > >> >>>> On Fri, Mar 22, 2013 at 10:19 PM, Abdelrahman Shettia < > >> > >> >>>> ashet...@hortonworks.com> wrote: > >> > >> >>>> > >> > >> >>>>> Hi Redwane, > >> > >> >>>>> > >> > >> >>>>> Please run the following command as hdfs user on any > datanode. > >> > >> >>>>> The > >> > >> >>>>> output will be something like this. Hope this helps > >> > >> >>>>> > >> > >> >>>>> hadoop dfsadmin -report > >> > >> >>>>> Configured Capacity: 81075068925 (75.51 GB) > >> > >> >>>>> Present Capacity: 70375292928 (65.54 GB) > >> > >> >>>>> DFS Remaining: 69895163904 (65.09 GB) > >> > >> >>>>> DFS Used: 480129024 (457.89 MB) > >> > >> >>>>> DFS Used%: 0.68% > >> > >> >>>>> Under replicated blocks: 0 > >> > >> >>>>> Blocks with corrupt replicas: 0 > >> > >> >>>>> Missing blocks: 0 > >> > >> >>>>> > >> > >> >>>>> Thanks > >> > >> >>>>> -Abdelrahman > >> > >> >>>>> > >> > >> >>>>> > >> > >> >>>>> On Fri, Mar 22, 2013 at 12:35 PM, reduno1985 > >> > >> >>>>> <reduno1...@googlemail.com>wrote: > >> > >> >>>>> > >> > >> >>>>>> > >> > >> >>>>>> I have my hosts running on openstack virtual machine > instances > >> > >> >>>>>> each > >> > >> >>>>>> instance has 10gb hard disc . Is there a way too see how > much > >> > >> >>>>>> space > >> > >> >>>>>> is in > >> > >> >>>>>> the hdfs without web ui . > >> > >> >>>>>> > >> > >> >>>>>> > >> > >> >>>>>> Sent from Samsung Mobile > >> > >> >>>>>> > >> > >> >>>>>> Serge Blazhievsky <hadoop...@gmail.com> wrote: > >> > >> >>>>>> Check web ui how much space you have on hdfs??? > >> > >> >>>>>> > >> > >> >>>>>> Sent from my iPhone > >> > >> >>>>>> > >> > >> >>>>>> On Mar 22, 2013, at 11:41 AM, Abdelrahman Shettia < > >> > >> >>>>>> ashet...@hortonworks.com> wrote: > >> > >> >>>>>> > >> > >> >>>>>> Hi Redwane , > >> > >> >>>>>> > >> > >> >>>>>> It is possible that the hosts which are running tasks are do > >> > >> >>>>>> not > >> > >> >>>>>> have > >> > >> >>>>>> enough space. Those dirs are confiugred in mapred-site.xml > >> > >> >>>>>> > >> > >> >>>>>> > >> > >> >>>>>> > >> > >> >>>>>> On Fri, Mar 22, 2013 at 8:42 AM, Redwane belmaati cherkaoui > < > >> > >> >>>>>> reduno1...@googlemail.com> wrote: > >> > >> >>>>>> > >> > >> >>>>>>> > >> > >> >>>>>>> > >> > >> >>>>>>> ---------- Forwarded message ---------- > >> > >> >>>>>>> From: Redwane belmaati cherkaoui < > reduno1...@googlemail.com> > >> > >> >>>>>>> Date: Fri, Mar 22, 2013 at 4:39 PM > >> > >> >>>>>>> Subject: About running a simple wordcount mapreduce > >> > >> >>>>>>> To: mapreduce-iss...@hadoop.apache.org > >> > >> >>>>>>> > >> > >> >>>>>>> > >> > >> >>>>>>> Hi > >> > >> >>>>>>> I am trying to run a wordcount mapreduce job on several > >> > >> >>>>>>> files > >> > >> >>>>>>> (<20 > >> > >> >>>>>>> mb) using two machines . I get stuck on 0% map 0% reduce. > >> > >> >>>>>>> The jobtracker log file shows the following warning: > >> > >> >>>>>>> WARN org.apache.hadoop.mapred.JobInProgress: No room for > map > >> > >> >>>>>>> task. > >> > >> >>>>>>> Node hadoop0.novalocal has 8791384064 bytes free; but we > >> > >> >>>>>>> expect > >> > >> >>>>>>> map to > >> > >> >>take > >> > >> >>>>>>> 1317624576693539401 > >> > >> >>>>>>> > >> > >> >>>>>>> Please help me , > >> > >> >>>>>>> Best Regards, > >> > >> >>>>>>> > >> > >> >>>>>>> > >> > >> >>>>>> > >> > >> >>>>> > >> > >> >>>> > >> > >> >>>> > >> > >> >>> > >> > >> > > >> > >> > > >> > >> > Matteo Lanati > >> > >> > Distributed Resources Group > >> > >> > Leibniz-Rechenzentrum (LRZ) > >> > >> > Boltzmannstrasse 1 > >> > >> > 85748 Garching b. München (Germany) > >> > >> > Phone: +49 89 35831 8724 > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Harsh J > >> > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Harsh J > >> > > >> > >> Matteo Lanati > >> Distributed Resources Group > >> Leibniz-Rechenzentrum (LRZ) > >> Boltzmannstrasse 1 > >> 85748 Garching b. München (Germany) > >> Phone: +49 89 35831 8724 > >> > > > > > > -- > Harsh J >
