Re: Very strange Java Collection behavior in Hadoop
Hi Jay Just follow this to install http://jugnu-life.blogspot.in/2012/03/hadoop-installation-tutorial.html The official tutorial at link below is also useful http://hadoop.apache.org/common/docs/r1.0.1/single_node_setup.html Thanks Jagat On Fri, Mar 23, 2012 at 12:08 PM, Jay Vyas wrote: > Hi guys : What the latest, simplest, best directions to get a tiny, > psuedodistributed hadoop setup running on my ubuntu machine ? > > On Wed, Mar 21, 2012 at 5:14 PM, wrote: > > > Owen, > > > > Is there interest in reverting hadoop-2399 in 0.23.x ? > > > > - Milind > > > > --- > > Milind Bhandarkar > > Greenplum Labs, EMC > > (Disclaimer: Opinions expressed in this email are those of the author, > and > > do not necessarily represent the views of any organization, past or > > present, the author might be affiliated with.) > > > > > > > > On 3/19/12 11:20 PM, "Owen O'Malley" wrote: > > > > >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak > > >wrote: > > > > > >> Hi Owen O'Malley, > > >> Thank you for that Instant reply. It's working now. Can you explain > me > > >> what you mean by "input to reducer is reused" in little detail? > > > > > > > > >Each time the statement "Text value = values.next();" is executed it > > >always > > >returns the same Text object with the contents of that object changed. > > >When > > >you add the Text to the list, you are adding a pointer to the same Text > > >object. At the end you have 6 copies of the same pointer instead of 6 > > >different Text objects. > > > > > >The reason that I said it is my fault, is because I added the > optimization > > >that causes it. If you are interested in Hadoop archeology, it was > > >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the > > >documentation in the area. > > > > > >-- Owen > > > > > > > -- > Jay Vyas > MMSB/UCHC >
Re: Very strange Java Collection behavior in Hadoop
Hi guys : What the latest, simplest, best directions to get a tiny, psuedodistributed hadoop setup running on my ubuntu machine ? On Wed, Mar 21, 2012 at 5:14 PM, wrote: > Owen, > > Is there interest in reverting hadoop-2399 in 0.23.x ? > > - Milind > > --- > Milind Bhandarkar > Greenplum Labs, EMC > (Disclaimer: Opinions expressed in this email are those of the author, and > do not necessarily represent the views of any organization, past or > present, the author might be affiliated with.) > > > > On 3/19/12 11:20 PM, "Owen O'Malley" wrote: > > >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak > >wrote: > > > >> Hi Owen O'Malley, > >> Thank you for that Instant reply. It's working now. Can you explain me > >> what you mean by "input to reducer is reused" in little detail? > > > > > >Each time the statement "Text value = values.next();" is executed it > >always > >returns the same Text object with the contents of that object changed. > >When > >you add the Text to the list, you are adding a pointer to the same Text > >object. At the end you have 6 copies of the same pointer instead of 6 > >different Text objects. > > > >The reason that I said it is my fault, is because I added the optimization > >that causes it. If you are interested in Hadoop archeology, it was > >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the > >documentation in the area. > > > >-- Owen > > -- Jay Vyas MMSB/UCHC
Re: Very strange Java Collection behavior in Hadoop
Owen, Is there interest in reverting hadoop-2399 in 0.23.x ? - Milind --- Milind Bhandarkar Greenplum Labs, EMC (Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.) On 3/19/12 11:20 PM, "Owen O'Malley" wrote: >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak >wrote: > >> Hi Owen O'Malley, >> Thank you for that Instant reply. It's working now. Can you explain me >> what you mean by "input to reducer is reused" in little detail? > > >Each time the statement "Text value = values.next();" is executed it >always >returns the same Text object with the contents of that object changed. >When >you add the Text to the list, you are adding a pointer to the same Text >object. At the end you have 6 copies of the same pointer instead of 6 >different Text objects. > >The reason that I said it is my fault, is because I added the optimization >that causes it. If you are interested in Hadoop archeology, it was >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the >documentation in the area. > >-- Owen
Re: Very strange Java Collection behavior in Hadoop
Thanks a lot :) On Tue, Mar 20, 2012 at 11:50 AM, Owen O'Malley wrote: > On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak > wrote: > > > Hi Owen O'Malley, > > Thank you for that Instant reply. It's working now. Can you explain me > > what you mean by "input to reducer is reused" in little detail? > > > Each time the statement "Text value = values.next();" is executed it always > returns the same Text object with the contents of that object changed. When > you add the Text to the list, you are adding a pointer to the same Text > object. At the end you have 6 copies of the same pointer instead of 6 > different Text objects. > > The reason that I said it is my fault, is because I added the optimization > that causes it. If you are interested in Hadoop archeology, it was > HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the > documentation in the area. > > -- Owen > -- https://github.com/zinnia-phatak-dev/Nectar
Re: Very strange Java Collection behavior in Hadoop
On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak wrote: > Hi Owen O'Malley, > Thank you for that Instant reply. It's working now. Can you explain me > what you mean by "input to reducer is reused" in little detail? Each time the statement "Text value = values.next();" is executed it always returns the same Text object with the contents of that object changed. When you add the Text to the list, you are adding a pointer to the same Text object. At the end you have 6 copies of the same pointer instead of 6 different Text objects. The reason that I said it is my fault, is because I added the optimization that causes it. If you are interested in Hadoop archeology, it was HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the documentation in the area. -- Owen
Re: Very strange Java Collection behavior in Hadoop
Hi Owen O'Malley, Thank you for that Instant reply. It's working now. Can you explain me what you mean by "input to reducer is reused" in little detail? On Tue, Mar 20, 2012 at 11:28 AM, Owen O'Malley wrote: > On Mon, Mar 19, 2012 at 10:52 PM, madhu phatak > wrote: > > > Hi All, > > I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java > > Collection's . I have following code in reducer > > > That is my fault. *sigh* The input to the reducer is reused. Replace: > > list.add(value); > > with: > > list.add(new Text(value)); > > and the problem will go away. > > -- Owen > -- https://github.com/zinnia-phatak-dev/Nectar
Re: Very strange Java Collection behavior in Hadoop
On Mon, Mar 19, 2012 at 10:52 PM, madhu phatak wrote: > Hi All, > I am using Hadoop 0.20.2 . I am observing a Strange behavior of Java > Collection's . I have following code in reducer That is my fault. *sigh* The input to the reducer is reused. Replace: list.add(value); with: list.add(new Text(value)); and the problem will go away. -- Owen
