Re: Hadoop - non disk based sorting?

2011-12-02 Thread Chandraprakash Bhagtani
pier.shuffleInMemory(ReduceTask.java:1592) > > The problem has occured,after the inmemory copy was chosen, > > Regards, > Ravi Teja > > From: Robert Evans [ev...@yahoo-inc.com] > Sent: 01 December 2011 21:44:50 > To: common-dev@h

RE: Hadoop - non disk based sorting?

2011-12-01 Thread Ravi teja ch n v
21:44:50 To: common-dev@hadoop.apache.org Subject: Re: Hadoop - non disk based sorting? Mingxi, My understanding was that just like with the maps that when a reducer's in memory buffer fills up it too will spill to disk as part of the sort. In fact I think it uses the exact same code fo

Re: Hadoop - non disk based sorting?

2011-12-01 Thread Todd Lipcon
he Map or Reduce JVMs, it uses the > mapred.child.java.opts as JVM arguments for the new child JVMs. > > Regards, > Ravi Teja > ________________ > From: Mingxi Wu [mingxi...@turn.com] > Sent: 01 December 2011 12:37:54 > To: common-dev@hadoop.apache.org > Sub

Re: Hadoop - non disk based sorting?

2011-12-01 Thread Robert Evans
opts as JVM arguments for the new child JVMs. Regards, Ravi Teja From: Mingxi Wu [mingxi...@turn.com] Sent: 01 December 2011 12:37:54 To: common-dev@hadoop.apache.org Subject: RE: Hadoop - non disk based sorting? Thanks Ravi. So, why when map outputs are h

RE: Hadoop - non disk based sorting?

2011-12-01 Thread Ravi teja ch n v
the new child JVMs. Regards, Ravi Teja From: Mingxi Wu [mingxi...@turn.com] Sent: 01 December 2011 12:37:54 To: common-dev@hadoop.apache.org Subject: RE: Hadoop - non disk based sorting? Thanks Ravi. So, why when map outputs are huge, reducer will not abl

RE: Hadoop - non disk based sorting?

2011-12-01 Thread Ravi teja ch n v
the new child JVMs. Regards, Ravi Teja From: Mingxi Wu [mingxi...@turn.com] Sent: 01 December 2011 12:37:54 To: common-dev@hadoop.apache.org Subject: RE: Hadoop - non disk based sorting? Thanks Ravi. So, why when map outputs are huge, reducer will not abl

RE: Hadoop - non disk based sorting?

2011-11-30 Thread Mingxi Wu
i.com] Sent: Tuesday, November 29, 2011 9:46 PM To: common-dev@hadoop.apache.org Subject: RE: Hadoop - non disk based sorting? Hi Mingxi, >From your stacktrace, I understand that the OutOfMemoryError has actually >occured while copying the MapOutputs, not while sorting them. Since your Map

RE: Hadoop - non disk based sorting?

2011-11-29 Thread Ravi teja ch n v
Hi Mingxi, >From your stacktrace, I understand that the OutOfMemoryError has actually >occured while copying the MapOutputs, not while sorting them. Since your Mapoutputs are huge and your reducer does have enough heap memory, you got the problem. When you have made the reducers to 200, your Ma