A simple fix, aside of applying a possible combiner, would be to raise the number of reducers by a small margin such that per-partition data falls in size.
On Mon, Dec 31, 2012 at 9:44 PM, Majid Azimi <majid.merk...@gmail.com> wrote: > hi guys, > > Hadoop the definitive guide says: > intermediate results on the mapper side is written to local disk at > mapred.local.dir location so if this location does not have enough space the > map will fail. > > I want to know if this is true on the reducer side. Output of all mappers > will merge at reducer side. In which location this merge happens? If that > location does not have enough space does reducer fail? What is the solution > for MapReduce jobs if intermediat results for some keys is more than local > disk of reducer? -- Harsh J
