Besides this, we're also getting this error: "java.lang.OutOfMemoryError: GC overhead limit exceeded"
For the above error, try increasing the heap size. It worked for me when i came across the same error in 0.17.0 version. "We're only running a total of 20 reducers which I suspect is very low, since there are a few thousand map tasks. Could that be the cause or is there anything else we're doing that's obviously wrong?" I doubt if less number of reduce tasks is an issue. I worked with around 6000 map tasks and 18 reduce tasks for a job and it had no problems. I'm not sure of this issue when maps are increased to tens of thousands, though. V.V.Chaitanya Krishna IIIT,Hyderabad India On Sat, Oct 18, 2008 at 6:16 AM, Per Jacobsson <[EMAIL PROTECTED]> wrote: > Hi all, > We've been running a pretty big job on 20 extra-large high-CPU EC2 servers > (Hadoop version 0.18, Java 1.6, the standard AMIs), and started getting the > dreaded "Could not find any valid local directory" error during the final > reduce phase. > > I've confirmed that some of the boxes are running out of space, but the > disk > usage seems to be very uneven across the servers. The datanodes report 50% > of available space used on all servers, which matches what I'm seeing in > the > /mnt/hadoop/dfs/data/folder (an even ~200 Gb / server). But the space used > by files in /mnt/hadoop/mapred/local differs a lot from server to server > (going from 70 Gb to 190 Gb). > > Is there any way to predict how much space will be used by the temporary > data stored outside of HDFS? We're only running a total of 20 reducers > which > I suspect is very low, since there are a few thousand map tasks. Could that > be the cause or is there anything else we're doing that's obviously wrong? > > Besides this, we're also getting this error: "java.lang.OutOfMemoryError: > GC > overhead limit exceeded" > > Thanks for any help, > / Per >
