This happens because child processes try to allocate the same memory as the parent. One way to solve this is setting memory overcommit to your linux system.
On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens <bradfordsteph...@gmail.com> wrote: > Hey guys, > > I'm running into issues when doing a moderate-size EMR job on 12 m1.large > nodes. Mappers and Reducers will randomly fail. > > The EMR defaults are 2 mappers / 2 reducers per node. I've tried running > with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No > difference. > > There are about 1,000 map tasks. Not very much data, maybe 50G at most? > > My job fails to complete. Looking in syslog shows this: > > java.io.IOException: Cannot run program "bash": java.io.IOException: > error=12, Cannot allocate memory > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) > at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) > at org.apache.hadoop.util.Shell.run(Shell.java:134) > at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) > at > org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > Caused by: java.io.IOException: java.io.IOException: error=12, Cannot > allocate memory > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) > at java.lang.ProcessImpl.start(ProcessImpl.java:65) > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) > ... 11 more > > > > I would ask the EMR forums, but I think I may get faster feedback here :) > > > -- > Bradford Stephens, > Founder, Drawn to Scale > drawntoscalehq.com > 727.697.7528 > > http://www.drawntoscalehq.com -- The intuitive, cloud-scale data solution. > Process, store, query, search, and serve all your data. > > http://www.roadtofailure.com -- The Fringes of Scalability, Social Media, > and Computer Science >
