Tom White wrote:
Hi David,
The MapReduce framework will attempt to rerun failed tasks
automatically. However, if a task is running out of memory on one
machine, it's likely to run out of memory on another, isn't it? Have a
look at the mapred.child.java.opts configuration property for the
amount of memory that each task VM is given (200MB by default). You
can also control the memory that each daemon gets using the
HADOOP_HEAPSIZE variable in hadoop-env.sh. Or you can specify it on a
per-daemon basis using the HADOOP_<DAEMON_NAME>_OPTS variables in the
same file.
Tom
This looks not so much a VM out of memory problem as OS thread
provisioning. ulimit may be useful, as is the java -Xss option
http://candrews.integralblue.com/2009/01/preventing-outofmemoryerror-native-thread/
On Wed, May 6, 2009 at 1:28 AM, David Batista <dsbati...@gmail.com> wrote:
I get this error when running Reduce tasks on a machine:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:2591)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:454)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:190)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:387)
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:117)
at
org.apache.hadoop.mapred.lib.MultipleTextOutputFormat.getBaseRecordWriter(MultipleTextOutputFormat.java:44)
at
org.apache.hadoop.mapred.lib.MultipleOutputFormat$1.write(MultipleOutputFormat.java:99)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
is it possible to move a reduce task to other machine in the cluster on the fly?
--
./david
