What is the heap size (see mapred.child.java.opts in your job.xml)? By the
way what is the job configuration i.e what is the job type (sort etc), number of nodes
etc. Can you provide more details? Anyways try increasing it which you can
do it using
hadoop jar jar-file.jar program -Dmapred.child.java.opts=-Xmx512m <args>
Amar
On Mon, 25 Feb 2008, ma qiang wrote:
Hi all;
A error happen in my program and system print as below:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:349)
at MyTest.map(UserMinHashMap.java:43)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
and I think the error happen here: output.collect(new
Text(testKeyString),new Text(testValueString));
I'm not sure why? I guess the reason is the length of testValueString
is too large to construct Text or write into HD. Who can tell why?
Thanks very much
Best Wishes!
