I do not instantiate 3 GB of objects, that is for sure. The wordcount
example does not run anymore so I dont think this is something
releated to my code, beside the wordcount example many other users
report the same problem:
See:
http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask%24MapOutputBuffer.collect+order%3Adate-backward
Thanks for your help!
Stefan
On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote:
It might have something to do with your application itself. By any
chance
are you doing a lot of huge object allocation (directly or indirectly)
within the map method? Which version of hadoop are you on?
-----Original Message-----
From: Stefan Groschupf [mailto:[EMAIL PROTECTED]
Sent: Sunday, March 16, 2008 9:07 AM
To: core-user@hadoop.apache.org
Subject: [memory leak?] Re: MapReduce failure
Hi there,
we see the same situation and browsing the posts there are
quite a lot of people running into this OOM problem.
We run a own Mapper and our mapred.child.java.opts is
-Xmx3048m, I think that should be more then enough.
Also I changed io.sort.mb to 10, which had also no impact.
Any ideas what might cause the OutOfMemoryError ?
Thanks.
Stefan
On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
What is the heap size you are using for your tasks? Check
'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
it. This will happen if you try running the random-writer + sort
examples with default parameters. The maps are not able to
spill the
data to the disk. Btw what version of HADOOP are you using?
Amar
On Mon, 10 Mar 2008, Ved Prakash wrote:
Hi friends,
I have made a cluster of 3 machines, one of them is
master, and other
2 slaves. I executed a mapreduce job on master but after Map, the
execution terminates and Reduce doesn't happen. I have checked dfs
and no output folder gets created.
this is the error I see
08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
task_200803101001_0001_m_000064_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount
$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount
$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java
:1787)
08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
17% 08/03/10
10:35:31 INFO mapred.JobClient: map 56% reduce 17%
08/03/10 10:35:51
INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
mapred.JobClient: Task Id :
task_200803101001_0001_m_000071_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount
$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount
$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java
:1787)
though it tries to overcome this problem but the mapreduce
application doesn't create output, can anyone tell me why is this
happening?
Thanks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com
