Re: Shuffle In Memory OutOfMemoryError

Tue, 09 Mar 2010 16:21:14 -0800 

No, MR-1182 is included in 0.20.2

What heap size have you set for your reduce tasks? -C

Sent from my iPhone

On Mar 9, 2010, at 2:34 PM, "Ted Yu" <yuzhih...@gmail.com> wrote:


Andy:
You need to manually apply the patch.

Cheers
On Tue, Mar 9, 2010 at 2:23 PM, Andy Sautins <andy.saut...@returnpath.net >wrote:
Thanks Ted. My understanding is that MAPREDUCE-1182 is included in the 0.20.2 release. We upgraded our cluster to 0.20.2 this weekend and re-ran the same job scenarios. Running with mapred.reduce.parallel.copies set to 1 
and continue to have the same Java heap space error.



-----Original Message-----
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Tuesday, March 09, 2010 12:56 PM
To: common-user@hadoop.apache.org
Subject: Re: Shuffle In Memory OutOfMemoryError

This issue has been resolved in
http://issues.apache.org/jira/browse/MAPREDUCE-1182

Please apply the patch
M1182-1v20.patch<
http://issues.apache.org/jira/secure/attachment/12424116/M1182-1v20.patch > 

On Sun, Mar 7, 2010 at 3:57 PM, Andy Sautins <andy.saut...@returnpath.net

wrote:




Thanks Ted.  Very helpful.  You are correct that I misunderstood the

code
at ReduceTask.java:1535. I missed the fact that it's in a IOException 
catch

block.  My mistake.  That's what I get for being in a rush.

For what it's worth I did re-run the job with
mapred.reduce.parallel.copies set with values from 5 all the way down to 
1.

All failed with the same error:

Error: java.lang.OutOfMemoryError: Java heap space
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1408) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1261) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run (ReduceTask.java:1195)
So from that it does seem like something else might be going on, yes? 
I

need to do some more research.

I appreciate your insights.

Andy

-----Original Message-----
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Sunday, March 07, 2010 3:38 PM
To: common-user@hadoop.apache.org
Subject: Re: Shuffle In Memory OutOfMemoryError

My observation is based on this call chain:
MapOutputCopier.run() calling copyOutput() calling getMapOutput() calling 
ramManager.canFitInMemory(decompressedLength)
Basically ramManager.canFitInMemory() makes decision without considering 
the
number of MapOutputCopiers that are running. Thus 1.25 * 0.7 of total 
heap

may be used in shuffling if default parameters were used.
Of course, you should check the value for mapred.reduce.parallel.copies 
to

see if it is 5. If it is 4 or lower, my reasoning wouldn't apply.
About ramManager.unreserve() call, ReduceTask.java from hadoop 0.20.2 
only
has 2731 lines. So I have to guess the location of the code snippet you 
provided.
I found this around line 1535:
      } catch (IOException ioe) {
        LOG.info("Failed to shuffle from " +
mapOutputLoc.getTaskAttemptId(),
                 ioe);

        // Inform the ram-manager
        ramManager.closeInMemoryFile(mapOutputLength);
        ramManager.unreserve(mapOutputLength);

        // Discard the map-output
        try {
          mapOutput.discard();
        } catch (IOException ignored) {
          LOG.info("Failed to discard map-output from " +
                   mapOutputLoc.getTaskAttemptId(), ignored);
        }
Please confirm the line number.
If we're looking at the same code, I am afraid I don't see how we can 
improve it. First, I assume IOException shouldn't happen that often.
Second,
mapOutput.discard() just sets:
        data = null;
for in memory case. Even if we call mapOutput.discard() before
ramManager.unreserve(), we don't know when GC would kick in and make more 
memory available.
Of course, given the large number of map outputs in your system, it

became

more likely that the root cause from my reasoning made OOME happen

sooner.


Thanks




On Sun, Mar 7, 2010 at 1:03 PM, Andy Sautins <

andy.saut...@returnpath.net

wrote:




 Ted,

 I'm trying to follow the logic in your mail and I'm not sure I'm
following. If you would mind helping me understand I would appreciate 
it.
Looking at the code maxSingleShuffleLimit is only used in determining 
if

the copy _can_ fit into memory:

   boolean canFitInMemory(long requestedSize) {
      return (requestedSize < Integer.MAX_VALUE &&
              requestedSize < maxSingleShuffleLimit);
    }
It also looks like the RamManager.reserve should wait until memory 
is

available so it should hit a memory limit for that reason.

  What does seem a little strange to me is the following (

ReduceTask.java

starting at 2730 ):

        // Inform the ram-manager
        ramManager.closeInMemoryFile(mapOutputLength);
        ramManager.unreserve(mapOutputLength);

        // Discard the map-output
        try {
          mapOutput.discard();
        } catch (IOException ignored) {
          LOG.info("Failed to discard map-output from " +
                   mapOutputLoc.getTaskAttemptId(), ignored);
        }
        mapOutput = null;
So to me that looks like the ramManager unreserves the memory before 
the
mapOutput is discarded. Shouldn't the mapOutput be discarded _before_ 
the
ramManager unreserves the memory? If the memory is unreserved before 
the

actual underlying data references are removed then it seems like

another
thread can try to allocate memory ( ReduceTask.java:2730 ) before the 
previous memory is disposed ( mapOutput.discard() ).
Not sure that makes sense. One thing to note is that the particular 
job
that is failing does have a good number ( 200k+ ) of map outputs. The 
large

number of small map outputs may be why we are triggering a problem.

 Thanks again for your thoughts.

 Andy


-----Original Message-----
From: Jacob R Rideout [mailto:apa...@jacobrideout.net]
Sent: Sunday, March 07, 2010 1:21 PM
To: common-user@hadoop.apache.org
Cc: Andy Sautins; Ted Yu
Subject: Re: Shuffle In Memory OutOfMemoryError

Ted,
Thank you. I filled MAPREDUCE-1571 to cover this issue. I might have 
some time to write a patch later this week.

Jacob Rideout
On Sat, Mar 6, 2010 at 11:37 PM, Ted Yu <yuzhih...@gmail.com> wrote: 
I think there is mismatch (in ReduceTask.java) between:
    this.numCopiers = conf.getInt("mapred.reduce.parallel.copies",

5);

and:
      maxSingleShuffleLimit = (long)(maxSize *
MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION);
where MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION is 0.25f

because
    copiers = new ArrayList<MapOutputCopier>(numCopiers);
so the total memory allocated for in-mem shuffle is 1.25 * maxSize

A JIRA should be filed to correlate the constant 5 above and
MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION.

Cheers

On Sat, Mar 6, 2010 at 8:31 AM, Jacob R Rideout <

apa...@jacobrideout.net

wrote:


Hi all,

We are seeing the following error in our reducers of a particular

job:


Error: java.lang.OutOfMemoryError: Java heap space
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1408) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1261) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run (ReduceTask.java:1195)
After enough reducers fail the entire job fails. This error occurs 
regardless of whether mapred.compress.map.output is true. We were

able

to avoid the issue by reducing

mapred.job.shuffle.input.buffer.percent
to 20%. Shouldn't the framework via ShuffleRamManager.canFitInMemory 
and.ShuffleRamManager.reserve correctly detect the the memory
available for allocation? I would think that with poor configuration settings (and default settings in particular) the job may not be as 
efficient, but wouldn't die.

Here is some more context in the logs, I have attached the full
reducer log here: http://gist.github.com/323746


2010-03-06 07:54:49,621 INFO org.apache.hadoop.mapred.ReduceTask:
Shuffling 4191933 bytes (435311 raw bytes) into RAM from
attempt_201003060739_0002_m_000061_0
2010-03-06 07:54:50,222 INFO org.apache.hadoop.mapred.ReduceTask:

Task

attempt_201003060739_0002_r_000000_0: Failed fetch #1 from
attempt_201003060739_0002_m_000202_0
2010-03-06 07:54:50,223 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201003060739_0002_r_000000_0 adding host
hd37.dfs.returnpath.net to penalty box, next contact in 4 seconds
2010-03-06 07:54:50,223 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201003060739_0002_r_000000_0: Got 1 map-outputs from

previous

failures
2010-03-06 07:54:50,223 FATAL org.apache.hadoop.mapred.TaskRunner: 
attempt_201003060739_0002_r_000000_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.getMapOutput(ReduceTask.java:1408) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier $MapOutputCopier.copyOutput(ReduceTask.java:1261) 
      at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run (ReduceTask.java:1195)
We tried this both in 0.20.1 and 0.20.2. We had hoped MAPREDUCE-1182 would address the issue in 0.20.2, but it did not. Does anyone have any comments or suggestions? Is this a bug I should file a JIRA for? 

Jacob Rideout
Return Path

Reply via email to