[
https://issues.apache.org/jira/browse/HADOOP-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated HADOOP-1984:
-------------------------------
Status: Patch Available (was: Open)
Taking into consideration Arun's comment, hard coding the computations for
base=2. The way it works now is as follows
1. the first notification is sent to the jobtracker after {{max-backoff}} time.
The backoff values are ({{4, 8, 16, 32 ..... k}}) such that {{4 + 8 + 16 + 32
.... k ~ max-backoff}}, {{max-backoff}} can be set using
{{mapred.reduce.copy.backoff}} [default is 300 sec].
2. subsequent notifications are also sent after {{max-backoff}} time but the
backoff values now are ({{max-backoff/2, max-backoff/2}}) , i.e 2 retries
3. at the max the reducer waits for {{3 * max-backoff}} before the maps get
re-executed.
Comments ?
> some reducer stuck at copy phase and progress extremely slowly
> --------------------------------------------------------------
>
> Key: HADOOP-1984
> URL: https://issues.apache.org/jira/browse/HADOOP-1984
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Runping Qi
> Assignee: Amar Kamat
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: HADOOP-1984-simple.patch, HADOOP-1984-simple.patch,
> HADOOP-1984-simple.patch, HADOOP-1984.patch
>
>
> In many cases, some reducers got stuck at copy phase, progressing extremely
> slowly.
> The entire cluster seems doing nothing. This causes a very bad long tails of
> otherwise well tuned map/red jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
