[
https://issues.apache.org/jira/browse/HADOOP-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615865#action_12615865
]
Christian Kunz commented on HADOOP-3813:
----------------------------------------
Arun, this was first time with a large job.
We just had another job completed with a similar number of maps, but only a
single reducer, and the JobTracker exhibited this problem just for a couple of
seconds (57 discarded heartbeats vs 130,000 discarded rpc calls in the previous
job). But from this point of view, yes it is reproducible.
In my first comment I probably should not have said 'Promotion', but 'Removal
of completed tasks', because the first job had a lot of failed and
speculatively executed tasks with a lot of temporary output in dfs, making the
cleanup operation more intense.
> RPC queue overload of JobTracker
> --------------------------------
>
> Key: HADOOP-3813
> URL: https://issues.apache.org/jira/browse/HADOOP-3813
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.1
> Reporter: Christian Kunz
>
> On a cluster with about 1700 nodes, when a job with about 100,000 maps and
> 10,000 reduces completed, the JobTracker, even with 80 handlers, could not
> handle the rpc call load during promotion of the job, such that at the end,
> because of the discarded heartbeats, the JobTracker lost nearly all
> TaskTrackers (about 10 TaskTrackers left). Promotion took more than 40
> minutes.
> They reconnected and everything recovered, but this might have been just luck.
> Shouldn't there be an adaptive throttling of the rate in heartbeats and
> TaskCompletionEvents?
> Sample messsages:
> 2008-07-22 18:21:55,831 WARN org.apache.hadoop.ipc.Server: Call queue
> overflow discarding oldest call heartbeat([EMAIL PROTECTED], false, true,
> 18137) from xxx
> 2008-07-22 18:21:55,834WARN org.apache.hadoop.ipc.Server: Call queue overflow
> discarding oldest call getTaskCompletionEvents(job_200807190635_0012, 119567,
> 50) from yyy
> ...
> 2008-07-22 19:02:28,821 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 9020, call heartbeat([EMAIL PROTECTED], false, true, 18199) from zzz:
> discarded for being too old (40936)
> 2008-07-22 19:02:28,821 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 34 on 9020, call getTaskCompletionEvents(job_200807190635_0012, 119567, 50)
> from uuu: discarded for being too old (40978)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
