[
https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668103#action_12668103
]
dhruba borthakur commented on HADOOP-5083:
------------------------------------------
We are 0.17 (and 0.19) with long-running JTs and one problem we are seeing is
that the JT sometimes runs out of its 3GB heap space. The system is used by
around 50-80 users. The max jobs per user before retirement is 5. But this
still means that the JT keeps info about 80*5=400 completed jobs in memory.
Sometimes these jobs have huge number of tasks. This eats up most of the memory
in the JT.
Can part of this fix (i.e. purge all jobs from completed queue when there is
memory usage on the JT exceeds a configured threshold) be ported to 0.19. It is
very useful to have it in 0.19, otherwise the JT just hangs up and the cluster
has to be restarted.
> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>
> Key: HADOOP-5083
> URL: https://issues.apache.org/jira/browse/HADOOP-5083
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Arun C Murthy
> Assignee: Amar Kamat
> Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
>
>
> Currently the JobTracker serves the JobHistory to end-users off files
> local-disk/hdfs. While running very large clusters with a large user-base
> might result in lots of traffic for job-history which needlessly taxes the
> JobTracker. The proposal is to have an optional daemon which handles serving
> of job-history requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
