[
https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668369#action_12668369
]
Amar Kamat commented on HADOOP-5083:
------------------------------------
bq. We are 0.17 (and 0.19) with long-running JTs and one problem we are seeing
is that the JT sometimes runs out of its 3GB heap space. The system is used by
around 50-80 users. The max jobs per user before retirement is 5. But this
still means that the JT keeps info about 80*5=400 completed jobs in memory.
Sometimes these jobs have huge number of tasks. This eats up most of the memory
in the JT.
Wouldnt making the number of completed jobs in memory = 0 help? But the problem
is that the completed jobs will be available only via history and that will
risk the jobtracker.
bq. I was wondering if ay type of fix for this problem is going to be checked
into the 019 branch?
I think it will be big change to go in 0.19, no? Comments?
> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>
> Key: HADOOP-5083
> URL: https://issues.apache.org/jira/browse/HADOOP-5083
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Arun C Murthy
> Assignee: Amar Kamat
> Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
>
>
> Currently the JobTracker serves the JobHistory to end-users off files
> local-disk/hdfs. While running very large clusters with a large user-base
> might result in lots of traffic for job-history which needlessly taxes the
> JobTracker. The proposal is to have an optional daemon which handles serving
> of job-history requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
