[
https://issues.apache.org/jira/browse/HADOOP-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671094#action_12671094
]
Hemanth Yamijala commented on HADOOP-5186:
------------------------------------------
Another point to consider (though I don't know if it belongs to the same JIRA)
is that currently the scheduler initializes all jobs submitted to the cluster
immediately. Initialized jobs add to the memory footprint on the jobtracker.
This could impact JT scale and performance with respect to number of jobs, on
very busy clusters.
If limits are set, maybe we can initialize only as many plus a few more of the
jobs in the queue so that the memory footprint is kept low. This may help the
JT to scale better.
> Improve limit handling in fairshare scheduler
> ---------------------------------------------
>
> Key: HADOOP-5186
> URL: https://issues.apache.org/jira/browse/HADOOP-5186
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Hemanth Yamijala
> Priority: Minor
>
> The fairshare scheduler has a way by which it can limit the number of jobs in
> a pool by setting the maxRunningJobs parameter in its allocations definition.
> This limit is treated as a hard limit, and comes into effect even if the
> cluster is free to run more jobs, resulting in underutilization. Possibly the
> same thing happens with the parameter maxRunningJobs for user and
> userMaxJobsDefault. It may help to treat these as a soft limit and run
> additional jobs to keep the cluster fully utilized.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
