[
https://issues.apache.org/jira/browse/SPARK-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035462#comment-16035462
]
Xuefu Zhang commented on SPARK-20662:
-------------------------------------
[~lyc] I'm talking about mapreduce.job.max.map, which is the maximum number of
map tasks that a MR job may have. If a submitted MR job contains more map tasks
than that, it will be rejected. Similar to mapreduce.job.max.reduce.
[~sowen], [~vanzin], I don't think blocking a large (perhaps ridiculously) job
is equivalent to letting it run slowly and for ever. The use case I have is:
while yarn queue can be used to limit how much resources can be used, but a
queue can be shared by a team or multiple applications. It's probably not a
good idea to let one job takes all resources while starving others. Secondly,
many those users who submit ridiculously large job have no idea on what they
are doing and they don't even realize that their jobs are huge. Lastly and more
importantly, our application environment has a global timeout, beyond which a
job will be killed. If a large job gets killed this way, significant resources
is wasted. Thus, blocking such a job at submission time helps preserve the
resources.
BTW, if the scenarios don't apply to a user, there is nothing for him/her to
worry about because the default should keep them happy.
In addition to spark.job.max.tasks, I'd also propose spark.stage.max.tasks,
which limits the number of tasks any stage of a job may contain. The rationale
behind this is that spark.job.max.tasks tends to favor jobs with small number
of stages. With both, we can not only cover MR's mapreduce.job.max.map and
mapreduce.job.max.reduce, but also control the overall size of a job.
> Block jobs that have greater than a configured number of tasks
> --------------------------------------------------------------
>
> Key: SPARK-20662
> URL: https://issues.apache.org/jira/browse/SPARK-20662
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.6.0, 2.0.0
> Reporter: Xuefu Zhang
>
> In a shared cluster, it's desirable for an admin to block large Spark jobs.
> While there might not be a single metrics defining the size of a job, the
> number of tasks is usually a good indicator. Thus, it would be useful for
> Spark scheduler to block a job whose number of tasks reaches a configured
> limit. By default, the limit could be just infinite, to retain the existing
> behavior.
> MapReduce has mapreduce.job.max.map and mapreduce.job.max.reduce to be
> configured, which blocks a MR job at job submission time.
> The proposed configuration is spark.job.max.tasks with a default value -1
> (infinite).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
