[
https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328466#comment-15328466
]
Kay Ousterhout commented on SPARK-15176:
----------------------------------------
I thought about this a little more and I think I'm in favor of maxShare instead
of maxRunningTasks. The reason is that maxRunningTasks seems brittle to the
underlying setup -- if someone configures a certain maximum number of tasks,
and then a few machines die, the maximum may no longer be reasonable (e.g., it
may become larger than the number of machines in the cluster). The other
benefit is symmetry with minShare, as Mark mentioned.
[~njw45] why did you chose maxRunningTasks, as opposed to maxShare? Are there
other reasons that maxRunningTasks makes more sense?
> Job Scheduling Within Application Suffers from Priority Inversion
> -----------------------------------------------------------------
>
> Key: SPARK-15176
> URL: https://issues.apache.org/jira/browse/SPARK-15176
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Nick White
>
> Say I have two pools, and N cores in my cluster:
> * I submit a job to one, which has M >> N tasks
> * N of the M tasks are scheduled
> * I submit a job to the second pool - but none of its tasks get scheduled
> until a task from the other pool finishes!
> This can lead to unbounded denial-of-service for the second pool - regardless
> of `minShare` or `weight` settings. Ideally Spark would support a pre-emption
> mechanism, or an upper bound on a pool's resource usage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
