[
https://issues.apache.org/jira/browse/SPARK-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617712#comment-14617712
]
Sean Owen commented on SPARK-8881:
----------------------------------
I think this needs better explanation. So you are asking for 8 cores per
executor and all workers have 7 cores available, and the result is that no
executors are allocated, and the app is still waiting for executors. That seems
like correct behavior, right?
Cores aren't really allocated one at a time; in "spreadOut" mode the target
allocation amount is spread around, but executors (only) launch with the # of
cores desired. Grabbing 8 cores at that phase in each pass wouldn't help, since
none have 8 cores available.
What does it have to do with the number of workers?
> Scheduling fails if num_executors < num_workers
> -----------------------------------------------
>
> Key: SPARK-8881
> URL: https://issues.apache.org/jira/browse/SPARK-8881
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.4.0, 1.5.0
> Reporter: Nishkam Ravi
>
> Current scheduling algorithm (in Master.scala) has two issues:
> 1. cores are allocated one at a time instead of spark.executor.cores at a time
> 2. when spark.cores.max/spark.executor.cores < num_workers, executors are not
> launched and the app hangs (due to 1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
