[
https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906047#comment-14906047
]
Jonathan Kelly commented on SPARK-10790:
----------------------------------------
I did search through all dynamicAllocation-related JIRAs targeted for 1.5.1+
before cutting this one, and I didn't find anything. Also, I don't see any diff
at all in core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
between v1.5.0 and the current master (d91967e), though maybe it's possible for
this issue to have been fixed elsewhere. I can try the latest master, but it
doesn't seem like it will have been fixed. Thanks for the response though.
> Dynamic Allocation does not request any executors if first stage needs less
> than or equal to spark.dynamicAllocation.initialExecutors
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-10790
> URL: https://issues.apache.org/jira/browse/SPARK-10790
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.5.0
> Reporter: Jonathan Kelly
> Priority: Critical
>
> If you set spark.dynamicAllocation.initialExecutors > 0 (or
> spark.dynamicAllocation.minExecutors, since
> spark.dynamicAllocation.initialExecutors defaults to
> spark.dynamicAllocation.minExecutors), and the number of tasks in the first
> stage of your job is less than or equal to this min/init number of executors,
> dynamic allocation won't actually request any executors and will just hang
> indefinitely with the warning "Initial job has not accepted any resources;
> check your cluster UI to ensure that workers are registered and have
> sufficient resources".
> The cause appears to be that ExecutorAllocationManager does not request any
> executors while the application is still initializing, but it still sets the
> initial value of numExecutorsTarget to
> spark.dynamicAllocation.initialExecutors. Once the job is running and has
> submitted its first task, if the first task does not need more than
> spark.dynamicAllocation.initialExecutors,
> ExecutorAllocationManager.updateAndSyncNumExecutorsTarget() does not think
> that it needs to request any executors, so it doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
