Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/4051#issuecomment-71406518
> @sryza Is the point of not requiring these configs that the users don't
really know how many executors they actually want?
Exactly. From my perspective, one of the main motivations for dynamic
allocation in the first place was removing the need to set `--num-executors`.
People have long complained about the long list configurations users need to
know for MapReduce, but at this point Spark actually requires more to get
decent resource utilization. This particular JIRA was filed Hive didn't know
how to set the max executors in a general way. What specific advice would you
give to a user about how to set this property?
> Also defaulting the max to Int.MaxValue doesn't make much sense in my
opinion because the whole point of doing this is to share resources in the
cluster.
This is the job of the cluster resource manager. Users and apps shouldn't
need self-regulate their usage. YARN will throttle resources when there is
contention. The main responsibility of the app is just to not ask for / sit on
more resources than it can take advantage of, which dynamic allocation helps
with in general, and which SPARK-4136 will help enforce more strictly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
