[ https://issues.apache.org/jira/browse/SPARK-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557911#comment-14557911 ]
Sandy Ryza commented on SPARK-7699: ----------------------------------- [~sowen] I think the possible flaw in your argument is that it relies on "initial load" being defined in some reasonable. I.e. I think the worry is that the following can happen: * initial = 3 and min = 1 * cluster is large and uncontended * first line of user code is a job submission that can make use of at least 3 * because the executor allocation thread starts immediately, requested executors ramps down to 1 before the user code has a chance to submit the job Which is to say: what guarantees do we provide about initialExecutors other than that it's the number of executors requests we have before some opaque internal thing happens to adjust it down? One possible such guarantee we could provide is that we won't adjust down for some fixed number of seconds after the SparkContext starts. > Config "spark.dynamicAllocation.initialExecutors" has no effect > ---------------------------------------------------------------- > > Key: SPARK-7699 > URL: https://issues.apache.org/jira/browse/SPARK-7699 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: meiyoula > > spark.dynamicAllocation.minExecutors 2 > spark.dynamicAllocation.initialExecutors 3 > spark.dynamicAllocation.maxExecutors 4 > Just run the spark-shell with above configurations, the initial executor > number is 2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org