[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371975#comment-15371975 ]
Saisai Shao commented on SPARK-16435: ------------------------------------- OK, I will file a small patch to add the warning log about this invalid configuration. > Behavior changes if initialExecutor is less than minExecutor for dynamic > allocation > ----------------------------------------------------------------------------------- > > Key: SPARK-16435 > URL: https://issues.apache.org/jira/browse/SPARK-16435 > Project: Spark > Issue Type: Bug > Components: Scheduler, Spark Core > Affects Versions: 2.0.0 > Reporter: Saisai Shao > Priority: Minor > > After SPARK-13723, the behavior changed for > {{spark.dynamicAllocation.initialExecutors}} less then > {{spark.dynamicAllocation.minExecutors}} situation. > initialExecutors < minExecutors is an invalid setting, > h4. Before SPARK-13723 > If initialExecutors < minExecutors, Spark will throw exception with: > {code} > java.lang.IllegalArgumentException: requirement failed: initial executor > number xxx must between min executor number xxx and max executor number xxx > {code} > This will clearly let user know that current configuration is invalid. > h4. After SPARK-13723 > Because we also consider {{spark.executor.instances}}, so the initial number > is the max value between minExecutors, initialExecutors, numExecutors. > This will silently ignore the situation where initialExecutors < minExecutors. > So at least we should add some warning logs to let user know this is an > invalid configuration. > What do you think [~tgraves], [~rdblue]? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org