Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4051#issuecomment-70606552 when we set initial number to min, there is a delay to request more executors. because in ExecutorAllocationManager's addExecutors numExecutorsToAdd starts from 1, and next minute is double. so when we request more executors, there is very slow and it spends several minutes.so maybe we need to change this scaling mechanism? another situation is: when stage has skew task and individual tasks need to run for a long time, number of executor will be to a small value, and then next stage need many executors to run tasks, so ExecutorAllocationManager need to request many executors. at this time it also need to spend a few minutes for reaching number of pending tasks. @andrewor14
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org