Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/2746#discussion_r18920780
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExecutorAllocationManager.scala
---
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
+
+/**
+ * An agent that dynamically allocates and removes executors based on the
workload.
+ *
+ * The add policy depends on the number of pending tasks. If the queue of
pending tasks is not
+ * drained in N seconds, then new executors are added. If the queue
persists for another M
+ * seconds, then more executors are added and so on. The number added in
each round increases
+ * exponentially from the previous round until an upper bound on the
number of executors has
+ * been reached.
+ *
+ * The rationale for the exponential increase is twofold: (1) Executors
should be added slowly
+ * in the beginning in case the number of extra executors needed turns out
to be small. Otherwise,
+ * we may add more executors than we need just to remove them later. (2)
Executors should be added
+ * quickly over time in case the maximum number of executors is very high.
Otherwise, it will take
+ * a long time to ramp up under heavy workloads.
+ *
+ * The remove policy is simpler: If an executor has been idle for K
seconds (meaning it has not
+ * been scheduled to run any tasks), then it is removed. This requires
starting a timer on each
+ * executor instead of just starting a global one as in the add case.
+ *
+ * Both add and remove attempts are retried on failure up to a maximum
number of times.
+ *
+ * The relevant Spark properties include the following:
+ *
+ * spark.dynamicAllocation.enabled - Whether this feature is enabled
+ * spark.dynamicAllocation.minExecutors - Lower bound on the number of
executors
+ * spark.dynamicAllocation.maxExecutors - Upper bound on the number of
executors
+ *
+ * spark.dynamicAllocation.addExecutorThreshold - How long before new
executors are added (N)
+ * spark.dynamicAllocation.addExecutorInterval - How often to add new
executors (M)
+ * spark.dynamicAllocation.removeExecutorThreshold - How long before an
executor is removed (K)
+ *
+ * spark.dynamicAllocation.addExecutorRetryInterval - How often to retry
adding executors
+ * spark.dynamicAllocation.removeExecutorRetryInterval - How often to
retry removing executors
+ * spark.dynamicAllocation.maxAddExecutorRetryAttempts - Max retries in
re-adding executors
+ * spark.dynamicAllocation.maxRemoveExecutorRetryAttempts - Max retries
in re-removing executors
+ *
+ * Synchronization: Because the schedulers in Spark are single-threaded,
contention should only
+ * arise when new executors register or when existing executors have been
removed, both of which
+ * are relatively rare events with respect to task scheduling. Thus,
synchronizing each method on
+ * the same lock should not be expensive assuming biased locking is
enabled in the JVM (on by
+ * default for Java 6+). This may not be true, however, if the application
itself runs multiple
+ * jobs concurrently.
+ *
+ * Note: This is part of a larger implementation (SPARK-3174) and
currently does not actually
+ * request to add or remove executors. The mechanism to actually do this
will be added separately,
+ * e.g. in SPARK-3822 for Yarn.
+ */
+private[scheduler] class ExecutorAllocationManager(scheduler:
TaskSchedulerImpl) extends Logging {
+ private val conf = scheduler.conf
+
+ // Lower and upper bounds on the number of executors. These are required.
+ private val minNumExecutors =
conf.getInt("spark.dynamicAllocation.minExecutors", -1)
+ private val maxNumExecutors =
conf.getInt("spark.dynamicAllocation.maxExecutors", -1)
+ if (minNumExecutors < 0 || maxNumExecutors < 0) {
+ throw new SparkException("spark.dynamicAllocation.{min/max}Executors
must be set!")
+ }
+
+ // How frequently to add and remove executors (seconds)
+ private val addThreshold =
+ conf.getLong("spark.dynamicAllocation.addExecutorThreshold", 60)
+ private val addInterval =
+ conf.getLong("spark.dynamicAllocation.addExecutorInterval",
addThreshold)
+ private val addRetryInterval =
+ conf.getLong("spark.dynamicAllocation.addExecutorRetryInterval",
addInterval)
+ private val removeThreshold =
+ conf.getLong("spark.dynamicAllocation.removeExecutorThreshold", 600)
+ private val removeRetryInterval =
+ conf.getLong("spark.dynamicAllocation.removeExecutorRetryInterval",
300)
+
+ // Number of executors to add in the next round
+ private var numExecutorsToAdd = 1
+
+ // Pending executors that have not actually been added/removed yet
--- End diff --
It would be helpful to expand on what "pending" means in the comment here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
