Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/7532#discussion_r36038415
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -804,6 +827,87 @@ private[master] class Master(
}
/**
+ * Handle a request to set the target number of executors for this
application.
+ *
+ * If the executor limit is adjusted upwards, new executors will be
launched provided
+ * that there are workers with sufficient resources. If it is adjusted
downwards, however,
+ * we do not kill existing executors until we explicitly receive a kill
request.
+ *
+ * @return whether the application has previously registered with this
Master.
+ */
+ private def handleRequestExecutors(appId: String, requestedTotal: Int):
Boolean = {
+ idToApp.get(appId) match {
+ case Some(appInfo) =>
+ logInfo(s"Application $appId requested to set total executors to
$requestedTotal.")
+ appInfo.executorLimit = requestedTotal
+ schedule()
+ true
+ case None =>
+ logWarning(s"Unknown application $appId requested $requestedTotal
total executors.")
+ false
+ }
+ }
+
+ /**
+ * Handle a kill request from the given application.
+ *
+ * This method assumes the executor limit has already been adjusted
downwards through
+ * a separate [[RequestExecutors]] message, such that we do not launch
new executors
+ * immediately after the old ones are removed.
+ *
+ * @return whether the application has previously registered with this
Master.
+ */
+ private def handleKillExecutors(appId: String, executorIds: Seq[Int]):
Boolean = {
+ idToApp.get(appId) match {
+ case Some(appInfo) =>
+ logInfo(s"Application $appId requests to kill executors: " +
executorIds.mkString(", "))
+ val (known, unknown) =
executorIds.partition(appInfo.executors.contains)
+ known.foreach { executorId =>
+ val desc = appInfo.executors(executorId)
+ appInfo.removeExecutor(desc)
+ killExecutor(desc)
+ }
+ if (unknown.nonEmpty) {
+ logWarning(s"Application $appId attempted to kill non-existent
executors: "
--- End diff --
Should this be logged at DEBUG level ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
