Github user erikerlandson commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186859389 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50 +322,83 @@ private[spark] class KubernetesClusterSchedulerBackend( override def eventReceived(action: Action, pod: Pod): Unit = { val podName = pod.getMetadata.getName val podIP = pod.getStatus.getPodIP - + val podPhase = pod.getStatus.getPhase action match { - case Action.MODIFIED if (pod.getStatus.getPhase == "Running" + case Action.MODIFIED if (podPhase == "Running" && pod.getMetadata.getDeletionTimestamp == null) => val clusterNodeName = pod.getSpec.getNodeName logInfo(s"Executor pod $podName ready, launched at $clusterNodeName as IP $podIP.") executorPodsByIPs.put(podIP, pod) - case Action.DELETED | Action.ERROR => + case Action.MODIFIED if (podPhase == "Init:Error" || podPhase == "Init:CrashLoopBackoff") + && pod.getMetadata.getDeletionTimestamp == null => val executorId = getExecutorId(pod) - logDebug(s"Executor pod $podName at IP $podIP was at $action.") - if (podIP != null) { - executorPodsByIPs.remove(podIP) + failedInitExecutors.add(executorId) + if (failedInitExecutors.size >= executorMaxInitErrors) { + val errorMessage = s"Aborting Spark application because $executorMaxInitErrors" + + s" executors failed to start. The maximum number of allowed startup failures is" + + s" $executorMaxInitErrors. Please contact your cluster administrator or increase" + + s" your setting of ${KUBERNETES_EXECUTOR_MAX_INIT_ERRORS.key}." + logError(errorMessage) + KubernetesClusterSchedulerBackend.this.scheduler.sc.stopInNewThread() --- End diff -- Leaning on configured minimum executors could be construed as a way to let the user express application-dependent throughput minimums. If the use case is terabyte-scale, then setting minimum executors appropriately would address that. A %-threshold is also reasonable, it just adds a new knob. Requesting new executors indefinitely seems plausible, as long as failed pods don't accumulate. Having a submission hang indefinitely while kube churns may be confusing behavior, at least if users are not familiar with kube and hoping to treat it transparently. Maybe mesos backend policies could provide an analogy?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org