[
https://issues.apache.org/jira/browse/SPARK-59691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59691:
-----------------------------------
Labels: pull-request-available (was: )
> Report a failed suspend check of a SparkApplication or SparkCluster
> -------------------------------------------------------------------
>
> Key: SPARK-59691
> URL: https://issues.apache.org/jira/browse/SPARK-59691
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.1.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> *This is a follow-up of SPARK-59680, raised while reviewing
> https://github.com/apache/spark-kubernetes-operator/pull/857.*
> SPARK-59680 makes both init steps stop reading a failed lookup of the driver
> pod or the master {{StatefulSet}} as "never requested". When the lookup
> fails, the suspend branch returns {{completeAndDefaultRequeue()}} without
> going through {{SuspendUtils.holdForSuspend}}, so that no {{SuspendHeld}}
> event claims that no driver or master was requested.
> That is right, but it publishes nothing at all, so a *persistent* failure
> leaves the resource with no signal:
> * a suspended resource never gets its initial {{Submitted}} status persisted,
> which {{docs/spark_custom_resources.md}} states outright, so {{kubectl get}}
> shows an empty {{Current State}};
> * the branch returns before the event, so {{kubectl describe}} shows nothing
> either;
> * the reconciliation loops at the default interval indefinitely, because the
> operator cannot decide whether the driver or master is live.
> Silence is the right answer for a transport-level failure, for the same
> reason the Kueue admission branch gives: writing an event would only add load
> to an API server that is often the cause. It is the persistent half that is
> asymmetric, since the admission branch a few lines down classifies the
> identical exception with {{ReconcilerUtils.isTransientError}} and reports it
> as {{KueueAdmissionRequestFailed}}.
> {{KueueAdmissionRequestFailed}} does not fit here, since no admission is
> involved, so this needs its own {{Warning}} reason:
> * a {{REASON_SUSPEND_CHECK_FAILED}} constant in {{EventUtils}};
> * the classified publish in both suspend branches, {{AppInitStep}} (which
> came in with SPARK-59677) and {{ClusterInitStep}} (SPARK-59680);
> * a row for the reason in the {{Warning}} table of {{docs/configuration.md}}.
> The reason is user-facing on its own, which is why it is a separate change
> rather than part of SPARK-59680.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
