[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444881#comment-16444881 ]
Anirudh Ramanathan commented on SPARK-24028: -------------------------------------------- This is curious. Is this a recent change? You mention a 1.9.4 cluster has this issue. Like Yinan said, it doesn't sound like the right behavior if an empty file is found instead of what you expect - the expectation is that the secret/configmap will exist and be mounted, or not exist and cause the pod to go pending and retrying till mounting succeeds. > [K8s] Creating secrets and config maps before creating the driver pod has > unpredictable behavior > ------------------------------------------------------------------------------------------------ > > Key: SPARK-24028 > URL: https://issues.apache.org/jira/browse/SPARK-24028 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 2.3.0 > Reporter: Matt Cheah > Priority: Critical > > Currently we create the Kubernetes resources the driver depends on - such as > the properties config map and secrets to mount into the pod - only after we > create the driver pod. This is because we want these extra objects to > immediately have an owner reference to be tied to the driver pod. > On our Kubernetes 1.9.4. cluster, we're seeing that sometimes this works > fine, but other times the driver ends up being started with empty volumes > instead of volumes with the contents of the secrets we expect. The result is > that sometimes the driver will start without these files mounted, which leads > to various failures if the driver requires these files to be present early on > in their code. Missing the properties file config map, for example, would > mean spark-submit doesn't have a properties file to read at all. See the > warning on [https://kubernetes.io/docs/concepts/storage/volumes/#secret.] > Unfortunately we cannot link owner references to non-existent objects, so we > have to do this instead: > # Create the auxiliary resources without any owner references. > # Create the driver pod mounting these resources into volumes, as before. > # If #2 fails, clean up the resources created in #1. > # Edit the auxiliary resources to have an owner reference for the driver pod. > The multi-step approach leaves a small chance for us to leak resources - for > example, if we fail to make the resource edits in #4 for some reason. This > also changes the permissioning mode required for spark-submit - credentials > provided to spark-submit need to be able to edit resources in addition to > creating them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org