James Carter created SPARK-25162: ------------------------------------ Summary: Kubernetes 'in-cluster' client mode and value spark.driver.host Key: SPARK-25162 URL: https://issues.apache.org/jira/browse/SPARK-25162 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 2.4.0 Environment: A java program, deployed to kubernetes, that establishes a Spark Context in client mode.
Not using spark-submit. Kubernetes 1.10 AWS EKS Reporter: James Carter When creating Kubernetes scheduler 'in-cluster' using client mode, the value for spark.driver.host can be derived from the IP address of the driver pod. I observed that the value of _spark.driver.host_ defaulted to the value of _spark.kubernetes.driver.pod.name_, which is not a valid hostname. This caused the executors to fail to establish a connection back to the driver. As a work around, in my configuration I pass the driver's pod name _and_ the driver's ip address to ensure that executors can establish a connection with the driver. _spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath: metadata.name _spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp Ideally only _spark.kubernetes.driver.pod.name_ need be provided in this deployment scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org