[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host
[
https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589286#comment-16589286
]
Yinan Li commented on SPARK-25162:
--
> Where the driver is running _outside-cluster client_ mode, would you
>recommend a default behavior of deriving the IP address of the host on which
>the driver is running (provided that IP address is routable from inside the
>cluster) and giving the user the option to override and supply a FQDN or
>routable IP address for the driver?
The philosophy behind the client mode in the Kubernetes deployment mode is to
not be opinionated on how users setup network connectivity from the executors
to the driver. So it's really up to the users to decide what's the best way to
provide such connectivity. Please check out
https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md#client-mode.
> Kubernetes 'in-cluster' client mode and value of spark.driver.host
> --
>
> Key: SPARK-25162
> URL: https://issues.apache.org/jira/browse/SPARK-25162
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.4.0
> Environment: A java program, deployed to kubernetes, that establishes
> a Spark Context in client mode.
> Not using spark-submit.
> Kubernetes 1.10
> AWS EKS
>
>
>Reporter: James Carter
>Priority: Minor
>
> When creating Kubernetes scheduler 'in-cluster' using client mode, the value
> for spark.driver.host can be derived from the IP address of the driver pod.
> I observed that the value of _spark.driver.host_ defaulted to the value of
> _spark.kubernetes.driver.pod.name_, which is not a valid hostname. This
> caused the executors to fail to establish a connection back to the driver.
> As a work around, in my configuration I pass the driver's pod name _and_ the
> driver's ip address to ensure that executors can establish a connection with
> the driver.
> _spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath:
> metadata.name
> _spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp
> e.g.
> Deployment:
> {noformat}
> env:
> - name: DRIVER_POD_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DRIVER_POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> {noformat}
>
> Application Properties:
> {noformat}
> config[spark.kubernetes.driver.pod.name]: ${DRIVER_POD_NAME}
> config[spark.driver.host]: ${DRIVER_POD_IP}
> {noformat}
>
> BasicExecutorFeatureStep.scala:
> {code:java}
> private val driverUrl = RpcEndpointAddress(
> kubernetesConf.get("spark.driver.host"),
> kubernetesConf.sparkConf.getInt("spark.driver.port", DEFAULT_DRIVER_PORT),
> CoarseGrainedSchedulerBackend.ENDPOINT_NAME).toString
> {code}
>
> Ideally only _spark.kubernetes.driver.pod.name_ would need be provided in
> this deployment scenario.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host
[
https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588723#comment-16588723
]
James Carter commented on SPARK-25162:
--
Thank you Yinan.
Deriving {{spark.driver.bindAddress}} from {{spark.kubernetes.driver.pod.name}}
makes sense to me for both _in-cluster client_ mode and _cluster_ mode.
Where the driver is running _outside-cluster client_ mode, would you recommend
a default behavior of deriving the IP address of the host on which the driver
is running (provided that IP address is routable from inside the cluster) and
giving the user the option to override and supply a FQDN or routable IP address
for the driver?
> Kubernetes 'in-cluster' client mode and value of spark.driver.host
> --
>
> Key: SPARK-25162
> URL: https://issues.apache.org/jira/browse/SPARK-25162
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.4.0
> Environment: A java program, deployed to kubernetes, that establishes
> a Spark Context in client mode.
> Not using spark-submit.
> Kubernetes 1.10
> AWS EKS
>
>
>Reporter: James Carter
>Priority: Minor
>
> When creating Kubernetes scheduler 'in-cluster' using client mode, the value
> for spark.driver.host can be derived from the IP address of the driver pod.
> I observed that the value of _spark.driver.host_ defaulted to the value of
> _spark.kubernetes.driver.pod.name_, which is not a valid hostname. This
> caused the executors to fail to establish a connection back to the driver.
> As a work around, in my configuration I pass the driver's pod name _and_ the
> driver's ip address to ensure that executors can establish a connection with
> the driver.
> _spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath:
> metadata.name
> _spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp
> e.g.
> Deployment:
> {noformat}
> env:
> - name: DRIVER_POD_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DRIVER_POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> {noformat}
>
> Application Properties:
> {noformat}
> config[spark.kubernetes.driver.pod.name]: ${DRIVER_POD_NAME}
> config[spark.driver.host]: ${DRIVER_POD_IP}
> {noformat}
>
> BasicExecutorFeatureStep.scala:
> {code:java}
> private val driverUrl = RpcEndpointAddress(
> kubernetesConf.get("spark.driver.host"),
> kubernetesConf.sparkConf.getInt("spark.driver.port", DEFAULT_DRIVER_PORT),
> CoarseGrainedSchedulerBackend.ENDPOINT_NAME).toString
> {code}
>
> Ideally only _spark.kubernetes.driver.pod.name_ would need be provided in
> this deployment scenario.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host
[
https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588014#comment-16588014
]
Yinan Li commented on SPARK-25162:
--
We actually moved away from using the IP address of the driver pod to set
{{spark.driver.host}}, to using a headless service to give the driver pod a
FQDN name and set {{spark.driver.host}} to the FQDN name. Internally, we set
{{spark.driver.bindAddress}} to the value of environment variable
{{SPARK_DRIVER_BIND_ADDRESS}} which gets its value from the IP address of the
pod using the downward API. We could do the same for
{{spark.kubernetes.driver.pod.name}} as you suggested for in-cluster client
mode.
> Kubernetes 'in-cluster' client mode and value of spark.driver.host
> --
>
> Key: SPARK-25162
> URL: https://issues.apache.org/jira/browse/SPARK-25162
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.4.0
> Environment: A java program, deployed to kubernetes, that establishes
> a Spark Context in client mode.
> Not using spark-submit.
> Kubernetes 1.10
> AWS EKS
>
>
>Reporter: James Carter
>Priority: Minor
>
> When creating Kubernetes scheduler 'in-cluster' using client mode, the value
> for spark.driver.host can be derived from the IP address of the driver pod.
> I observed that the value of _spark.driver.host_ defaulted to the value of
> _spark.kubernetes.driver.pod.name_, which is not a valid hostname. This
> caused the executors to fail to establish a connection back to the driver.
> As a work around, in my configuration I pass the driver's pod name _and_ the
> driver's ip address to ensure that executors can establish a connection with
> the driver.
> _spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath:
> metadata.name
> _spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp
> e.g.
> Deployment:
> {noformat}
> env:
> - name: DRIVER_POD_NAME
> valueFrom:
> fieldRef:
> fieldPath: metadata.name
> - name: DRIVER_POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> {noformat}
>
> Application Properties:
> {noformat}
> config[spark.kubernetes.driver.pod.name]: ${DRIVER_POD_NAME}
> config[spark.driver.host]: ${DRIVER_POD_IP}
> {noformat}
>
> BasicExecutorFeatureStep.scala:
> {code:java}
> private val driverUrl = RpcEndpointAddress(
> kubernetesConf.get("spark.driver.host"),
> kubernetesConf.sparkConf.getInt("spark.driver.port", DEFAULT_DRIVER_PORT),
> CoarseGrainedSchedulerBackend.ENDPOINT_NAME).toString
> {code}
>
> Ideally only _spark.kubernetes.driver.pod.name_ would need be provided in
> this deployment scenario.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
