This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 07ccaa6f4c7 [SPARK-39884][K8S] `KubernetesExecutorBackend` should
handle `IPv6` hostname
07ccaa6f4c7 is described below
commit 07ccaa6f4c7c37339ee10c3d7b337059fc7468a2
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Tue Jul 26 17:00:33 2022 -0700
[SPARK-39884][K8S] `KubernetesExecutorBackend` should handle `IPv6` hostname
### What changes were proposed in this pull request?
This PR aims to fix `KubernetesExecutorBackend` to handle IPv6 hostname.
### Why are the changes needed?
`entrypoint.sh` uses `SPARK_EXECUTOR_POD_IP` for `--hostname`.
https://github.com/apache/spark/blob/e9eb28e27d10497c8b36774609823f4bbd2c8500/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh#L97
However, IPv6 `status.podIP` has no `[]`. We need to handle both IPv6 cases
(`[]` or no `[]`).
```yaml
- name: SPARK_EXECUTOR_POD_IP
fieldPath: status.podIP
value: -Djava.net.preferIPv6Addresses=true
podIP: 2600:1f14:552:fb00:4c14::3
```
### Does this PR introduce _any_ user-facing change?
No, previously, it fails immediately.
```
22/07/26 19:20:46 INFO Executor: Starting executor ID 5 on host
2600:1f14:552:fb02:2dc2::1
22/07/26 19:20:46 ERROR CoarseGrainedExecutorBackend: Executor self-exiting
due to : Unable to create executor due to assertion failed: Expected hostname
or IPv6 IP enclosed in [] but got 2600:1f14:552:fb02:2dc2::1
```
### How was this patch tested?
Pass the CIs.
Closes #37306 from dongjoon-hyun/SPARK-39884.
Authored-by: Dongjoon Hyun <dongj...@apache.org>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
.../apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
index fbf485cfa2f..bec54a11366 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala
@@ -158,7 +158,8 @@ private[spark] object KubernetesExecutorBackend extends
Logging {
bindAddress = value
argv = tail
case ("--hostname") :: value :: tail =>
- hostname = value
+ // entrypoint.sh sets SPARK_EXECUTOR_POD_IP without '[]'
+ hostname = Utils.addBracketsIfNeeded(value)
argv = tail
case ("--cores") :: value :: tail =>
cores = value.toInt
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
