Chao Sun created SPARK-58749:
--------------------------------
Summary: Preserve advertised Kubernetes driver host when bind
address is wildcard
Key: SPARK-58749
URL: https://issues.apache.org/jira/browse/SPARK-58749
Project: Spark
Issue Type: Bug
Components: Kubernetes, Spark Core, Connect
Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
Reporter: Chao Sun
Spark Kubernetes client-mode drivers commonly bind RPC sockets to 0.0.0.0 or an
IPv6 wildcard while advertising a routable address through spark.driver.host.
When spark.kubernetes.executor.useDriverPodIP=true, SparkContext replaces
spark.driver.host with spark.driver.bindAddress, and BasicExecutorFeatureStep
independently uses that bind address to construct SPARK_DRIVER_URL. This
produces executor URLs such as spark://[email protected]:7078, so
executors cannot register with the driver.
The option was introduced in Spark 4.1 and is enabled by default in Spark 4.3
and current master, making client-mode and Spark Connect drivers vulnerable
without explicitly enabling the setting.
Reproduction: launch a Kubernetes client-mode driver with
spark.driver.bindAddress=0.0.0.0, spark.driver.host set to a routable driver
address, and spark.kubernetes.executor.useDriverPodIP=true. Observe executor
connection failures against 0.0.0.0.
Expected: preserve the advertised spark.driver.host whenever
spark.driver.bindAddress is an IPv4 or IPv6 wildcard; continue using and
normalizing concrete driver Pod IPs. Both SparkContext and executor driver URL
construction need the same wildcard guard.
Affected versions: 4.1.x and 4.2.x when explicitly enabled; 4.3.0 and 5.0.0 by
default. Prioritize a 4.3 backport.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
