Chao Sun created SPARK-58719:
--------------------------------
Summary: IPv6 RPC address normalization fails for hexadecimal
addresses and driver pod IPs
Key: SPARK-58719
URL: https://issues.apache.org/jira/browse/SPARK-58719
Project: Spark
Issue Type: Bug
Components: Kubernetes, Spark Core
Affects Versions: 4.2.0
Reporter: Chao Sun
h2. Problem
Spark's RpcAddress normalization recognizes IPv6 addresses only when their
characters are decimal digits and colons. Normal IPv6 literals containing
hexadecimal letters therefore bypass normalization and are embedded in Spark
RPC URLs without the brackets required by URI syntax.
For example:
{code}
RpcAddress("2001:db8::dead:beef", 7077).toSparkURL
{code}
Currently produces:
{code}
spark://2001:db8::dead:beef:7077
{code}
Expected:
{code}
spark://[2001:db8::dead:beef]:7077
{code}
h2. Root cause
SPARK-42173 introduced IPv6 address canonicalization but its matcher accepts
only [0-9:] in the address body and [0-9] in the final segment. Existing tests
cover ::1, ::0:1, and 2600::, but none contains hexadecimal letters.
h2. Kubernetes impact
When spark.kubernetes.executor.useDriverPodIP is enabled,
BasicExecutorFeatureStep constructs the executor driver RPC URL from the raw
driver pod IP. An IPv6 address such as 2001:DB8:0:0::BEEF therefore produces an
invalid unbracketed URL and can prevent executors from connecting. This option
is enabled by default on current Spark master.
h2. Proposed fix
Recognize uppercase and lowercase hexadecimal digits in both parts of the
existing IPv6 matcher while preserving address canonicalization, IPv4/hostname
behavior, and RpcAddress equality. Add RpcAddress regression coverage for raw,
bracketed, uppercase, zero-padded, and trailing-hexadecimal IPv6 literals, plus
a Kubernetes driver-pod-IP regression test.
Related historical reports and fixes: SPARK-22077, SPARK-39457, and SPARK-42173.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
