[
https://issues.apache.org/jira/browse/SPARK-58719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-58719.
------------------------------
Fix Version/s: 4.3.0
5.0.0
Resolution: Fixed
Issue resolved by pull request 57938
[https://github.com/apache/spark/pull/57938]
> IPv6 RPC address normalization fails for hexadecimal addresses and driver pod
> IPs
> ---------------------------------------------------------------------------------
>
> Key: SPARK-58719
> URL: https://issues.apache.org/jira/browse/SPARK-58719
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
> Affects Versions: 4.2.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0, 5.0.0
>
>
> h2. Problem
> Spark's RpcAddress normalization recognizes IPv6 addresses only when their
> characters are decimal digits and colons. Normal IPv6 literals containing
> hexadecimal letters therefore bypass normalization and are embedded in Spark
> RPC URLs without the brackets required by URI syntax.
> For example:
> {code}
> RpcAddress("2001:db8::dead:beef", 7077).toSparkURL
> {code}
> Currently produces:
> {code}
> spark://2001:db8::dead:beef:7077
> {code}
> Expected:
> {code}
> spark://[2001:db8::dead:beef]:7077
> {code}
> h2. Root cause
> SPARK-42173 introduced IPv6 address canonicalization but its matcher accepts
> only [0-9:] in the address body and [0-9] in the final segment. Existing
> tests cover ::1, ::0:1, and 2600::, but none contains hexadecimal letters.
> h2. Kubernetes impact
> When spark.kubernetes.executor.useDriverPodIP is enabled,
> BasicExecutorFeatureStep constructs the executor driver RPC URL from the raw
> driver pod IP. An IPv6 address such as 2001:DB8:0:0::BEEF therefore produces
> an invalid unbracketed URL and can prevent executors from connecting. This
> option is enabled by default on current Spark master.
> h2. Proposed fix
> Recognize uppercase and lowercase hexadecimal digits in both parts of the
> existing IPv6 matcher while preserving address canonicalization,
> IPv4/hostname behavior, and RpcAddress equality. Add RpcAddress regression
> coverage for raw, bracketed, uppercase, zero-padded, and trailing-hexadecimal
> IPv6 literals, plus a Kubernetes driver-pod-IP regression test.
> Related historical reports and fixes: SPARK-22077, SPARK-39457, and
> SPARK-42173.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
