Chenyu Zheng created SPARK-58499:
------------------------------------
Summary: Support hostNetwork mode for kubernetes
Key: SPARK-58499
URL: https://issues.apache.org/jira/browse/SPARK-58499
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 4.3.0
Environment: When running with hostNetwork enabled, driver ports can
collide across pods co-located on the same node. There are two ways to work
around this:
* 1. Fixed port — on collision, Spark retries the next port (+1) until it
finds a free one.
* 2. Bind port 0 — let the OS pick a free port.
Both share the same problem: the port configured on the Kubernetes Service no
longer matches the port the driver actually binds to. To make the Service
usable, its targetPort must be patched dynamically to the actual bound port.
I propose introducing a hostNetwork-enabled switch, similar to FLINK-24947.
When enabled, all driver ports that need binding are set to 0, and their
corresponding Service targetPorts are patched dynamically once the actual ports
are known.
Reporter: Chenyu Zheng
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
