[
https://issues.apache.org/jira/browse/SPARK-39546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-39546.
-----------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
Issue resolved by pull request 37803
[https://github.com/apache/spark/pull/37803]
> Respect port defininitions on K8S pod templates for both driver and executor
> ----------------------------------------------------------------------------
>
> Key: SPARK-39546
> URL: https://issues.apache.org/jira/browse/SPARK-39546
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.3.0
> Reporter: Oliver Koeth
> Assignee: Yilun Fan
> Priority: Minor
> Fix For: 3.4.0
>
>
> *Description:*
> Spark on K8S allows to open additional ports for custom purposes on the
> driver pod via the pod template, but ignores the port specification in the
> executor pod template. Port specifications from the pod template should be
> preserved (and extended) for both drivers and executors.
> *Scenario:*
> I want to run functionality in the executor that exposes data on an
> additional port. In my case, this is monitoring data exposed by Spark's JMX
> metrics sink via the JMX prometheus exporter java agent
> https://github.com/prometheus/jmx_exporter -- the java agent opens an extra
> port inside the container, but for prometheus to detect and scrape the port,
> it must be exposed in the K8S pod resource.
> (More background if desired: This seems to be the "classic" Spark 2 way to
> expose prometheus metrics. Spark 3 introduced a native equivalent servlet for
> the driver, but for the executor, only a rather limited set of metrics is
> forwarded via the driver, and that also follows a completely different naming
> scheme. So the JMX + exporter approach still turns out to be more useful for
> me, even in Spark 3)
> Expected behavior:
> I add the following to my pod template to expose the extra port opened by the
> JMX exporter java agent
> spec:
> containers:
> - ...
> ports:
> - containerPort: 8090
> name: jmx-prometheus
> protocol: TCP
> Observed behavior:
> The port is exposed for driver pods but not for executor pods
> *Corresponding code:*
> driver pod creation just adds ports
> [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala]
> (currently line 115)
> val driverContainer = new ContainerBuilder(pod.container)
> ...
> .addNewPort()
> ...
> .addNewPort()
> while executor pod creation replaces the ports
> [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala]
> (currently line 211)
> val executorContainer = new ContainerBuilder(pod.container)
> ...
> .withPorts(requiredPorts.asJava)
> The current handling is incosistent and unnecessarily limiting. It seems that
> the executor creation could/should just as well preserve pods from the
> template and add extra required ports.
> *Workaround:*
> It is possible to work around this limitation by adding a full sidecar
> container to the executor pod spec which declares the port. Sidecar
> containers are left unchanged by pod template handling.
> As all containers in a pod share the same network, it does not matter which
> container actually declares to expose the port.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
