[
https://issues.apache.org/jira/browse/SPARK-58691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yalin Dogu Sahin updated SPARK-58691:
-------------------------------------
Description:
h2. Problem
For a standalone {{SparkCluster}} resource, the operator generates a worker
{{NetworkPolicy}} ({{SparkClusterResourceSpec#buildWorkerNetworkPolicy}}) whose
ingress rule only admits traffic from pods carrying the
cluster label ({{spark.operator/spark-cluster-name: <cluster>}}) or the
driver label ({{spark-role: driver}}). No {{ports}} or {{namespaceSelector}}
are specified.
Because a Prometheus scraper (e.g. Prometheus Operator's
ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its
connection to a worker is dropped. As a result, *worker Prometheus metrics are
un-scrapable in standalone mode*.
* The worker web UI / Prometheus servlet runs on *port 8081*
({{/metrics/prometheus}}), but the worker pod is inside the
NetworkPolicy-restricted set, so the scrape is rejected / times out.
* The *master has no {{NetworkPolicy}} at all*, so master metrics on *8080*
remain reachable — the failure is specific to workers.
This is the same label-based allow-list mechanism addressed for
*block-manager / task-result* traffic in
[SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649], but here it
surfaces as blocked metrics scraping rather than task failures.
h2. Reproduction
# Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor
targeting worker pods.
# Attempt to scrape {{<worker-pod>:8081/metrics/prometheus}}.
# The scrape fails because the worker pod is not an allowed ingress source in
the {{NetworkPolicy}} named {{<cluster>-worker}}.
h2. Evidence
|| Policy element || Value ||
| Name | {{<cluster>-worker}} |
| Worker podSelector | {{spark-role: worker}} AND
{{spark.operator/spark-cluster-name: <cluster>}} |
| Allowed ingress {{from}} | (a) pods with
{{spark.operator/spark-cluster-name: <cluster>}}, (b) pods with {{spark-role:
driver}} |
| Ports rule | none (all ports, but only to the {{from}} sources above) |
| Master | no {{NetworkPolicy}} — unrestricted |
h2. Suggested approach (for discussion)
A few options; maintainers' input is welcome:
# *Configurable / default-open* — make the worker NetworkPolicy ingress
configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side
labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
current behavior by default.
# *Add a scrape source* — extend the {{from}} list with a well-known
metrics-scraper label and/or a {{namespaceSelector}} for the monitoring
namespace.
# *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}}
ingress rule for 8081 so the intent is explicit and auditable.
Happy to prepare a patch once the preferred approach is agreed.
h2. Additional context (verified against code on main)
* The operator has no Prometheus/metrics configuration for Spark pods today.
{{MetricsSystem}}/{{MetricsService}} expose only the operator's own metrics.
The worker/master PrometheusServlet on 8081/8080 is not
wired up by the operator — it only exists if the user enables Spark's
built-in PrometheusServlet via their own {{spark.metrics.conf}}.
* {{ClusterSpec}} currently has *no* {{networkPolicy}} (and no metrics)
field. So a fully configurable approach (option 1) introduces new API surface
and requires regenerating the CRD; options 2 and 3 (label/port
carve-out) are self-contained in the worker NetworkPolicy builder and need no
CRD change.
* The core issue is unchanged: the worker NetworkPolicy's ingress allow-list
(cluster label + {{spark-role: driver}}, no ports) excludes arbitrary scraper
pods.
h2. Related
* [SPARK-55099|https://issues.apache.org/jira/browse/SPARK-55099] —
introduced the {{NetworkPolicy}} for {{SparkCluster}}.
* [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649] — the same
NetworkPolicy blocking worker ingress for attached drivers.
was:
h2. Summary
The {{NetworkPolicy}} generated for standalone {{SparkCluster}} workers blocks
Prometheus metrics scraping.
h2. Problem
For a standalone {{SparkCluster}} resource, the operator generates a worker
{{NetworkPolicy}} ({{{}SparkClusterResourceSpec#buildWorkerNetworkPolicy{}}})
whose ingress rule only admits traffic from pods carrying the
cluster label ({{{}spark.operator/spark-cluster-name: <cluster>{}}}) or the
driver label ({{{}spark-role: driver{}}}). No {{ports}} or
{{namespaceSelector}} are specified.
Because a Prometheus scraper (e.g. Prometheus Operator's
ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its
connection to a worker is dropped. As a result, *worker Prometheus metrics are
un-scrapable in standalone mode*.
* The worker web UI / Prometheus servlet runs on *port 8081*
({{{}/metrics/prometheus{}}}), but the worker pod is inside the
NetworkPolicy-restricted set, so the scrape is rejected / times out.
* The {*}master has no {{NetworkPolicy}} at all{*}, so master metrics on
*8080* remain reachable — the failure is specific to workers.
This is the same label-based allow-list mechanism addressed for *block-manager
/ task-result* traffic in
[SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649]], but here it
surfaces as blocked metrics scraping rather than task failures.
h2. Reproduction
# Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor
targeting worker pods.
# Attempt to scrape {{{}<worker-pod>:8081/metrics/prometheus{}}}.
# The scrape fails because the worker pod is not an allowed ingress source in
the {{NetworkPolicy}} named {{{}<cluster>-worker{}}}.
h2. Evidence
||Policy element||Value||
|Name|{{<cluster>-worker}}|
|Worker podSelector|{{spark-role: worker}} AND
{{spark.operator/spark-cluster-name: <cluster>}}|
|Allowed ingress {{from}}|(a) pods with {{{}spark.operator/spark-cluster-name:
<cluster>{}}}, (b) pods with {{spark-role: driver}}|
|Ports rule|none (all ports, but only to the {{from}} sources above)|
|Master|no {{NetworkPolicy}} — unrestricted|
h2. Suggested approach (for discussion)
A few options; maintainers' input is welcome:
# *Configurable / default-open* — make the worker NetworkPolicy ingress
configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side
labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
current behavior by default.
# *Add a scrape source* — extend the {{from}} list with a well-known
metrics-scraper label and/or a {{namespaceSelector}} for the monitoring
namespace.
# *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}}
ingress rule for 8081 so the intent is explicit and auditable.
Happy to prepare a patch once the preferred approach is agreed.
h2. Related
* [SPARK-55099|https://issues.apache.org/jira/browse/SPARK-55099] — introduced
the {{NetworkPolicy}} for {{{}SparkCluster{}}}.
* [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649] — the same
NetworkPolicy blocking worker ingress for attached drivers.
> NetworkPolicy generated for standalone SparkCluster workers blocks Prometheus
> metrics scraping
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-58691
> URL: https://issues.apache.org/jira/browse/SPARK-58691
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.0.0
> Reporter: Yalin Dogu Sahin
> Priority: Minor
>
> h2. Problem
> For a standalone {{SparkCluster}} resource, the operator generates a worker
> {{NetworkPolicy}} ({{SparkClusterResourceSpec#buildWorkerNetworkPolicy}})
> whose ingress rule only admits traffic from pods carrying the
> cluster label ({{spark.operator/spark-cluster-name: <cluster>}}) or the
> driver label ({{spark-role: driver}}). No {{ports}} or {{namespaceSelector}}
> are specified.
> Because a Prometheus scraper (e.g. Prometheus Operator's
> ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its
> connection to a worker is dropped. As a result, *worker Prometheus metrics are
> un-scrapable in standalone mode*.
> * The worker web UI / Prometheus servlet runs on *port 8081*
> ({{/metrics/prometheus}}), but the worker pod is inside the
> NetworkPolicy-restricted set, so the scrape is rejected / times out.
> * The *master has no {{NetworkPolicy}} at all*, so master metrics on *8080*
> remain reachable — the failure is specific to workers.
> This is the same label-based allow-list mechanism addressed for
> *block-manager / task-result* traffic in
> [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649], but here it
> surfaces as blocked metrics scraping rather than task failures.
> h2. Reproduction
> # Deploy a {{SparkCluster}} (master + worker) and a
> Prometheus/ServiceMonitor targeting worker pods.
> # Attempt to scrape {{<worker-pod>:8081/metrics/prometheus}}.
> # The scrape fails because the worker pod is not an allowed ingress source
> in the {{NetworkPolicy}} named {{<cluster>-worker}}.
> h2. Evidence
> || Policy element || Value ||
> | Name | {{<cluster>-worker}} |
> | Worker podSelector | {{spark-role: worker}} AND
> {{spark.operator/spark-cluster-name: <cluster>}} |
> | Allowed ingress {{from}} | (a) pods with
> {{spark.operator/spark-cluster-name: <cluster>}}, (b) pods with {{spark-role:
> driver}} |
> | Ports rule | none (all ports, but only to the {{from}} sources above) |
> | Master | no {{NetworkPolicy}} — unrestricted |
> h2. Suggested approach (for discussion)
> A few options; maintainers' input is welcome:
> # *Configurable / default-open* — make the worker NetworkPolicy ingress
> configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side
> labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
> current behavior by default.
> # *Add a scrape source* — extend the {{from}} list with a well-known
> metrics-scraper label and/or a {{namespaceSelector}} for the monitoring
> namespace.
> # *Port-scoped carve-out* — keep the allow-list but add an explicit
> {{ports}} ingress rule for 8081 so the intent is explicit and auditable.
> Happy to prepare a patch once the preferred approach is agreed.
> h2. Additional context (verified against code on main)
> * The operator has no Prometheus/metrics configuration for Spark pods
> today. {{MetricsSystem}}/{{MetricsService}} expose only the operator's own
> metrics. The worker/master PrometheusServlet on 8081/8080 is not
> wired up by the operator — it only exists if the user enables Spark's
> built-in PrometheusServlet via their own {{spark.metrics.conf}}.
> * {{ClusterSpec}} currently has *no* {{networkPolicy}} (and no metrics)
> field. So a fully configurable approach (option 1) introduces new API surface
> and requires regenerating the CRD; options 2 and 3 (label/port
> carve-out) are self-contained in the worker NetworkPolicy builder and need
> no CRD change.
> * The core issue is unchanged: the worker NetworkPolicy's ingress
> allow-list (cluster label + {{spark-role: driver}}, no ports) excludes
> arbitrary scraper pods.
> h2. Related
> * [SPARK-55099|https://issues.apache.org/jira/browse/SPARK-55099] —
> introduced the {{NetworkPolicy}} for {{SparkCluster}}.
> * [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649] — the
> same NetworkPolicy blocking worker ingress for attached drivers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
