Yalin Dogu Sahin created SPARK-58691:
----------------------------------------
Summary: NetworkPolicy generated for standalone SparkCluster
workers blocks Prometheus metrics scraping
Key: SPARK-58691
URL: https://issues.apache.org/jira/browse/SPARK-58691
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: kubernetes-operator-1.0.0
Reporter: Yalin Dogu Sahin
h2. Summary
The {{NetworkPolicy}} generated for standalone {{SparkCluster}} workers
blocks Prometheus metrics scraping.
h2. Problem
For a standalone {{SparkCluster}} resource, the operator generates a worker
{{NetworkPolicy}} ({{SparkClusterResourceSpec#buildWorkerNetworkPolicy}}) whose
ingress rule only admits traffic from pods carrying the
cluster label ({{spark.operator/spark-cluster-name: <cluster>}}) or the
driver label ({{spark-role: driver}}). No {{ports}} or {{namespaceSelector}}
are specified.
Because a Prometheus scraper (e.g. Prometheus Operator's
ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its
connection to a worker is dropped. As a result, *worker Prometheus metrics are
un-scrapable in standalone mode*.
* The worker web UI / Prometheus servlet runs on *port 8081*
({{/metrics/prometheus}}), but the worker pod is inside the
NetworkPolicy-restricted set, so the scrape is rejected / times out.
* The *master has no {{NetworkPolicy}} at all*, so master metrics on *8080*
remain reachable — the failure is specific to workers.
This is the same label-based allow-list mechanism addressed for
*block-manager / task-result* traffic in SPARK-58649, but here it surfaces as
blocked metrics scraping rather than task failures.
h2. Reproduction
# Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor
targeting worker pods.
# Attempt to scrape {{<worker-pod>:8081/metrics/prometheus}}.
# The scrape fails because the worker pod is not an allowed ingress source in
the {{NetworkPolicy}} named {{<cluster>-worker}}.
h2. Evidence
|| Policy element || Value ||
| Name | {{<cluster>-worker}} |
| Worker podSelector | {{spark-role: worker}} AND
{{spark.operator/spark-cluster-name: <cluster>}} |
| Allowed ingress {{from}} | (a) pods with
{{spark.operator/spark-cluster-name: <cluster>}}, (b) pods with {{spark-role:
driver}} |
| Ports rule | none (all ports, but only to the {{from}} sources above) |
| Master | no {{NetworkPolicy}} — unrestricted |
h2. Suggested approach (for discussion)
A few options; maintainers' input is welcome:
# *Configurable / default-open* — make the worker NetworkPolicy ingress
configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side
labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
current behavior by default.
# *Add a scrape source* — extend the {{from}} list with a well-known
metrics-scraper label and/or a {{namespaceSelector}} for the monitoring
namespace.
# *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}}
ingress rule for 8081 so the intent is explicit and auditable.
Happy to prepare a patch once the preferred approach is agreed.
h2. Related
* SPARK-55099 — introduced the {{NetworkPolicy}} for {{SparkCluster}}.
https://issues.apache.org/jira/browse/SPARK-55099
* SPARK-58649 — the same NetworkPolicy blocking worker ingress for attached
drivers. https://issues.apache.org/jira/browse/SPARK-58649
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
