[
https://issues.apache.org/jira/browse/SPARK-33564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Roberto de Oliveira Castro updated SPARK-33564:
-----------------------------------------------------
Description:
Following the [PR|https://github.com/apache/spark/pull/25769] that introduced
the Prometheus sink, I downloaded the {{spark-3.0.1-bin-hadoop2.7.tgz}} (also
tested with 3.0.0), uncompressed the tgz and created a file called
{{metrics.properties}} adding this content:
{quote}{{*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet}}
{{*.sink.prometheusServlet.path=/metrics/prometheus}}
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus
{quote}
Then I ran:
{quote}{{$ sbin/start-master.sh}}
{{$ sbin/start-slave.sh spark://`hostname`:7077}}
{{$ bin/spark-shell --master spark://`hostname`:7077
--files=./metrics.properties --conf spark.metrics.conf=./metrics.properties}}
{quote}
{{The Spark shell opens without problems:}}
{quote}{{20/11/25 17:36:07 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable}}
{{Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties}}
{{Setting default log level to "WARN".}}
{{To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).}}
{{Spark context Web UI available at
[http://192.168.0.6:4040|http://192.168.0.6:4040/]}}
{{Spark context available as 'sc' (master =
spark://MacBook-Pro-de-Paulo-2.local:7077, app id = app-20201125173618-0002).}}
{{Spark session available as 'spark'.}}
{{Welcome to}}
{{ ____ __}}
{{ / __/_ ____ _____/ /__}}
{{ _\ \/ _ \/ _ `/ __/ '_/}}
{{ /___/ .__/_,_/_/ /_/_\ version 3.0.0}}
{{ /_/}}
{{ }}
{{Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)}}
{{Type in expressions to have them evaluated.}}
{{Type :help for more information. }}
{{scala>}}
{quote}
{{And when I try to fetch prometheus metrics for driver, everything works
fine:}}
{quote}$ curl -s [http://localhost:4040/metrics/prometheus/] | head -n 5
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Number\{type="gauges"}
0
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Value\{type="gauges"}
0
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Number\{type="gauges"}
732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Value\{type="gauges"}
732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxOffHeapMem_MB_Number\{type="gauges"}
0
{quote}
*The problem appears when I try accessing master metrics*, and I get the
following problem:
{quote}{{$ curl -s [http://localhost:8080/metrics/master/prometheus]}}
{{<!DOCTYPE html><html>}}
{{ <head>}}
{{ <meta http-equiv="Content-type" content="text/html;
charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css"
type="text/css"/><link rel="stylesheet"
href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link
rel="stylesheet" href="/static/webui.css" type="text/css"/><link
rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script
src="/static/sorttable.js"></script><script
src="/static/jquery-3.4.1.min.js"></script><script
src="/static/vis-timeline-graph2d.min.js"></script><script
src="/static/bootstrap-tooltip.js"></script><script
src="/static/initialize-tooltips.js"></script><script
src="/static/table.js"></script><script
src="/static/timeline-view.js"></script><script
src="/static/log-view.js"></script><script
src="/static/webui.js"></script><script>setUIRoot('')</script>}}
{{ }}
{{ <link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>}}
{{ <title>Spark Master at
spark://MacBook-Pro-de-Paulo-2.local:7077</title>}}
{{ </head>}}
{{ <body>}}
{{ <div class="container-fluid">}}
{{ <div class="row-fluid">}}
{{ <div class="span12">}}
{{ <h3 style="vertical-align: middle; display: inline-block;">}}
{{ <a style="text-decoration: none" href="/">}}
{{ <img src="/static/spark-logo-77x50px-hd.png"/>}}
{{ <span class="version" style="margin-right:
15px;">3.0.0</span>}}
{{ </a>}}
{{ Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077}}
{{ </h3>}}
{{ </div>}}
{{ </div>}}
{{ <div class="row-fluid">}}
{{ <div class="span12">}}
{{ <ul class="unstyled">}}
{{ <li><strong>URL:</strong>
spark://MacBook-Pro-de-Paulo-2.local:7077</li>}}
...
{quote}
The same happens for all of those here:
{quote}{{$ curl -s [http://localhost:8080/metrics/applications/prometheus/]}}
{{$ curl -s [http://localhost:8081/metrics/prometheus/]}}
{quote}
Instead, *I expected metrics in prometheus metrics*. All related JSON endpoints
seem to be working fine.
was:
Following the [PR|https://github.com/apache/spark/pull/25769] that introduced
the Prometheus sink, I downloaded the {{spark-3.0.1-bin-hadoop2.7.tgz}} (also
tested with 3.0.0), uncompressed the tgz and created a file called
{{metrics.properties}} adding this content:
{quote}{{{{*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet}}}}
{{*.sink.prometheusServlet.path=/metrics/prometheus}}
\{{ master.sink.prometheusServlet.path=/metrics/master/prometheus}}
\{{ applications.sink.prometheusServlet.path=/metrics/applications/prometheus}}
{quote}
Then I ran:
{quote}{{$ sbin/start-master.sh}}
{{$ sbin/start-slave.sh spark://`hostname`:7077}}
{{$ bin/spark-shell --master spark://`hostname`:7077
--files=./metrics.properties --conf spark.metrics.conf=./metrics.properties}}
{quote}
{{The Spark shell opens without problems:}}
{quote}{{20/11/25 17:36:07 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable}}
{{Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties}}
{{Setting default log level to "WARN".}}
{{To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).}}
{{Spark context Web UI available at
[http://192.168.0.6:4040|http://192.168.0.6:4040/]}}
{{Spark context available as 'sc' (master =
spark://MacBook-Pro-de-Paulo-2.local:7077, app id = app-20201125173618-0002).}}
{{Spark session available as 'spark'.}}
{{Welcome to}}
{{ ____ __}}
{{ / __/_ ____ _____/ /__}}
{{ _\ \/ _ \/ _ `/ __/ '_/}}
{{ /___/ .__/_,_/_/ /_/_\ version 3.0.0}}
{{ /_/}}
{{ }}
{{Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)}}
{{Type in expressions to have them evaluated.}}
{{Type :help for more information. }}
{{scala>}}
{quote}
{{And when I try to fetch prometheus metrics for driver, everything works
fine:}}
{quote}$ curl -s [http://localhost:4040/metrics/prometheus/] | head -n 5
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Number\{type="gauges"}
0
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Value\{type="gauges"}
0
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Number\{type="gauges"}
732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Value\{type="gauges"}
732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxOffHeapMem_MB_Number\{type="gauges"}
0
{quote}
*The problem appears when I try accessing master metrics*, and I get the
following problem:
{quote}{{$ curl -s [http://localhost:8080/metrics/master/prometheus]}}
{{<!DOCTYPE html><html>}}
{{ <head>}}
{{ <meta http-equiv="Content-type" content="text/html;
charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css"
type="text/css"/><link rel="stylesheet"
href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link
rel="stylesheet" href="/static/webui.css" type="text/css"/><link
rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script
src="/static/sorttable.js"></script><script
src="/static/jquery-3.4.1.min.js"></script><script
src="/static/vis-timeline-graph2d.min.js"></script><script
src="/static/bootstrap-tooltip.js"></script><script
src="/static/initialize-tooltips.js"></script><script
src="/static/table.js"></script><script
src="/static/timeline-view.js"></script><script
src="/static/log-view.js"></script><script
src="/static/webui.js"></script><script>setUIRoot('')</script>}}
{{ }}
{{ <link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>}}
{{ <title>Spark Master at
spark://MacBook-Pro-de-Paulo-2.local:7077</title>}}
{{ </head>}}
{{ <body>}}
{{ <div class="container-fluid">}}
{{ <div class="row-fluid">}}
{{ <div class="span12">}}
{{ <h3 style="vertical-align: middle; display: inline-block;">}}
{{ <a style="text-decoration: none" href="/">}}
{{ <img src="/static/spark-logo-77x50px-hd.png"/>}}
{{ <span class="version" style="margin-right:
15px;">3.0.0</span>}}
{{ </a>}}
{{ Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077}}
{{ </h3>}}
{{ </div>}}
{{ </div>}}
{{ <div class="row-fluid">}}
{{ <div class="span12">}}
{{ <ul class="unstyled">}}
{{ <li><strong>URL:</strong>
spark://MacBook-Pro-de-Paulo-2.local:7077</li>}}
...
{quote}
The same happens for all of those here:
{quote}{{$ curl -s [http://localhost:8080/metrics/applications/prometheus/]}}
{{$ curl -s [http://localhost:8081/metrics/prometheus/]}}
{quote}
Instead, *I expected metrics in prometheus metrics*. All related JSON endpoints
seem to be working fine.
> Prometheus metrics for Master and Worker isn't working
> -------------------------------------------------------
>
> Key: SPARK-33564
> URL: https://issues.apache.org/jira/browse/SPARK-33564
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Shell
> Affects Versions: 3.0.0, 3.0.1
> Reporter: Paulo Roberto de Oliveira Castro
> Priority: Major
> Labels: Metrics, metrics, prometheus
>
> Following the [PR|https://github.com/apache/spark/pull/25769] that introduced
> the Prometheus sink, I downloaded the {{spark-3.0.1-bin-hadoop2.7.tgz}}
> (also tested with 3.0.0), uncompressed the tgz and created a file called
> {{metrics.properties}} adding this content:
> {quote}{{*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet}}
> {{*.sink.prometheusServlet.path=/metrics/prometheus}}
> master.sink.prometheusServlet.path=/metrics/master/prometheus
> applications.sink.prometheusServlet.path=/metrics/applications/prometheus
> {quote}
> Then I ran:
> {quote}{{$ sbin/start-master.sh}}
> {{$ sbin/start-slave.sh spark://`hostname`:7077}}
> {{$ bin/spark-shell --master spark://`hostname`:7077
> --files=./metrics.properties --conf spark.metrics.conf=./metrics.properties}}
> {quote}
> {{The Spark shell opens without problems:}}
> {quote}{{20/11/25 17:36:07 WARN NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable}}
> {{Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties}}
> {{Setting default log level to "WARN".}}
> {{To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).}}
> {{Spark context Web UI available at
> [http://192.168.0.6:4040|http://192.168.0.6:4040/]}}
> {{Spark context available as 'sc' (master =
> spark://MacBook-Pro-de-Paulo-2.local:7077, app id =
> app-20201125173618-0002).}}
> {{Spark session available as 'spark'.}}
> {{Welcome to}}
> {{ ____ __}}
> {{ / __/_ ____ _____/ /__}}
> {{ _\ \/ _ \/ _ `/ __/ '_/}}
> {{ /___/ .__/_,_/_/ /_/_\ version 3.0.0}}
> {{ /_/}}
> {{ }}
> {{Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)}}
> {{Type in expressions to have them evaluated.}}
> {{Type :help for more information. }}
> {{scala>}}
> {quote}
> {{And when I try to fetch prometheus metrics for driver, everything works
> fine:}}
> {quote}$ curl -s [http://localhost:4040/metrics/prometheus/] | head -n 5
> metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Number\{type="gauges"}
> 0
> metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Value\{type="gauges"}
> 0
> metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Number\{type="gauges"}
> 732
> metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Value\{type="gauges"}
> 732
> metrics_app_20201125173618_0002_driver_BlockManager_memory_maxOffHeapMem_MB_Number\{type="gauges"}
> 0
> {quote}
> *The problem appears when I try accessing master metrics*, and I get the
> following problem:
> {quote}{{$ curl -s [http://localhost:8080/metrics/master/prometheus]}}
> {{<!DOCTYPE html><html>}}
> {{ <head>}}
> {{ <meta http-equiv="Content-type" content="text/html;
> charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css"
> type="text/css"/><link rel="stylesheet"
> href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link
> rel="stylesheet" href="/static/webui.css" type="text/css"/><link
> rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script
> src="/static/sorttable.js"></script><script
> src="/static/jquery-3.4.1.min.js"></script><script
> src="/static/vis-timeline-graph2d.min.js"></script><script
> src="/static/bootstrap-tooltip.js"></script><script
> src="/static/initialize-tooltips.js"></script><script
> src="/static/table.js"></script><script
> src="/static/timeline-view.js"></script><script
> src="/static/log-view.js"></script><script
> src="/static/webui.js"></script><script>setUIRoot('')</script>}}
> {{ }}
> {{ <link rel="shortcut icon"
> href="/static/spark-logo-77x50px-hd.png"></link>}}
> {{ <title>Spark Master at
> spark://MacBook-Pro-de-Paulo-2.local:7077</title>}}
> {{ </head>}}
> {{ <body>}}
> {{ <div class="container-fluid">}}
> {{ <div class="row-fluid">}}
> {{ <div class="span12">}}
> {{ <h3 style="vertical-align: middle; display: inline-block;">}}
> {{ <a style="text-decoration: none" href="/">}}
> {{ <img src="/static/spark-logo-77x50px-hd.png"/>}}
> {{ <span class="version" style="margin-right:
> 15px;">3.0.0</span>}}
> {{ </a>}}
> {{ Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077}}
> {{ </h3>}}
> {{ </div>}}
> {{ </div>}}
> {{ <div class="row-fluid">}}
> {{ <div class="span12">}}
> {{ <ul class="unstyled">}}
> {{ <li><strong>URL:</strong>
> spark://MacBook-Pro-de-Paulo-2.local:7077</li>}}
> ...
> {quote}
> The same happens for all of those here:
> {quote}{{$ curl -s [http://localhost:8080/metrics/applications/prometheus/]}}
> {{$ curl -s [http://localhost:8081/metrics/prometheus/]}}
> {quote}
> Instead, *I expected metrics in prometheus metrics*. All related JSON
> endpoints seem to be working fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
