GitHub user pgandhi999 opened a pull request:
https://github.com/apache/spark/pull/22498
[SPARK-18364] : Expose metrics for YarnShuffleService
ExternalShuffleService exposes metrics as of SPARK-16405. However,
YarnShuffleService does not.
The work of instrumenting ExternalShuffleBlockHandler was already done in
SPARK-16405, so this JIRA is for creating a MetricsSystem in YarnShuffleService
similarly to how ExternalShuffleService already does it.
## What changes were proposed in this pull request?
Added shuffle server metrics for Spark Yarn shuffle service. I have made my
changes on top of Andrew Ash's PR and have additionally added two more metrics
on top of them: numRegisteredConnections which indicate the number of
registered connections to the shuffle service and numActiveConnections which
indicate the number of active connections to the shuffle service at any given
point in time.
## How was this patch tested?
If these metrics are outputted to a file, we get something like this:
1533674653489 default.shuffleService:
Hostname=openqe26blue-n9.blue.ygrid.yahoo.com,
openBlockRequestLatencyMillis_count=729,
openBlockRequestLatencyMillis_rate15=0.7110833548897356,
openBlockRequestLatencyMillis_rate5=1.657808981793011,
openBlockRequestLatencyMillis_rate1=2.2404486061620474,
openBlockRequestLatencyMillis_rateMean=0.9242558551196706,
numRegisteredConnections=35,
blockTransferRateBytes_count=2635880512,
blockTransferRateBytes_rate15=2578547.6094160094,
blockTransferRateBytes_rate5=6048721.726302424,
blockTransferRateBytes_rate1=8548922.518223226,
blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5,
registerExecutorRequestLatencyMillis_count=5,
registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836,
registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426,
registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6,
registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053,
numActiveConnections=35
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pgandhi999/spark SPARK-18364
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22498.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22498
----
commit cf74f36dbefd131e10d3fb0148b5f259077fff86
Author: pgandhi <pgandhi@...>
Date: 2018-09-20T15:04:24Z
[SPARK-18364] : Expose metrics for YarnShuffleService
Added shuffle server metrics for Spark Yarn shuffle service. I have made my
changes on top of Andrew Ash's PR and have additionally added two more metrics
on top of them: numRegisteredConnections which indicate the number of
registered connections to the shuffle service and numActiveConnections which
indicate the number of active connections to the shuffle service at any given
point in time. If these metrics are outputted to a file, we get something like
this:
1533674653489 default.shuffleService:
Hostname=openqe26blue-n9.blue.ygrid.yahoo.com,
openBlockRequestLatencyMillis_count=729,
openBlockRequestLatencyMillis_rate15=0.7110833548897356,
openBlockRequestLatencyMillis_rate5=1.657808981793011,
openBlockRequestLatencyMillis_rate1=2.2404486061620474,
openBlockRequestLatencyMillis_rateMean=0.9242558551196706,
numRegisteredConnections=35,
blockTransferRateBytes_count=2635880512,
blockTransferRateBytes_rate15=2578547.6094160094,
blockTransferRateBytes_rate5=6048721.726302424,
blockTransferRateBytes_rate1=8548922.518223226,
blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5,
registerExecutorRequestLatencyMillis_count=5,
registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836,
registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426,
registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6,
registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053,
numActiveConnections=35
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
