Tom van Bussel created SPARK-27071:
--------------------------------------
Summary: Expose additional metrics in status.api.v1.StageData
Key: SPARK-27071
URL: https://issues.apache.org/jira/browse/SPARK-27071
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Tom van Bussel
Currently StageData exposes the following metrics:
* executorRunTime
* executorCpuTime
* inputBytes
* inputRecords
* outputBytes
* outputRecords
* shuffleReadBytes
* shuffleReadRecords
* shuffleWriteBytes
* shuffleWriteRecords
* memoryBytesSpilled
* diskBytesSpilled
These metrics are computed by aggregating the metrics of the tasks in the
stage. For the task metrics however we keep track of a lot more metrics.
Currently these metrics are also computed for stages (such shuffle read fetch
wait time), but these are not exposed through the api. It would be very useful
if these were also exposed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
