[
https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723610#comment-16723610
]
ASF GitHub Bot commented on SPARK-23431:
----------------------------------------
edwinalu opened a new pull request #23340: [SPARK-23431][CORE] Expose the new
executor memory metrics at the stage level
URL: https://github.com/apache/spark/pull/23340
## What changes were proposed in this pull request?
Collect and show the new executor memory metrics for each stage, to provide
more information on how memory is used per stage. Peak values for metrics are
show for each stage. For executor summaries for each stage, the peak values per
executor are also shown.
## How was this patch tested?
Added new unit tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Expose the new executor memory metrics at the stage level
> ---------------------------------------------------------
>
> Key: SPARK-23431
> URL: https://issues.apache.org/jira/browse/SPARK-23431
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Edwina Lu
> Priority: Major
>
> Collect and show the new executor memory metrics for each stage, to provide
> more information on how memory is used per stage.
> Modify the AppStatusListener to track the peak values for JVM used memory,
> execution memory, storage memory, and unified memory for each executor for
> each stage.
> Add the peak values for the metrics to the stages REST API. Also add a new
> stageSummary REST API, which will return executor summary metrics for a
> specified stage:
> {code:java}
> curl http://<spark history server>:18080/api/v1/applications/<application
> id>/<application attempt/stages/<stage id>/<stage
> attempt>/executorSummary{code}
> This is a subtask for SPARK-23206. Please refer to the design doc for that
> ticket for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
