[spark] branch master updated: [SPARK-36038][CORE] Speculation metrics summary at stage level
This is an automated email from the ASF dual-hosted git repository. sarutak pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 4158d35 [SPARK-36038][CORE] Speculation metrics summary at stage level 4158d35 is described below commit 4158d3544030058a23c7741d4283213a56ae7cc7 Author: Thejdeep Gudivada AuthorDate: Mon Dec 13 09:58:27 2021 +0900 [SPARK-36038][CORE] Speculation metrics summary at stage level ### What changes were proposed in this pull request? Currently there are no speculation metrics available for Spark either at application/job/stage level. This PR is to add some basic speculation metrics for a stage when speculation execution is enabled. This is similar to the existing stage level metrics tracking numTotal (total number of speculated tasks), numCompleted (total number of successful speculated tasks), numFailed (total number of failed speculated tasks), numKilled (total number of killed speculated tasks) etc. With this new set of metrics, it helps further understanding speculative execution feature in the context of the application and also helps in further tuning the speculative execution config knobs. ### Why are the changes needed? Additional metrics for speculative execution. ### Does this PR introduce _any_ user-facing change? Yes, Stages Page in SHS UI will have an additional table for speculation metrics, if present. ### How was this patch tested? Unit tests added and also tested on our internal platform. Absence of speculation metrics : ![Screen Shot 2021-11-15 at 10 12 23 AM](https://user-images.githubusercontent.com/1708757/141836847-b0768265-35eb-4963-b83a-8b785318e82d.png) Presence of speculation metrics : ![Screen Shot 2021-11-15 at 10 11 52 AM](https://user-images.githubusercontent.com/1708757/141836945-1078bfcb-1c07-44db-9f33-5dbd09ece6f2.png) Closes #34607 from thejdeep/SPARK-36038. Lead-authored-by: Thejdeep Gudivada Co-authored-by: Ron Hu Co-authored-by: Venkata krishnan Sowrirajan Signed-off-by: Kousuke Saruta --- .../org/apache/spark/ui/static/stagepage.js| 32 ++ .../spark/ui/static/stagespage-template.html | 15 + .../resources/org/apache/spark/ui/static/webui.css | 10 + .../apache/spark/status/AppStatusListener.scala| 14 + .../org/apache/spark/status/AppStatusStore.scala | 11 + .../scala/org/apache/spark/status/LiveEntity.scala | 26 ++ .../scala/org/apache/spark/status/api/v1/api.scala | 8 + .../scala/org/apache/spark/status/storeTypes.scala | 12 + .../scala/org/apache/spark/ui/jobs/JobPage.scala | 1 + .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 15 + .../minEndDate_app_list_json_expectation.json | 15 + ...stage_with_speculation_summary_expectation.json | 507 + .../spark-events/application_1628109047826_1317105 | 52 +++ .../spark/deploy/history/HistoryServerSuite.scala | 5 +- .../spark/status/AppStatusListenerSuite.scala | 10 + .../apache/spark/status/AppStatusStoreSuite.scala | 51 ++- .../scala/org/apache/spark/ui/StagePageSuite.scala | 1 + dev/.rat-excludes | 3 +- 21 files changed, 830 insertions(+), 18 deletions(-) diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js index 584e1a7..a5955f3 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js +++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js @@ -652,6 +652,38 @@ $(document).ready(function () { executorSummaryTableSelector.column(14).visible(dataToShow.showBytesSpilledData); }); +// Prepare data for speculation metrics +$("#speculationSummaryTitle").hide(); +$("#speculationSummary").hide(); +var speculationSummaryInfo = responseBody.speculationSummary; +var speculationData; +if(speculationSummaryInfo) { + speculationData = [[ +speculationSummaryInfo.numTasks, +speculationSummaryInfo.numActiveTasks, +speculationSummaryInfo.numCompletedTasks, +speculationSummaryInfo.numFailedTasks, +speculationSummaryInfo.numKilledTasks + ]]; + if (speculationSummaryInfo.numTasks > 0) { +// Show speculationSummary if there is atleast one speculated task that ran +$("#speculationSummaryTitle").show(); +$("#speculationSummary").show(); + } +} +var speculationMetricsTableConf = { + "data":
[spark] branch master updated: [SPARK-36038][CORE] Speculation metrics summary at stage level
This is an automated email from the ASF dual-hosted git repository. sarutak pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 73747ec [SPARK-36038][CORE] Speculation metrics summary at stage level 73747ec is described below commit 73747ecb970595d49c478b0eb65f5132c8b0bf02 Author: Venkata krishnan Sowrirajan AuthorDate: Fri Oct 1 16:59:29 2021 +0900 [SPARK-36038][CORE] Speculation metrics summary at stage level ### What changes were proposed in this pull request? Currently there are no speculation metrics available for Spark either at application/job/stage level. This PR is to add some basic speculation metrics for a stage when speculation execution is enabled. This is similar to the existing stage level metrics tracking numTotal (total number of speculated tasks), numCompleted (total number of successful speculated tasks), numFailed (total number of failed speculated tasks), numKilled (total number of killed speculated tasks) etc. With this new set of metrics, it helps further understanding speculative execution feature in the context of the application and also helps in further tuning the speculative execution config knobs. Screenshot of Spark UI with speculation summary: ![Screen Shot 2021-09-22 at 12 12 20 PM](https://user-images.githubusercontent.com/8871522/135321311-db7699ad-f1ae-4729-afea-d1e2c4e86103.png) Screenshot of Spark UI with API output: ![Screen Shot 2021-09-22 at 12 10 37 PM](https://user-images.githubusercontent.com/8871522/135321486-4dbb7a67-5580-47f8-bccf-81c758c2e988.png) ### Why are the changes needed? Additional metrics for speculative execution. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests added and also deployed in our internal platform for quite some time now. Lead-authored by: Venkata krishnan Sowrirajan Co-authored by: Ron Hu Co-authored by: Thejdeep Gudivada Closes #33253 from venkata91/speculation-metrics. Authored-by: Venkata krishnan Sowrirajan Signed-off-by: Kousuke Saruta --- .../org/apache/spark/ui/static/stagepage.js| 29 ++ .../spark/ui/static/stagespage-template.html | 15 + .../resources/org/apache/spark/ui/static/webui.css | 10 + .../apache/spark/status/AppStatusListener.scala| 19 + .../org/apache/spark/status/AppStatusStore.scala | 11 + .../scala/org/apache/spark/status/LiveEntity.scala | 26 ++ .../scala/org/apache/spark/status/api/v1/api.scala | 8 + .../scala/org/apache/spark/status/storeTypes.scala | 14 + .../scala/org/apache/spark/ui/jobs/JobPage.scala | 1 + .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 15 + .../minEndDate_app_list_json_expectation.json | 15 + ...stage_with_speculation_summary_expectation.json | 507 + .../spark-events/application_1628109047826_1317105 | 52 +++ .../spark/deploy/history/HistoryServerSuite.scala | 5 +- .../spark/status/AppStatusListenerSuite.scala | 10 + .../apache/spark/status/AppStatusStoreSuite.scala | 57 ++- .../scala/org/apache/spark/ui/StagePageSuite.scala | 1 + dev/.rat-excludes | 3 +- 21 files changed, 840 insertions(+), 18 deletions(-) diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js index db1a148..595635a 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js +++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js @@ -652,6 +652,35 @@ $(document).ready(function () { executorSummaryTableSelector.column(14).visible(dataToShow.showBytesSpilledData); }); +// Prepare data for speculation metrics +$("#speculationSummaryTitle").hide() +$("#speculationSummary").hide() +var speculationSummaryInfo = responseBody.speculationSummary; +var speculationData = [[ + speculationSummaryInfo.numTasks, + speculationSummaryInfo.numActiveTasks, + speculationSummaryInfo.numCompletedTasks, + speculationSummaryInfo.numFailedTasks, + speculationSummaryInfo.numKilledTasks +]]; +if (speculationSummaryInfo.numTasks > 0) { + // Show speculationSummary if there is atleast one speculated task ran + $("#speculationSummaryTitle").show() + $("#speculationSummary").show() +} +var speculationMetricsTableConf = { + "data": speculationData, + "paging": false, + "searching": false,