[spark] branch master updated: [SPARK-30300][SQL][WEB-UI] Fix updating the UI max value string when driver updates the same metric id as the tasks

[tgraves]

This is an automated email from the ASF dual-hosted git repository.

tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 0d2ef3a  [SPARK-30300][SQL][WEB-UI] Fix updating the UI max value 
string when driver updates the same metric id as the tasks
0d2ef3a is described below

commit 0d2ef3ae2b2d0b66f763d6bb2e490a667c83f9f2
Author: Niranjan Artal <nar...@nvidia.com>
AuthorDate: Fri Dec 20 07:29:28 2019 -0600

    [SPARK-30300][SQL][WEB-UI] Fix updating the UI max value string when driver 
updates the same metric id as the tasks
    
    ### What changes were proposed in this pull request?
    
    In this PR, For a given metrics id we are checking if the driver side 
accumulator's value is greater than max of all stages value. If it's true, then 
we are removing that entry from the Hashmap. By doing this, for this metrics, 
"driver" would be displayed on the UI(As the driver would have the maximum 
value)
    
    ### Why are the changes needed?
    
    This PR fixes https://issues.apache.org/jira/browse/SPARK-30300. Currently 
driver's metric value is not compared while caluculating the max.
    
    ### Does this PR introduce any user-facing change?
    
    For the metrics where driver's value is greater than max of all stages, 
this is the change.
    Previous : (min, median, max (stageId 0( attemptId 1): taskId 2))
    Now:   (min, median, max (driver))
    
    ### How was this patch tested?
    
    Ran unit tests.
    
    Closes #26941 from nartal1/SPARK-30300.
    
    Authored-by: Niranjan Artal <nar...@nvidia.com>
    Signed-off-by: Thomas Graves <tgra...@apache.org>
---
 .../org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala    | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
index 64d2f33..d5bb36e 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
@@ -237,6 +237,12 @@ class SQLAppStatusListener(
       if (metricTypes.contains(id)) {
         val prev = allMetrics.getOrElse(id, null)
         val updated = if (prev != null) {
+          // If the driver updates same metrics as tasks and has higher value 
then remove
+          // that entry from maxMetricsFromAllStage. This would make 
stringValue function default
+          // to "driver" that would be displayed on UI.
+          if (maxMetricsFromAllStages.contains(id) && value > 
maxMetricsFromAllStages(id)(0)) {
+            maxMetricsFromAllStages.remove(id)
+          }
           val _copy = Arrays.copyOf(prev, prev.length + 1)
           _copy(prev.length) = value
           _copy


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org