[GitHub] spark pull request #20408: [SPARK-23189][Core][UI] Reflect stage level black...

[attilapiros]

GitHub user attilapiros opened a pull request:

    https://github.com/apache/spark/pull/20408

    [SPARK-23189][Core][UI] Reflect stage level blacklisting on executor tab 

    ## What changes were proposed in this pull request?
    
    The purpose of this PR to reflect the stage level blacklisting on the 
executor tab for the currently active stages.
    
    After this change in the executor tab at the Status column one of the 
following label will be:
    
    - "Blacklisted" when the executor is blacklisted application level (old 
flag)
    - "Dead" when the executor is not Blacklisted and not Active
    - "Blacklisted in Stages: [...]" when the executor is Active but the 
blacklistedInStages set is not empty where within the [] coma separated active 
stageIDs are listed.
    - "Active" when the executor is Active and blacklistedInStages set is empty
    
    ## How was this patch tested?
    
    Both with unit tests and manually.
    
    #### Manual test
    
    Spark was started as:
    
    ```bash
     bin/spark-shell --master "local-cluster[2,1,1024]" --conf 
"spark.blacklist.enabled=true" --conf 
"spark.blacklist.stage.maxFailedTasksPerExecutor=1" --conf 
"spark.blacklist.application.maxFailedTasksPerExecutor=10" 
    ```
    
    And the job was:
    ```scala
    import org.apache.spark.SparkEnv
    
    val pairs = sc.parallelize(1 to 10000, 10).map { x =>
      if (SparkEnv.get.executorId.toInt == 0) throw new RuntimeException("Bad 
executor")
      else  {
        Thread.sleep(10)
        (x % 10, x)
      }
    }
      
    val all = pairs.cogroup(pairs)
      
    all.collect()
    ```
    
    UI screenshots about the running:
    
    - One executor is blacklisted in the two stages: 
    
    ![One executor is blacklisted in two 
stages](https://issues.apache.org/jira/secure/attachment/12907862/multiple_stages_1.png)
    
    - One stage completes the other one is still running: 
    
    ![One stage completes the other is still 
running](https://issues.apache.org/jira/secure/attachment/12907862/multiple_stages_2.png)
    
    - Both stages are completed: 
    
    ![Both stages are 
completed](https://issues.apache.org/jira/secure/attachment/12907862/multiple_stages_3.png)
    
    ### Unit tests
    
    In AppStatusListenerSuite.scala both node blacklisting for stage and 
executor blacklisting for stage are tested.  
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/attilapiros/spark SPARK-23189

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20408
    
----
commit 4f5176be0d3da7794d20895d8a0bfce16d6b8e5c
Author: “attilapiros” <piros.attila.zsolt@...>
Date:   2018-01-25T22:17:11Z

    inital version

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org