GitHub user shahidki31 opened a pull request: https://github.com/apache/spark/pull/22549
[SPARK-25533][CORE][WEBUI]AppSummary should hold the information about succeeded Jobs and completed stages only ## What changes were proposed in this pull request? Currently, In the spark UI, when there are failed jobs or failed stages, display message for the completed jobs and completed stages are not consistent with the previous versions of spark. Reason is because, AppSummary holds the information about all the jobs and stages. But, In the below code, it checks against the completedJobs and completedStages. So, AppSummary should hold only successful jobs and stages. https://github.com/apache/spark/blob/66d29870c09e6050dd846336e596faaa8b0d14ad/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala#L306 https://github.com/apache/spark/blob/66d29870c09e6050dd846336e596faaa8b0d14ad/core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala#L119 So, we should keep only completed jobs and stage information in the AppSummary, to make it consistent with Spark2.2 ## How was this patch tested? Test steps: bin/spark-shell ``` sc.parallelize(1 to 5, 5).collect() sc.parallelize(1 to 5, 2).map{ x => throw new RuntimeException("Fail")}.collect() ``` **Before fix:** ![screenshot from 2018-09-26 03-24-53](https://user-images.githubusercontent.com/23054875/46045669-f60bcd80-c13b-11e8-9aa6-a2e5a2038dba.png) ![screenshot from 2018-09-26 03-25-08](https://user-images.githubusercontent.com/23054875/46045699-0ae86100-c13c-11e8-94e5-ad35944c7615.png) **After fix:** ![screenshot from 2018-09-26 03-16-14](https://user-images.githubusercontent.com/23054875/46045636-d83e6880-c13b-11e8-98df-f49d15c18958.png) ![screenshot from 2018-09-26 03-16-28](https://user-images.githubusercontent.com/23054875/46045645-e1c7d080-c13b-11e8-8c9c-d32e1f663356.png) You can merge this pull request into a Git repository by running: $ git pull https://github.com/shahidki31/spark SPARK-25533 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22549.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22549 ---- commit a006b74829a857086d705a25c4bda866b47a1a2a Author: Shahid <shahidki31@...> Date: 2018-09-25T19:24:59Z [SPARK-25533]AppSummary should hold the information about completed Jobs and Succeeded Task only commit 7fe78123feb1b0b42f31765dcb2aae8bb9360cde Author: Shahid <shahidki31@...> Date: 2018-09-25T19:29:27Z [SPARK-25533]AppSummary should hold the information about succeeded Jobs and completed stages only ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org