GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/19750
[SPARK-20650][core] Remove JobProgressListener. The only remaining use of this class was the SparkStatusTracker, which was modified to use the new status store. The test code to wait for executors was moved to TestUtils and now uses the SparkStatusTracker API. Indirectly, ConsoleProgressBar also uses this data. Because it has some lower latency requirements, a shortcut to efficiently get the active stages from the active listener was added to the AppStateStore. Now that all UI code goes through the status store to get its data, the FsHistoryProvider can be cleaned up to only replay event logs when needed - that is, when there is no pre-existing disk store for the application. As part of this change I also modified the streaming UI to read the needed data from the store, which was missed in the previous patch that made JobProgressListener redundant. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark SPARK-20650 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19750.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19750 ---- commit 406779bf05cbab7afd8c632ebb7035fb0f2cbd28 Author: Marcelo Vanzin <van...@cloudera.com> Date: 2017-03-29T18:18:23Z [SPARK-20650][core] Remove JobProgressListener. The only remaining use of this class was the SparkStatusTracker, which was modified to use the new status store. The test code to wait for executors was moved to TestUtils and now uses the SparkStatusTracker API. Indirectly, ConsoleProgressBar also uses this data. Because it has some lower latency requirements, a shortcut to efficiently get the active stages from the active listener was added to the AppStateStore. Now that all UI code goes through the status store to get its data, the FsHistoryProvider can be cleaned up to only replay event logs when needed - that is, when there is no pre-existing disk store for the application. As part of this change I also modified the streaming UI to read the needed data from the store, which was missed in the previous patch that made JobProgressListener redundant. ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org