GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/19750
[SPARK-20650][core] Remove JobProgressListener.
The only remaining use of this class was the SparkStatusTracker, which
was modified to use the new status store. The test code to wait for
executors was moved to TestUtils and now uses the SparkStatusTracker API.
Indirectly, ConsoleProgressBar also uses this data. Because it has
some lower latency requirements, a shortcut to efficiently get the
active stages from the active listener was added to the AppStateStore.
Now that all UI code goes through the status store to get its data,
the FsHistoryProvider can be cleaned up to only replay event logs
when needed - that is, when there is no pre-existing disk store for
the application.
As part of this change I also modified the streaming UI to read the needed
data from the store, which was missed in the previous patch that made
JobProgressListener redundant.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-20650
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19750
----
commit 406779bf05cbab7afd8c632ebb7035fb0f2cbd28
Author: Marcelo Vanzin <van...@cloudera.com>
Date: 2017-03-29T18:18:23Z
[SPARK-20650][core] Remove JobProgressListener.
The only remaining use of this class was the SparkStatusTracker, which
was modified to use the new status store. The test code to wait for
executors was moved to TestUtils and now uses the SparkStatusTracker API.
Indirectly, ConsoleProgressBar also uses this data. Because it has
some lower latency requirements, a shortcut to efficiently get the
active stages from the active listener was added to the AppStateStore.
Now that all UI code goes through the status store to get its data,
the FsHistoryProvider can be cleaned up to only replay event logs
when needed - that is, when there is no pre-existing disk store for
the application.
As part of this change I also modified the streaming UI to read the needed
data from the store, which was missed in the previous patch that made
JobProgressListener redundant.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
