Hi Pierre,
I asked a similar question on this list about 6 weeks ago. Here is one
answer
<http://mail-archives.apache.org/mod_mbox/spark-user/201404.mbox/%3ccamjob8n3foaxd-dc5j57-n1oocwxefcg5chljwnut7qnreq...@mail.gmail.com%3E>
I got that is of particular note:
In the upcoming release of Spark 1.0 there will be a feature that
provides for exactly what you describe: capturing the information
displayed on the UI in JSON. More details will be provided in the
documentation, but for now, anything before 0.9.1 can only go through
JobLogger.scala, which outputs information in a somewhat arbitrary
format and will be deprecated soon. If you find this feature useful, you
can test it out by building the master branch of Spark yourself,
following the instructions in https://github.com/apache/spark/pull/42.
On 05/22/2014 08:51 AM, Pierre B wrote:
Is there a simple way to monitor the overall progress of an action using
SparkListener or anything else?
I see that one can name an RDD... Could that be used to determine which
action triggered a stage, ... ?
Thanks
Pierre
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
