Re: Use SparkListener to get overall progress of an action

Thanks Philip, I don’t want to go the JobLogger way (too hacky ;) ) In version 1.0, if i’m not mistaken, you can even do what I’m asking for, since they removed the “private” for TaskInfo and such and replaced it with the “@DeveloperApi” annotation. I was looking for a simple way to do this in

Re: Use SparkListener to get overall progress of an action

Hi Pierre, I asked a similar question on this list about 6 weeks ago. Here is one answer I got that is of particular note: In the upcoming release of Sp

Re: Use SparkListener to get overall progress of an action

/SendEmail.jtp?type=node&node=6258&i=0" >>>> target="_top" rel="nofollow" link="external">[hidden email]> wrote: >>>> Is there a simple way to monitor the overall progress of an action using >>>> SparkListener

Re: Use SparkListener to get overall progress of an action

gt;> -- >> If you reply to this email, your message will be added to the >> discussion below: >> <http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html> >> http://apache-spark-user-list.1001560.n3.nabble.co

Re: Use SparkListener to get overall progress of an action

can name an RDD... Could that be used to determine which >>>>> action triggered a stage, ... ? >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Pierre >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> >>>> >>>> If you reply to this email, your message will be added to the discussion >>>> below: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html >>>> To unsubscribe from Use SparkListener to get overall progress of an >>>> action, click here. >>>> NAML >>> >>> >>> View this message in context: Re: Use SparkListener to get overall progress >>> of an action >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Use SparkListener to get overall progress of an action

gt; >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> >>> >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html >>> To unsubscribe from Use SparkListener to get overall progress of an action, >>> click here. >>> NAML >> >> >> View this message in context: Re: Use SparkListener to get overall progress >> of an action >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Use SparkListener to get overall progress of an action

258.html > To unsubscribe from Use SparkListener to get overall progress of an > action, click here. > NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-na

Re: Use SparkListener to get overall progress of an action

apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> >> >> If you reply to this email, your message will be added to th

Re: Use SparkListener to get overall progress of an action

p6258.html > To unsubscribe from Use SparkListener to get overall progress of an > action, click here. > NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicN

Re: Use SparkListener to get overall progress of an action

Hi Andy! Yes Spark UI provides a lot of interesting informations for debugging purposes. Here I’m trying to integrate a simple progress monitoring in my app ui. I’m typically running a few “jobs” (or rather actions), and I’d like to be able to display the progress of each of those in my ui. I

Re: Use SparkListener to get overall progress of an action

SparkListener offers good stuffs. But I also completed it with another metrics stuffs on my own that use Akka to aggregate metrics from anywhere I'd like to collect them (without any deps on ganglia yet on Codahale). However, this was useful to gather some custom metrics (from within the tasks then