Sounds like just what we need. For Hadoop we have progress bar to show the current status of the job . We like to do the same for spark. The yarn client only shows the percentage progress does show any text info.
Does your PR works for yarn mode ? Chester Sent from my iPhone > On May 23, 2014, at 1:17 AM, Mayur Rustagi <mayur.rust...@gmail.com> wrote: > > We have an internal patched version of Spark webUI which exports application > related data as Json. We use monitoring systems as well as alternate UI for > that json data for our specific application. Found it much cleaner. Can > provide 0.9.1 version. > Would submit as a pull request soon. > > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > >> On Fri, May 23, 2014 at 10:57 AM, Chester <chesterxgc...@yahoo.com> wrote: >> This is something we are interested as well. We are planning to investigate >> more on this. If someone has suggestions, we would love to hear. >> >> Chester >> >> Sent from my iPad >> >> On May 22, 2014, at 8:02 AM, Pierre B >> <pierre.borckm...@realimpactanalytics.com> wrote: >> >>> Hi Andy! >>> >>> Yes Spark UI provides a lot of interesting informations for debugging >>> purposes. >>> >>> Here I’m trying to integrate a simple progress monitoring in my app ui. >>> >>> I’m typically running a few “jobs” (or rather actions), and I’d like to be >>> able to display the progress of each of those in my ui. >>> >>> I don’t really see how i could do that using SparkListener for the moment … >>> >>> Thanks for your help! >>> >>> Cheers! >>> >>> >>> >>> >>> Pierre Borckmans >>> Software team >>> >>> RealImpact Analytics | Brussels Office >>> www.realimpactanalytics.com | [hidden email] >>> >>> FR +32 485 91 87 31 | Skype pierre.borckmans >>> >>> >>> >>> >>> >>> >>>> On 22 May 2014, at 16:58, andy petrella [via Apache Spark User List] >>>> <[hidden email]> wrote: >>>> >>>> SparkListener offers good stuffs. >>>> But I also completed it with another metrics stuffs on my own that use >>>> Akka to aggregate metrics from anywhere I'd like to collect them (without >>>> any deps on ganglia yet on Codahale). >>>> However, this was useful to gather some custom metrics (from within the >>>> tasks then) not really to collect overall monitoring information about the >>>> spark thingies themselves. >>>> For that Spark UI offers already a pretty good insight no? >>>> >>>> Cheers, >>>> >>>> aℕdy ℙetrella >>>> about.me/noootsab >>>> >>>> >>>> >>>> >>>>> On Thu, May 22, 2014 at 4:51 PM, Pierre B <<a >>>>> href="x-msg://7/user/SendEmail.jtp?type=node&node=6258&i=0" >>>>> target="_top" rel="nofollow" link="external">[hidden email]> wrote: >>>>> Is there a simple way to monitor the overall progress of an action using >>>>> SparkListener or anything else? >>>>> >>>>> I see that one can name an RDD... Could that be used to determine which >>>>> action triggered a stage, ... ? >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Pierre >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> >>>> >>>> If you reply to this email, your message will be added to the discussion >>>> below: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html >>>> To unsubscribe from Use SparkListener to get overall progress of an >>>> action, click here. >>>> NAML >>> >>> >>> View this message in context: Re: Use SparkListener to get overall progress >>> of an action >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >
