We have an internal patched version of Spark webUI which exports application related data as Json. We use monitoring systems as well as alternate UI for that json data for our specific application. Found it much cleaner. Can provide 0.9.1 version. Would submit as a pull request soon.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Fri, May 23, 2014 at 10:57 AM, Chester <chesterxgc...@yahoo.com> wrote: > This is something we are interested as well. We are planning to > investigate more on this. If someone has suggestions, we would love to hear. > > Chester > > Sent from my iPad > > On May 22, 2014, at 8:02 AM, Pierre B < > pierre.borckm...@realimpactanalytics.com> wrote: > > Hi Andy! > > Yes Spark UI provides a lot of interesting informations for debugging > purposes. > > Here I’m trying to integrate a simple progress monitoring in my app ui. > > I’m typically running a few “jobs” (or rather actions), and I’d like to be > able to display the progress of each of those in my ui. > > I don’t really see how i could do that using SparkListener for the moment … > > Thanks for your help! > > Cheers! > > > > > *Pierre Borckmans* > Software team > > *Real**Impact* Analytics *| *Brussels Office > <http://www.realimpactanalytics.com/>www.realimpactanalytics.com *| *[hidden > email] <http://user/SendEmail.jtp?type=node&node=6259&i=0> > > *FR *+32 485 91 87 31 *| **Skype* pierre.borckmans > > > > > > > On 22 May 2014, at 16:58, andy petrella [via Apache Spark User List] <[hidden > email] <http://user/SendEmail.jtp?type=node&node=6259&i=1>> wrote: > > SparkListener offers good stuffs. > But I also completed it with another metrics stuffs on my own that use > Akka to aggregate metrics from anywhere I'd like to collect them (without > any deps on ganglia yet on Codahale). > However, this was useful to gather some custom metrics (from within the > tasks then) not really to collect overall monitoring information about the > spark thingies themselves. > For that Spark UI offers already a pretty good insight no? > > Cheers, > > aℕdy ℙetrella > about.me/noootsab > [image: aℕdy ℙetrella on about.me] > > <http://about.me/noootsab> > > > On Thu, May 22, 2014 at 4:51 PM, Pierre B <<a > href="x-msg://7/user/SendEmail.jtp?type=node&node=6258&i=0" > target="_top" rel="nofollow" link="external">[hidden email]> wrote: > >> Is there a simple way to monitor the overall progress of an action using >> SparkListener or anything else? >> >> I see that one can name an RDD... Could that be used to determine which >> action triggered a stage, ... ? >> >> >> Thanks >> >> Pierre >> >> >> >> -- >> View this message in context: >> <http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html> >> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >> Sent from the Apache Spark User List mailing list archive at >> <http://Nabble.com>Nabble.com. >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > <http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html> > http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html > To unsubscribe from Use SparkListener to get overall progress of an > action, click here. > NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > ------------------------------ > View this message in context: Re: Use SparkListener to get overall > progress of an > action<http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6259.html> > Sent from the Apache Spark User List mailing list > archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at > Nabble.com. > >
