Hi All, GitHub project SparkUIDashboard created here <https://github.com/search?utf8=%E2%9C%93&q=sparkuidashboard&ref=simplesearch>
[image: Inline images 2] Let use put the show on the road :) Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 27 August 2016 at 12:53, Sivakumaran S <siva.kuma...@icloud.com> wrote: > Hi Mich, > > Unlikely that we can use Zeppelin for dynamic, real time update > visualisation. It makes nice, static visuals. > > I was thinking more on the lines of http://dashingdemo. > herokuapp.com/sample > > The library is http://dashing.io > > There are more widgets that can be used https://github.com/ > Shopify/dashing/wiki/Additional-Widgets > > The Spark UI is functional, but I am looking forward to some aesthetics > and high level picture of the process. Using Websockets, the dashboard can > be updated real time without the need of refreshing the page. > > Regards, > > Sivakumaran S > > > On 27-Aug-2016, at 10:10 AM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Thanks Sivakumaran > > I don't think we can use Zeppelin for this purpose. It is not a real time > dashboard or can it be. I use it but much like Tableau with added Scala > programming. > > Does anyone know of open source real time dashboards? > > > Cheers > > Dr Mich Talebzadeh > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > http://talebzadehmich.wordpress.com > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 27 August 2016 at 09:42, Sivakumaran S <siva.kuma...@icloud.com> wrote: > >> I would love to participate in developing a dashboard of some sort in >> lieu (or at least complement it) of Spark UI . >> >> Regards, >> >> Sivakumaran S >> >> On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Are we actually looking for a eal time dashboard of some sort for Spark >> UI interface? >> >> After all one can think a real time dashboard can do this! >> >> HTH >> >> >> >> Dr Mich Talebzadeh >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> http://talebzadehmich.wordpress.com >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 26 August 2016 at 23:38, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Thanks Jacek, >> >> I will have a look. I think it is long overdue. >> >> I mean we try to micro batch and stream everything below seconds but when >> it comes to help monitor basics we are still miles behind :( >> >> Cheers, >> >> Dr Mich Talebzadeh >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> http://talebzadehmich.wordpress.com >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 26 August 2016 at 23:21, Jacek Laskowski <ja...@japila.pl> wrote: >> >> Hi Mich, >> >> I don't think so. There is support for a UI page refresh but I haven't >> seen it in use. >> >> See StreamingPage [1] where it schedules refresh every 5 secs, i.e. >> Some(5000). In SparkUIUtils.headerSparkPage [2] there is >> refreshInterval but it's not used in any place in Spark. >> >> Time to fill an JIRA issue? >> >> What about REST API and httpie updating regularly [3]? Perhaps Metrics >> with ConsoleSink [4]? >> >> [1] https://github.com/apache/spark/blob/master/streaming/src/ma >> in/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158 >> [2] https://github.com/apache/spark/blob/master/core/src/main/sc >> ala/org/apache/spark/ui/UIUtils.scala#L202 >> [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api >> [4] http://spark.apache.org/docs/latest/monitoring.html#metrics >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> >> On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh >> <mich.talebza...@gmail.com> wrote: >> > Hi, >> > >> > This may be already there. >> > >> > A spark job opens up a UI on port specified by --conf >> "spark.ui.port=${SP}" >> > that defaults to 4040. >> > >> > However, on UI one needs to refresh the page to see the progress. >> > >> > Can this be polled so it is refreshed automatically >> > >> > Thanks >> > >> > >> > Dr Mich Talebzadeh >> > >> > >> > >> > LinkedIn >> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJ >> d6zP6AcPCCdOABUrV8Pw >> > >> > >> > >> > http://talebzadehmich.wordpress.com >> > >> > >> > Disclaimer: Use it at your own risk. Any and all responsibility for any >> > loss, damage or destruction of data or any other property which may >> arise >> > from relying on this email's technical content is explicitly >> disclaimed. The >> > author will in no case be liable for any monetary damages arising from >> such >> > loss, damage or destruction. >> > >> > >> >> >> >> >> > >
