Hi Sivakumaran Thanks for your very useful research. Apologies have been very busy. Let me read through and come back.
Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 5 September 2016 at 10:21, Sivakumaran S <siva.kuma...@icloud.com> wrote: > Hi Mich, > > Here is a summary of what I have so far understood (May require > correction): > > *Facts*. > 1. Spark has an internal web (http) server based on Jetty ( > http://www.eclipse.org/jetty/) that serves a ‘read-only’ UI on port 4040. > Additional SCs are displayed in sequentially increased port numbers (4041, > 4042 and so on) > 2. Spark has a ReST API for job submissions. > 3. The current UI has seven tabs: Jobs, Stages, Storage, Environment, > Executors, Streaming and SQL. > 4. Some tabs has further links pointing to greater depth of information > about the job. > 5. Changing Spark source code is not an option for building a dashboard > because most users use the compiled binaries or custom solutions provided > by cloud providers. > 6. SparkListener can be used to monitor only from within the job > submitted and is therefore job-specific. > > *Assumptions* > 1. There is no API (ReST or otherwise) for retrieving statistics or > information about current jobs. > 2. There is no way of automatically refreshing the information in these > tabs. > > *Proposed Solution to build a dashboard* > 1. HTTP is stateless. Every call to the web server gives the current > snapshot of information about the Spark job. > 2. Python can be used to write this software. > 3. So this proposed software sits between the Spark Server and the > browser. It repetitively (say, every 5 seconds) gets data (the HTML pages) > from port 4040 and after parsing the required information from these HTML > pages, updates the dashboard using Websocket. A light weight web server > framework will also be required for hosting the dashboard. > 4. Identifying what information is most important to be displayed (Some > experienced Spark users can provide inputs) is important. > > > > If the folks at Spark provide a ReST API for retrieving important > information about running jobs, this would be far more simpler. > > Regards, > > Sivakumaran S > > > > On 27-Aug-2016, at 3:59 PM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Thanks Nguyen for the link. > > I installed Super Refresh as ADD on to Chrome. By default the refresh is > stop until you set it to x seconds. However, the issue we have is that > Spark UI comes with 6+ tabs and you have to repeat the process for each tab. > > However, that messes up the things. For example if I choose to refresh > "Executors" tab every 2 seconds and decide to refresh "Stages" tab, then > there is a race condition whereas you are thrown back to the last refresh > page which is not really what one wants. > > Ideally one wants the Spark UI page identified by host:port to be the > driver and every other tab underneath say host:port/Stages to be refreshed > once we open that tab and stay there. If I go back to say SQL tab, I like > to see that refreshed ever n seconds. > > I hope this makes sense. > > > > Dr Mich Talebzadeh > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > http://talebzadehmich.wordpress.com > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 27 August 2016 at 15:01, nguyen duc Tuan <newvalu...@gmail.com> wrote: > >> The simplest solution that I found: using an browser extension which do >> that for you :D. For example, if you are using Chrome, you can use this >> extension: https://chrome.google.com/webstore/detail/easy-auto-refresh/ >> aabcgdmkeabbnleenpncegpcngjpnjkc/related?hl=en >> An other way, but a bit more manually using javascript: start with a >> window, you will create a child window with your target url. The parent >> window will refresh that child window for you. Due to same-original >> pollicy, you should set parent url to the same url as your target url. Try >> this in your web console: >> wi = window.open("your target url") >> var timeInMinis = 2000 >> setInterval(function(){ wi.location.reload();}, timeInMinis) >> Hope this help. >> >> 2016-08-27 20:17 GMT+07:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >> >>> Hi All, >>> >>> GitHub project SparkUIDashboard created here >>> <https://github.com/search?utf8=%E2%9C%93&q=sparkuidashboard&ref=simplesearch> >>> >>> >>> >>> >>> <image.png> >>> Let use put the show on the road :) >>> >>> Cheers >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 27 August 2016 at 12:53, Sivakumaran S <siva.kuma...@icloud.com> >>> wrote: >>> >>>> Hi Mich, >>>> >>>> Unlikely that we can use Zeppelin for dynamic, real time update >>>> visualisation. It makes nice, static visuals. >>>> >>>> I was thinking more on the lines of http://dashingdemo.herokuap >>>> p.com/sample >>>> >>>> The library is http://dashing.io >>>> >>>> There are more widgets that can be used https://github.com/Shopif >>>> y/dashing/wiki/Additional-Widgets >>>> >>>> The Spark UI is functional, but I am looking forward to some aesthetics >>>> and high level picture of the process. Using Websockets, the dashboard can >>>> be updated real time without the need of refreshing the page. >>>> >>>> Regards, >>>> >>>> Sivakumaran S >>>> >>>> >>>> On 27-Aug-2016, at 10:10 AM, Mich Talebzadeh <mich.talebza...@gmail.com> >>>> wrote: >>>> >>>> Thanks Sivakumaran >>>> >>>> I don't think we can use Zeppelin for this purpose. It is not a real >>>> time dashboard or can it be. I use it but much like Tableau with added >>>> Scala programming. >>>> >>>> Does anyone know of open source real time dashboards? >>>> >>>> >>>> Cheers >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> On 27 August 2016 at 09:42, Sivakumaran S <siva.kuma...@icloud.com> >>>> wrote: >>>> >>>>> I would love to participate in developing a dashboard of some sort in >>>>> lieu (or at least complement it) of Spark UI . >>>>> >>>>> Regards, >>>>> >>>>> Sivakumaran S >>>>> >>>>> On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> >>>>> wrote: >>>>> >>>>> Are we actually looking for a eal time dashboard of some sort for >>>>> Spark UI interface? >>>>> >>>>> After all one can think a real time dashboard can do this! >>>>> >>>>> HTH >>>>> >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 26 August 2016 at 23:38, Mich Talebzadeh <mich.talebza...@gmail.com >>>>> > wrote: >>>>> >>>>> Thanks Jacek, >>>>> >>>>> I will have a look. I think it is long overdue. >>>>> >>>>> I mean we try to micro batch and stream everything below seconds but >>>>> when it comes to help monitor basics we are still miles behind :( >>>>> >>>>> Cheers, >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> On 26 August 2016 at 23:21, Jacek Laskowski <ja...@japila.pl> wrote: >>>>> >>>>> Hi Mich, >>>>> >>>>> I don't think so. There is support for a UI page refresh but I haven't >>>>> seen it in use. >>>>> >>>>> See StreamingPage [1] where it schedules refresh every 5 secs, i.e. >>>>> Some(5000). In SparkUIUtils.headerSparkPage [2] there is >>>>> refreshInterval but it's not used in any place in Spark. >>>>> >>>>> Time to fill an JIRA issue? >>>>> >>>>> What about REST API and httpie updating regularly [3]? Perhaps Metrics >>>>> with ConsoleSink [4]? >>>>> >>>>> [1] https://github.com/apache/spark/blob/master/streaming/src/ma >>>>> in/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L158 >>>>> [2] https://github.com/apache/spark/blob/master/core/src/main/sc >>>>> ala/org/apache/spark/ui/UIUtils.scala#L202 >>>>> [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api >>>>> [4] http://spark.apache.org/docs/latest/monitoring.html#metrics >>>>> >>>>> Pozdrawiam, >>>>> Jacek Laskowski >>>>> ---- >>>>> https://medium.com/@jaceklaskowski/ >>>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >>>>> Follow me at https://twitter.com/jaceklaskowski >>>>> >>>>> >>>>> On Thu, Aug 25, 2016 at 11:55 AM, Mich Talebzadeh >>>>> <mich.talebza...@gmail.com> wrote: >>>>> > Hi, >>>>> > >>>>> > This may be already there. >>>>> > >>>>> > A spark job opens up a UI on port specified by --conf >>>>> "spark.ui.port=${SP}" >>>>> > that defaults to 4040. >>>>> > >>>>> > However, on UI one needs to refresh the page to see the progress. >>>>> > >>>>> > Can this be polled so it is refreshed automatically >>>>> > >>>>> > Thanks >>>>> > >>>>> > >>>>> > Dr Mich Talebzadeh >>>>> > >>>>> > >>>>> > >>>>> > LinkedIn >>>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJ >>>>> d6zP6AcPCCdOABUrV8Pw >>>>> > >>>>> > >>>>> > >>>>> > http://talebzadehmich.wordpress.com >>>>> > >>>>> > >>>>> > Disclaimer: Use it at your own risk. Any and all responsibility for >>>>> any >>>>> > loss, damage or destruction of data or any other property which may >>>>> arise >>>>> > from relying on this email's technical content is explicitly >>>>> disclaimed. The >>>>> > author will in no case be liable for any monetary damages arising >>>>> from such >>>>> > loss, damage or destruction. >>>>> > >>>>> > >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >> > >
