Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi Sivakumaran Thanks for your very useful research. Apologies have been very busy. Let me read through and come back. Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Nguyen for the link. I installed Super Refresh as ADD on to Chrome. By default the refresh is stop until you set it to x seconds. However, the issue we have is that Spark UI comes with 6+ tabs and you have to repeat the process for each tab. However, that messes up the things. For example

Re: Is there anyway Spark UI is set to poll and refreshes itself

The simplest solution that I found: using an browser extension which do that for you :D. For example, if you are using Chrome, you can use this extension: https://chrome.google.com/webstore/detail/easy-auto-refresh/aabcgdmkeabbnleenpncegpcngjpnjkc/related?hl=en An other way, but a bit more

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi All, GitHub project SparkUIDashboard created here [image: Inline images 2] Let use put the show on the road :) Cheers Dr Mich Talebzadeh LinkedIn *

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi, There's no better way to start a project than...github it :-) Create a new project, clone it and do dzieła! (= go ahead in Polish). Jacek On 27 Aug 2016 10:42 a.m., "Sivakumaran S" wrote: > I would love to participate in developing a dashboard of some sort in lieu

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Sivakumaran I don't think we can use Zeppelin for this purpose. It is not a real time dashboard or can it be. I use it but much like Tableau with added Scala programming. Does anyone know of open source real time dashboards? Cheers Dr Mich Talebzadeh LinkedIn *

Re: Is there anyway Spark UI is set to poll and refreshes itself

Are we actually looking for a eal time dashboard of some sort for Spark UI interface? After all one can think a real time dashboard can do this! HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Jacek, I will have a look. I think it is long overdue. I mean we try to micro batch and stream everything below seconds but when it comes to help monitor basics we are still miles behind :( Cheers, Dr Mich Talebzadeh LinkedIn *

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi Mich, I don't think so. There is support for a UI page refresh but I haven't seen it in use. See StreamingPage [1] where it schedules refresh every 5 secs, i.e. Some(5000). In SparkUIUtils.headerSparkPage [2] there is refreshInterval but it's not used in any place in Spark. Time to fill an

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi you can take a look at: https://github.com/hammerlab/spree it's a bit outdated but maybe it's still possible to use with some more recent Spark version. M. 2016-08-25 11:55 GMT+02:00 Mich Talebzadeh : > Hi, > > This may be already there. > > A spark job opens up a

Is there anyway Spark UI is set to poll and refreshes itself

Hi, This may be already there. A spark job opens up a UI on port specified by --conf "spark.ui.port=${SP}" that defaults to 4040. However, on UI one needs to refresh the page to see the progress. Can this be polled so it is refreshed automatically Thanks Dr Mich Talebzadeh LinkedIn *