Hi all, The JSON version of the web UI is not officially supported; I don't believe this is documented anywhere.
The alternative is to set `spark.eventLog.enabled` to true before running your application. This will create JSON SparkListenerEvents with details about each task and stage as a log file. Then you can easily reconstruct the web UI after the application has exited. This is what the standalone Master and the History Server does, actually. For local mode, you can use the latter to generate your UI after the fact. (This is documented here: http://spark.apache.org/docs/latest/monitoring.html). -Andrew 2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A < oliver.ruebenac...@altisource.com>: > > > Hello, > > > > Thanks for the link – this is for standalone, though, and most URLs > don’t work for local. > > I will look into deploying as standalone on a single node for testing > and development. > > > > Best, Oliver > > > > *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] > *Sent:* Thursday, September 04, 2014 3:09 AM > *To:* Ruebenacker, Oliver A > *Cc:* Wonha Ryu; user@spark.apache.org > *Subject:* Re: Web UI > > > > Hi > > > > You can see this doc > <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security> > for all the available webUI ports. > > > > Yes there are ways to get the data metrics in Json format, One of them is > below: > > > > ** > > *http://webUI:8080/json/ <http://webUI:8080/json/>* Or simply > > ** > > *curl webUI:8080/json/* > > > > There are some PRs about it you can read it over here > https://github.com/apache/spark/pull/1682 > > > Thanks > > Best Regards > > > > On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > Interestingly, http://localhost:4040/metrics/json/ gives some numbers, > but only a few which never seem to change during the application’s lifetime. > > > > Either the web UI has some very strange limitations, or there are some > URLs yet to be discovered that do something interesting. > > > > Best, > > Oliver > > > > > > *From:* Wonha Ryu [mailto:wonha....@gmail.com] > *Sent:* Wednesday, September 03, 2014 4:27 PM > > > *To:* Ruebenacker, Oliver A > *Cc:* user@spark.apache.org > *Subject:* Re: Web UI > > > > Hey Oliver, > > > > IIRC there's no JSON endpoint for application web UI. They only exist for > cluster master and worker. > > > > - Wonha > > > > > > On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > Thanks for the help! But I tried starting with “–master local[4]” and > when I load http://localhost:4040/json I just get forwarded to > http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON. > > > > Best, > > Oliver > > > > > > *From:* Wonha Ryu [mailto:wonha....@gmail.com] > *Sent:* Wednesday, September 03, 2014 3:36 PM > *To:* Ruebenacker, Oliver A > *Cc:* user@spark.apache.org > *Subject:* Re: Web UI > > > > Hi Oliver, > > > > Spark standalone master and worker support '/json' endpoint in web UI, > which returns some of the information in JSON format. > > I wasn't able to find relevant documentation, though. > > > > - Wonha > > > > On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > What is included in the Spark web UI? What are the available URLs? Can > the information be obtained in a machine-readable way (e.g. JSON, XML, etc)? > > > > Thanks! > > > > Best, > > Oliver > > > > Oliver Ruebenacker | Solutions Architect > > > > Altisource™ > > 290 Congress St, 7th Floor | Boston, Massachusetts 02210 > > P: (617) 728-5582 | ext: 275585 > > oliver.ruebenac...@altisource.com | www.Altisource.com > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** >
