Re: Web UI

2014-09-03 Thread Wonha Ryu 
Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI,
which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha


On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
oliver.ruebenac...@altisource.com> wrote:

>
>
>  Hello,
>
>
>
>   What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
>   Thanks!
>
>
>
>  Best,
>
>  Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> oliver.ruebenac...@altisource.com | www.Altisource.com
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***
>

RE: Web UI

2014-09-03 Thread Ruebenacker, Oliver A 

 Hello,

  Thanks for the help! But I tried starting with “–master local[4]” and when I 
load http://localhost:4040/json I just get forwarded to 
http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org
Subject: Re: Web UI

Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI, which 
returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha

On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  What is included in the Spark web UI? What are the available URLs? Can the 
information be obtained in a machine-readable way (e.g. JSON, XML, etc)?

  Thanks!

 Best,
 Oliver

Oliver Ruebenacker | Solutions Architect

Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
oliver.ruebenac...@altisource.com<mailto:oliver.ruebenac...@altisource.com> | 
www.Altisource.com<http://www.Altisource.com>

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system.
This message and any attachments may contain information that is confidential, 
privileged or exempt from disclosure. Delivery of this message to any person 
other than the intended recipient is not intended to waive any right or 
privilege. Message transmission is not guaranteed to be secure or free of 
software viruses.
***

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system. This message and any attachments may contain information that 
is confidential, privileged or exempt from disclosure. Delivery of this message 
to any person other than the intended recipient is not intended to waive any 
right or privilege. Message transmission is not guaranteed to be secure or free 
of software viruses.
***

Re: Web UI

2014-09-03 Thread Wonha Ryu 
Hey Oliver,

IIRC there's no JSON endpoint for application web UI. They only exist for
cluster master and worker.

- Wonha



On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
oliver.ruebenac...@altisource.com> wrote:

>
>
>  Hello,
>
>
>
>   Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
>   Thanks!
>
>
>
>  Best,
>
>  Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> oliver.ruebenac...@altisource.com | www.Altisource.com
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***
>

RE: Web UI

2014-09-03 Thread Ruebenacker, Oliver A 

 Hello,

  Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but 
only a few which never seem to change during the application’s lifetime.

  Either the web UI has some very strange limitations, or there are some URLs 
yet to be discovered that do something interesting.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com]
Sent: Wednesday, September 03, 2014 4:27 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org
Subject: Re: Web UI

Hey Oliver,

IIRC there's no JSON endpoint for application web UI. They only exist for 
cluster master and worker.

- Wonha


On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  Thanks for the help! But I tried starting with “–master local[4]” and when I 
load http://localhost:4040/json I just get forwarded to 
http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com<mailto:wonha@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI, which 
returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha

On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  What is included in the Spark web UI? What are the available URLs? Can the 
information be obtained in a machine-readable way (e.g. JSON, XML, etc)?

  Thanks!

 Best,
 Oliver

Oliver Ruebenacker | Solutions Architect

Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
oliver.ruebenac...@altisource.com<mailto:oliver.ruebenac...@altisource.com> | 
www.Altisource.com<http://www.Altisource.com>

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system.
This message and any attachments may contain information that is confidential, 
privileged or exempt from disclosure. Delivery of this message to any person 
other than the intended recipient is not intended to waive any right or 
privilege. Message transmission is not guaranteed to be secure or free of 
software viruses.
***

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system.
This message and any attachments may contain information that is confidential, 
privileged or exempt from disclosure. Delivery of this message to any person 
other than the intended recipient is not intended to waive any right or 
privilege. Message transmission is not guaranteed to be secure or free of 
software viruses.
***

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system. This message and any attachments may contain information that 
is confidential, privileged or exempt from disclosure. Delivery of this message 
to any person other than the intended recipient is not intended to waive any 
right or privilege. Message transmission is not guaranteed to be secure or free 
of software viruses.
***

Re: Web UI

2014-09-04 Thread Akhil Das 
Hi

You can see this doc
<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
for all the available webUI ports.

Yes there are ways to get the data metrics in Json format, One of them is
below:

*​​http://webUI:8080/json/ <http://webUI:8080/json/>*  Or simply
> *​​curl webUI:8080/json/*


There are some PRs about it you can read it over here
https://github.com/apache/spark/pull/1682

Thanks
Best Regards


On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
oliver.ruebenac...@altisource.com> wrote:

>
>
>  Hello,
>
>
>
>   Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
>   Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
>   Thanks!
>
>
>
>  Best,
>
>  Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> oliver.ruebenac...@altisource.com | www.Altisource.com
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not

RE: Web UI

2014-09-04 Thread Ruebenacker, Oliver A 

 Hello,

  Thanks for the link – this is for standalone, though, and most URLs don’t 
work for local.
  I will look into deploying as standalone on a single node for testing and 
development.

 Best, Oliver

From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; user@spark.apache.org
Subject: Re: Web UI

Hi

You can see this 
doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
 for all the available webUI ports.

Yes there are ways to get the data metrics in Json format, One of them is below:

​​
http://webUI:8080/json/  Or simply
​​
curl webUI:8080/json/

There are some PRs about it you can read it over here 
https://github.com/apache/spark/pull/1682

Thanks
Best Regards

On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but 
only a few which never seem to change during the application’s lifetime.

  Either the web UI has some very strange limitations, or there are some URLs 
yet to be discovered that do something interesting.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com<mailto:wonha@gmail.com>]
Sent: Wednesday, September 03, 2014 4:27 PM

To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hey Oliver,

IIRC there's no JSON endpoint for application web UI. They only exist for 
cluster master and worker.

- Wonha


On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  Thanks for the help! But I tried starting with “–master local[4]” and when I 
load http://localhost:4040/json I just get forwarded to 
http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com<mailto:wonha@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI, which 
returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha

On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  What is included in the Spark web UI? What are the available URLs? Can the 
information be obtained in a machine-readable way (e.g. JSON, XML, etc)?

  Thanks!

 Best,
 Oliver

Oliver Ruebenacker | Solutions Architect

Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
oliver.ruebenac...@altisource.com<mailto:oliver.ruebenac...@altisource.com> | 
www.Altisource.com<http://www.Altisource.com>

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system.
This message and any attachments may contain information that is confidential, 
privileged or exempt from disclosure. Delivery of this message to any person 
other than the intended recipient is not intended to waive any right or 
privilege. Message transmission is not guaranteed to be secure or free of 
software viruses.
***

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please 
promptly notify the sender by reply email and immediately delete this message 
from your system.
This message and any attachments may contain information that is confidential, 
privileged or exempt from disclosure. Delivery of this message to any person 
other than the intended recipient is not intended to waive any right or 
privilege.

Re: Web UI

2014-09-04 Thread Andrew Or 
Hi all,

The JSON version of the web UI is not officially supported; I don't believe
this is documented anywhere.

The alternative is to set `spark.eventLog.enabled` to true before running
your application. This will create JSON SparkListenerEvents with details
about each task and stage as a log file. Then you can easily reconstruct
the web UI after the application has exited. This is what the standalone
Master and the History Server does, actually. For local mode, you can use
the latter to generate your UI after the fact. (This is documented here:
http://spark.apache.org/docs/latest/monitoring.html).

-Andrew


2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <
oliver.ruebenac...@altisource.com>:

>
>
>  Hello,
>
>
>
>   Thanks for the link – this is for standalone, though, and most URLs
> don’t work for local.
>
>   I will look into deploying as standalone on a single node for testing
> and development.
>
>
>
>  Best, Oliver
>
>
>
> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
> *Sent:* Thursday, September 04, 2014 3:09 AM
> *To:* Ruebenacker, Oliver A
> *Cc:* Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi
>
>
>
> You can see this doc
> <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
> for all the available webUI ports.
>
>
>
> Yes there are ways to get the data metrics in Json format, One of them is
> below:
>
>
>
>  *​​*
>
> *http://webUI:8080/json/ <http://webUI:8080/json/>*  Or simply
>
> *​​*
>
> *curl webUI:8080/json/*
>
>
>
> There are some PRs about it you can read it over here
> https://github.com/apache/spark/pull/1682
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
>   Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
>   Thanks!
>
>
>
>  Best,
>
>  Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> oliver.ruebenac...@altisource.com | www.Altisource.com
>
>
>
>
> ***
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify

RE: Web UI

2014-09-05 Thread Ruebenacker, Oliver A 

 Hello,

  Thanks for the explanation. So events are stored internally as JSON, but 
there is no official support for having Spark serve that JSON via HTTP? So if I 
wanted to write an app that monitors Spark, I would either have to scrape the 
web UI in HTML or rely on unofficial JSON features? That is quite surprising, 
because I would expect dumping out the JSON would be easier for Spark 
developers to implement than converting it to HTML.

  Do I get that right? Should I make a feature request? Thanks!

 Best, Oliver

From: Andrew Or [mailto:and...@databricks.com]
Sent: Thursday, September 04, 2014 2:11 PM
To: Ruebenacker, Oliver A
Cc: Akhil Das; Wonha Ryu; user@spark.apache.org
Subject: Re: Web UI

Hi all,

The JSON version of the web UI is not officially supported; I don't believe 
this is documented anywhere.

The alternative is to set `spark.eventLog.enabled` to true before running your 
application. This will create JSON SparkListenerEvents with details about each 
task and stage as a log file. Then you can easily reconstruct the web UI after 
the application has exited. This is what the standalone Master and the History 
Server does, actually. For local mode, you can use the latter to generate your 
UI after the fact. (This is documented here: 
http://spark.apache.org/docs/latest/monitoring.html).

-Andrew

2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>>:

 Hello,

  Thanks for the link – this is for standalone, though, and most URLs don’t 
work for local.
  I will look into deploying as standalone on a single node for testing and 
development.

 Best, Oliver

From: Akhil Das 
[mailto:ak...@sigmoidanalytics.com<mailto:ak...@sigmoidanalytics.com>]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi

You can see this 
doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
 for all the available webUI ports.

Yes there are ways to get the data metrics in Json format, One of them is below:

​​
http://webUI:8080/json/  Or simply
​​
curl webUI:8080/json/

There are some PRs about it you can read it over here 
https://github.com/apache/spark/pull/1682

Thanks
Best Regards

On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but 
only a few which never seem to change during the application’s lifetime.

  Either the web UI has some very strange limitations, or there are some URLs 
yet to be discovered that do something interesting.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com<mailto:wonha@gmail.com>]
Sent: Wednesday, September 03, 2014 4:27 PM

To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hey Oliver,

IIRC there's no JSON endpoint for application web UI. They only exist for 
cluster master and worker.

- Wonha


On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  Thanks for the help! But I tried starting with “–master local[4]” and when I 
load http://localhost:4040/json I just get forwarded to 
http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.

 Best,
 Oliver


From: Wonha Ryu [mailto:wonha@gmail.com<mailto:wonha@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Web UI

Hi Oliver,

Spark standalone master and worker support '/json' endpoint in web UI, which 
returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.

- Wonha

On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A 
mailto:oliver.ruebenac...@altisource.com>> 
wrote:

 Hello,

  What is included in the Spark web UI? What are the available URLs? Can the 
information be obtained in a machine-readable way (e.g. JSON, XML, etc)?

  Thanks!

 Best,
 Oliver

Oliver Ruebenacker | Solutions Architect

Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
oliver.ruebenac...@altisource.com<mailto:oliver.ruebenac...@altisource.com> | 
www.Altisource.com<http://www.Altisource.com>

***

This email message and any attachments are intended solely for the use of the 
addressee. If you are not the intended recipient, you are prohibited from 
reading, disclosing, reproducing, distributing, disseminating or otherwise 
using this transmission. If you have received this message in error, please

Re: Web UI

2014-09-05 Thread Andrew Or 
Sure, you can request it by filing an issue here:
https://issues.apache.org/jira/browse/SPARK


2014-09-05 6:50 GMT-07:00 Ruebenacker, Oliver A <
oliver.ruebenac...@altisource.com>:

>
>
>  Hello,
>
>
>
>   Thanks for the explanation. So events are stored internally as JSON, but
> there is no official support for having Spark serve that JSON via HTTP? So
> if I wanted to write an app that monitors Spark, I would either have to
> scrape the web UI in HTML or rely on unofficial JSON features? That is
> quite surprising, because I would expect dumping out the JSON would be
> easier for Spark developers to implement than converting it to HTML.
>
>
>
>   Do I get that right? Should I make a feature request? Thanks!
>
>
>
>  Best, Oliver
>
>
>
> *From:* Andrew Or [mailto:and...@databricks.com]
> *Sent:* Thursday, September 04, 2014 2:11 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* Akhil Das; Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi all,
>
>
>
> The JSON version of the web UI is not officially supported; I don't
> believe this is documented anywhere.
>
>
>
> The alternative is to set `spark.eventLog.enabled` to true before running
> your application. This will create JSON SparkListenerEvents with details
> about each task and stage as a log file. Then you can easily reconstruct
> the web UI after the application has exited. This is what the standalone
> Master and the History Server does, actually. For local mode, you can use
> the latter to generate your UI after the fact. (This is documented here:
> http://spark.apache.org/docs/latest/monitoring.html).
>
>
>
> -Andrew
>
>
>
> 2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com>:
>
>
>
>  Hello,
>
>
>
>   Thanks for the link – this is for standalone, though, and most URLs
> don’t work for local.
>
>   I will look into deploying as standalone on a single node for testing
> and development.
>
>
>
>  Best, Oliver
>
>
>
> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
> *Sent:* Thursday, September 04, 2014 3:09 AM
> *To:* Ruebenacker, Oliver A
> *Cc:* Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi
>
>
>
> You can see this doc
> <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
> for all the available webUI ports.
>
>
>
> Yes there are ways to get the data metrics in Json format, One of them is
> below:
>
>
>
>  *​​*
>
> *http://webUI:8080/json/ <http://webUI:8080/json/>*  Or simply
>
> *​​*
>
> *curl webUI:8080/json/*
>
>
>
> There are some PRs about it you can read it over here
> https://github.com/apache/spark/pull/1682
>
>
>   Thanks
>
> Best Regards
>
>
>
> On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
>   Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> oliver.ruebenac...@altisource.com> wrote:
>
>
>
>  Hello,
>
>
>
>   Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
>  Best,
>
>  Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns som

Re: Web UI Links

2015-07-20 Thread Bob Corsaro 
I figured this out after spelunking the UI code a little. The trick is to
set the SPARK_PUBLIC_DNS environmental variable to the public DNS name of
each server in the cluster, per node. I'm running in standalone mode, so it
was just a matter of adding the setting to spark-env.sh.

On Mon, Jul 20, 2015 at 9:59 AM Bob Corsaro  wrote:

> I'm running a spark cluster and I'd like to access the Spark-UI from
> outside the LAN. The problem is all the links are to internal IP addresses.
> Is there anyway to config hostnames for each of the hosts in the cluster
> and use those for the links?
>

Re: Web UI is not showing up

2015-09-01 Thread Sonal Goyal 
The web ui is at port 8080. 4040 will show up something when you have a
running job or if you have configured history server.
On Sep 1, 2015 8:57 PM, "Sunil Rathee"  wrote:

>
> Hi,
>
>
> localhost:4040 is not showing anything on the browser. Do we have to start
> some service?
>
> --
>
>
> Sunil Rathee
>
>
>
>

Re: Web UI is not showing up

2015-09-01 Thread Sunil Rathee 
localhost:8080 is also not showing anything. Does some application running
at the same time?

On Tue, Sep 1, 2015 at 9:04 PM, Sonal Goyal  wrote:

> The web ui is at port 8080. 4040 will show up something when you have a
> running job or if you have configured history server.
> On Sep 1, 2015 8:57 PM, "Sunil Rathee"  wrote:
>
>>
>> Hi,
>>
>>
>> localhost:4040 is not showing anything on the browser. Do we have to
>> start some service?
>>
>> --
>>
>>
>> Sunil Rathee
>>
>>
>>
>>


-- 


Sunil Rathee

Re: Web UI is not showing up

2015-09-01 Thread Sonal Goyal 
Is your master up? Check the java processes to see if they are running.

Best Regards,
Sonal
Founder, Nube Technologies 
Reifier covered in YourStory 
Reifier at Spark Summit 2015






On Tue, Sep 1, 2015 at 9:06 PM, Sunil Rathee 
wrote:

> localhost:8080 is also not showing anything. Does some application running
> at the same time?
>
> On Tue, Sep 1, 2015 at 9:04 PM, Sonal Goyal  wrote:
>
>> The web ui is at port 8080. 4040 will show up something when you have a
>> running job or if you have configured history server.
>> On Sep 1, 2015 8:57 PM, "Sunil Rathee"  wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>> localhost:4040 is not showing anything on the browser. Do we have to
>>> start some service?
>>>
>>> --
>>>
>>>
>>> Sunil Rathee
>>>
>>>
>>>
>>>
>
>
> --
>
>
> Sunil Rathee
>
>
>
>

Re: Web UI vs History Server Bugs

2015-06-18 Thread Akhil Das 
You could possibly open up a JIRA and shoot an email to the dev list.

Thanks
Best Regards

On Wed, Jun 17, 2015 at 11:40 PM, jcai  wrote:

> Hi,
>
> I am running this on Spark stand-alone mode. I find that when I examine the
> web UI, a couple bugs arise:
>
> 1. There is a discrepancy between the number denoting the duration of the
> application when I run the history server and the number given by the web
> UI
> (default address is master:8080). I checked more specific details,
> including
> task and stage durations (when clicking on the application), and these
> appear to be the same for both avenues.
>
> 2. Sometimes the web UI on master:8080 is unable to display more specific
> information for an application that has finished (when clicking on the
> application), even when there is a log file in the appropriate directory.
> But when the history server is opened, it is able to read this file and
> output information.
>
> Any ideas on how to approach these?
>
> I am trying to do accurate performance measurements on Spark workloads. I
> believe these might be bugs.
>
> Thanks,
>
> Jonathon
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Web-UI-vs-History-Server-Bugs-tp23371.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Web UI vs History Server Bugs

2015-06-20 Thread Steve Loughran 

> On 17 Jun 2015, at 19:10, jcai  wrote:
> 
> Hi,
> 
> I am running this on Spark stand-alone mode. I find that when I examine the
> web UI, a couple bugs arise:
> 
> 1. There is a discrepancy between the number denoting the duration of the
> application when I run the history server and the number given by the web UI
> (default address is master:8080). I checked more specific details, including
> task and stage durations (when clicking on the application), and these
> appear to be the same for both avenues.
> 
> 2. Sometimes the web UI on master:8080 is unable to display more specific
> information for an application that has finished (when clicking on the
> application), even when there is a log file in the appropriate directory.
> But when the history server is opened, it is able to read this file and
> output information. 
> 

There's a JIRA open on the history server caching incomplete work...if you 
click on the link to a job while it's in progress, you don't get any updates 
later. 

does this sound like what you are seeing?


-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Web UI vs History Server Bugs

2015-06-22 Thread Jonathon Cai 
No, what I'm seeing is that while the cluster is running, I can't see the
app info after the app is completed. That is to say, when I click on the
application name on master:8080, no info is shown. However, when I examine
the same file on the History Server, the application information opens fine.

On Sat, Jun 20, 2015 at 6:47 AM, Steve Loughran 
wrote:

>
> > On 17 Jun 2015, at 19:10, jcai  wrote:
> >
> > Hi,
> >
> > I am running this on Spark stand-alone mode. I find that when I examine
> the
> > web UI, a couple bugs arise:
> >
> > 1. There is a discrepancy between the number denoting the duration of the
> > application when I run the history server and the number given by the
> web UI
> > (default address is master:8080). I checked more specific details,
> including
> > task and stage durations (when clicking on the application), and these
> > appear to be the same for both avenues.
> >
> > 2. Sometimes the web UI on master:8080 is unable to display more specific
> > information for an application that has finished (when clicking on the
> > application), even when there is a log file in the appropriate directory.
> > But when the history server is opened, it is able to read this file and
> > output information.
> >
>
> There's a JIRA open on the history server caching incomplete work...if you
> click on the link to a job while it's in progress, you don't get any
> updates later.
>
> does this sound like what you are seeing?
>
>

Re: Web UI vs History Server Bugs

2015-06-22 Thread Steve Loughran 
well, I'm afraid you've reached the limits of my knowledge ... hopefully 
someone else can answer

On 22 Jun 2015, at 16:37, Jonathon Cai 
mailto:jonathon@yale.edu>> wrote:

No, what I'm seeing is that while the cluster is running, I can't see the app 
info after the app is completed. That is to say, when I click on the 
application name on master:8080, no info is shown. However, when I examine the 
same file on the History Server, the application information opens fine.

On Sat, Jun 20, 2015 at 6:47 AM, Steve Loughran 
mailto:ste...@hortonworks.com>> wrote:

> On 17 Jun 2015, at 19:10, jcai 
> mailto:jonathon@yale.edu>> wrote:
>
> Hi,
>
> I am running this on Spark stand-alone mode. I find that when I examine the
> web UI, a couple bugs arise:
>
> 1. There is a discrepancy between the number denoting the duration of the
> application when I run the history server and the number given by the web UI
> (default address is master:8080). I checked more specific details, including
> task and stage durations (when clicking on the application), and these
> appear to be the same for both avenues.
>
> 2. Sometimes the web UI on master:8080 is unable to display more specific
> information for an application that has finished (when clicking on the
> application), even when there is a log file in the appropriate directory.
> But when the history server is opened, it is able to read this file and
> output information.
>

There's a JIRA open on the history server caching incomplete work...if you 
click on the link to a job while it's in progress, you don't get any updates 
later.

does this sound like what you are seeing?

RE: Web UI vs History Server Bugs

2015-06-23 Thread Evo Eftimov 
Probably your application has crashed or was terminated without invoking the
stop method of spark context - in such cases it doesn't create the empty
flag file which apparently tells the history server that it can safely show
the log data - simpy go to some of the other dirs of the history server to
see what the name of the flag file was and then create it manually in the
dirs of the missing apps - then they will appear in the history server ui

 

From: Steve Loughran [mailto:ste...@hortonworks.com] 
Sent: Monday, June 22, 2015 7:22 PM
To: Jonathon Cai
Cc: user@spark.apache.org
Subject: Re: Web UI vs History Server Bugs

 

well, I'm afraid you've reached the limits of my knowledge ... hopefully
someone else can answer 

 

On 22 Jun 2015, at 16:37, Jonathon Cai  wrote:

 

No, what I'm seeing is that while the cluster is running, I can't see the
app info after the app is completed. That is to say, when I click on the
application name on master:8080, no info is shown. However, when I examine
the same file on the History Server, the application information opens fine.

 

On Sat, Jun 20, 2015 at 6:47 AM, Steve Loughran 
wrote:


> On 17 Jun 2015, at 19:10, jcai  wrote:
>
> Hi,
>
> I am running this on Spark stand-alone mode. I find that when I examine
the
> web UI, a couple bugs arise:
>
> 1. There is a discrepancy between the number denoting the duration of the
> application when I run the history server and the number given by the web
UI
> (default address is master:8080). I checked more specific details,
including
> task and stage durations (when clicking on the application), and these
> appear to be the same for both avenues.
>
> 2. Sometimes the web UI on master:8080 is unable to display more specific
> information for an application that has finished (when clicking on the
> application), even when there is a log file in the appropriate directory.
> But when the history server is opened, it is able to read this file and
> output information.
>

There's a JIRA open on the history server caching incomplete work...if you
click on the link to a job while it's in progress, you don't get any updates
later.

does this sound like what you are seeing?

Re: Web UI doesn't show some stages

2014-08-20 Thread Patrick Wendell 
The reason is that some operators get pipelined into a single stage.
rdd.map(XX).filter(YY) - this executes in a single stage since there is no
data movement needed in between these operations.

If you call toDeubgString on the final RDD it will give you some
information about the exact lineage. In Spark 1.1 this will return
information about stage boudnaries as well.


On Wed, Aug 20, 2014 at 4:22 AM, Grzegorz Białek <
grzegorz.bia...@codilime.com> wrote:

> Hi,
>
> I am wondering why in web UI some stages (like join, filter) are not
> visible. For example this code:
>
> val simple = sc.parallelize(Array.range(0,100))
> val simple2 = sc.parallelize(Array.range(0,100))
>
>   val toJoin = simple.map(x => (x, x.toString + x.toString))
>   val rdd = simple2
> .map(x => (scala.util.Random.nextInt(100), x))
> .join(toJoin)
> .map { case (r, (x, s)) => (r, x)}
> .reduceByKey(_ + _)
> .sortByKey()
> .cache()
>   rdd.saveAsTextFile("output/1")
>
>   val rdd2 = toJoin
> .groupBy{ case (x, _) => x}
> .filter{ case (x, _) => x < 10}
>   rdd2.saveAsTextFile("output/2")
>
>   println(rdd2.join(toJoin).count())
>
> in UI doesn't show join and filter stages and moreover it shows sortByKey
> and reduceByKey twice.
> Could anyone explain how it works?
>
> Thanks,
> Grzegorz
>

Re: Web UI doesn't show some stages

2014-08-20 Thread Zhan Zhang 
Try to answer your another question.

One sortByKey is triggered by rangePartition which does sample to calculate the 
range boundaries, which again triggers the first reduceByKey.

The second sortByKey is doing the real work to sort based on the partition 
calculated, which again trigger the reduceByKey because it is not cached.

I agree with you it is very confusing.

Thanks.

Zhan Zhang

The f
On Aug 20, 2014, at 2:28 PM, Patrick Wendell  wrote:

> The reason is that some operators get pipelined into a single stage. 
> rdd.map(XX).filter(YY) - this executes in a single stage since there is no 
> data movement needed in between these operations.
> 
> If you call toDeubgString on the final RDD it will give you some information 
> about the exact lineage. In Spark 1.1 this will return information about 
> stage boudnaries as well.
> 
> 
> On Wed, Aug 20, 2014 at 4:22 AM, Grzegorz Białek 
>  wrote:
> Hi,
> 
> I am wondering why in web UI some stages (like join, filter) are not visible. 
> For example this code:
> 
> val simple = sc.parallelize(Array.range(0,100))
> val simple2 = sc.parallelize(Array.range(0,100))
> 
>   val toJoin = simple.map(x => (x, x.toString + x.toString))
>   val rdd = simple2
> .map(x => (scala.util.Random.nextInt(100), x))
> .join(toJoin)
> .map { case (r, (x, s)) => (r, x)}
> .reduceByKey(_ + _)
> .sortByKey()
> .cache()
>   rdd.saveAsTextFile("output/1")
> 
>   val rdd2 = toJoin
> .groupBy{ case (x, _) => x}
> .filter{ case (x, _) => x < 10}
>   rdd2.saveAsTextFile("output/2")
> 
>   println(rdd2.join(toJoin).count())
> 
> in UI doesn't show join and filter stages and moreover it shows sortByKey and 
> reduceByKey twice.
> Could anyone explain how it works?
> 
> Thanks,
> Grzegorz
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.