Hello,
Thanks for the explanation. So events are stored internally as JSON, but
there is no official support for having Spark serve that JSON via HTTP? So if I
wanted to write an app that monitors Spark, I would either have to scrape the
web UI in HTML or rely on unofficial JSON features? That is quite surprising,
because I would expect dumping out the JSON would be easier for Spark
developers to implement than converting it to HTML.
Do I get that right? Should I make a feature request? Thanks!
Best, Oliver
From: Andrew Or [mailto:[email protected]]
Sent: Thursday, September 04, 2014 2:11 PM
To: Ruebenacker, Oliver A
Cc: Akhil Das; Wonha Ryu; [email protected]
Subject: Re: Web UI
Hi all,
The JSON version of the web UI is not officially supported; I don't believe
this is documented anywhere.
The alternative is to set `spark.eventLog.enabled` to true before running your
application. This will create JSON SparkListenerEvents with details about each
task and stage as a log file. Then you can easily reconstruct the web UI after
the application has exited. This is what the standalone Master and the History
Server does, actually. For local mode, you can use the latter to generate your
UI after the fact. (This is documented here:
http://spark.apache.org/docs/latest/monitoring.html).
-Andrew
2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A
<[email protected]<mailto:[email protected]>>:
Hello,
Thanks for the link – this is for standalone, though, and most URLs don’t
work for local.
I will look into deploying as standalone on a single node for testing and
development.
Best, Oliver
From: Akhil Das
[mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; [email protected]<mailto:[email protected]>
Subject: Re: Web UI
Hi
You can see this
doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
for all the available webUI ports.
Yes there are ways to get the data metrics in Json format, One of them is below:
http://webUI:8080/json/ Or simply
curl webUI:8080/json/
There are some PRs about it you can read it over here
https://github.com/apache/spark/pull/1682
Thanks
Best Regards
On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A
<[email protected]<mailto:[email protected]>>
wrote:
Hello,
Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but
only a few which never seem to change during the application’s lifetime.
Either the web UI has some very strange limitations, or there are some URLs
yet to be discovered that do something interesting.
Best,
Oliver
From: Wonha Ryu [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, September 03, 2014 4:27 PM
To: Ruebenacker, Oliver A
Cc: [email protected]<mailto:[email protected]>
Subject: Re: Web UI
Hey Oliver,
IIRC there's no JSON endpoint for application web UI. They only exist for
cluster master and worker.
- Wonha
On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A
<[email protected]<mailto:[email protected]>>
wrote:
Hello,
Thanks for the help! But I tried starting with “–master local[4]” and when I
load http://localhost:4040/json I just get forwarded to
http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
Best,
Oliver
From: Wonha Ryu [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: [email protected]<mailto:[email protected]>
Subject: Re: Web UI
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI, which
returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A
<[email protected]<mailto:[email protected]>>
wrote:
Hello,
What is included in the Spark web UI? What are the available URLs? Can the
information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
[email protected]<mailto:[email protected]> |
www.Altisource.com<http://www.Altisource.com>
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system.
This message and any attachments may contain information that is confidential,
privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or
privilege. Message transmission is not guaranteed to be secure or free of
software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system.
This message and any attachments may contain information that is confidential,
privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or
privilege. Message transmission is not guaranteed to be secure or free of
software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system.
This message and any attachments may contain information that is confidential,
privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or
privilege. Message transmission is not guaranteed to be secure or free of
software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system.
This message and any attachments may contain information that is confidential,
privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or
privilege. Message transmission is not guaranteed to be secure or free of
software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message
to any person other than the intended recipient is not intended to waive any
right or privilege. Message transmission is not guaranteed to be secure or free
of software viruses.
***********************************************************************************************************************
