8080 is just the normal web UI. Which is the information I want, ie
Running Applications, but in HTML format. I want it in JSON so I don't
have to be scraping and parsing HTML.
From my understanding api/v1/applications should do the trick ...
except it doesn't.
Ah well.
On 1/12/2016 4:00 PM, Miguel Morales wrote:
Don't have a Spark cluster up to verify this, but try port 8080.
http://spark-master-ip:8080/api/v1/applications.
But glad to hear you're getting somewhere, best of luck.
On Wed, Nov 30, 2016 at 9:59 PM, Carl Ballantyne
<carl.ballant...@guvera.com <mailto:carl.ballant...@guvera.com>> wrote:
Hmmm getting closer I think.
I thought this was only for Mesos and Yarn clusters (from reading
the documentation). I tried anyway and initially received
Connection Refused. So I ran ./start-history-server.sh. This was
on the Spark Master instance.
I now get 404 not found.
Nothing in the log file for the history server indicates there was
a problem.
I will keep digging around. Thanks for your help so far Miguel.
On 1/12/2016 3:33 PM, Miguel Morales wrote:
Try hitting:http://<server-url>:18080/api/v1
Then hit /applications.
That should give you a list of running spark jobs on a given server.
On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
<carl.ballant...@guvera.com> <mailto:carl.ballant...@guvera.com> wrote:
Yes I was looking at this. But it says I need to access the driver
-http://<driver-node>:4040.
I don't have a running driver Spark instance since I am submitting jobs to
Spark using the SparkLauncher class. Or maybe I am missing something obvious.
Apologies if so.
On 1/12/2016 3:21 PM, Miguel Morales wrote:
Check the Monitoring and Instrumentation
API:http://spark.apache.org/docs/latest/monitoring.html
<http://spark.apache.org/docs/latest/monitoring.html>
On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne<carl.ballant...@guvera.com>
<mailto:carl.ballant...@guvera.com> wrote:
Hi All,
I want to get the running applications for my Spark Standalone cluster in
JSON format. The same information displayed on the web UI on port 8080 ... but
in JSON.
Is there an easy way to do this? It seems I need to scrap the HTML page in
order to get this information.
The reason I want to know this information is so I can ensure the Spark
cluster does not get too many jobs submitted at once. A Stand alone cluster
processes jobs FIFO. I would prefer to just send back a message to the user
telling them to try later then submit a job which has to wait for other jobs to
finish before starting.
Any help appreciated. Thanks.
Cheers,
Carl
---------------------------------------------------------------------
To unsubscribe e-mail:user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
--
Carl Ballantyne
Lead Reporting Developer
Guvera Operations Pty Ltd.
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
Phone+61 (0) 7 5578 8987 <tel:+61%207%205578%208987>
emailcarl.ballant...@guvera.com <mailto:carl.ballant...@guvera.com>
Webwww.guveralimited.com <http://www.guveralimited.com>
--
*Carl Ballantyne* Lead Reporting Developer
*Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212 Australia
PO Box 3330 Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987 <tel:+61%207%205578%208987> *Email
*carl.ballant...@guvera.com <mailto:carl.ballant...@guvera.com>
*Web *www.guveralimited.com <http://www.guveralimited.com/>
--
*Carl Ballantyne* Lead Reporting Developer
*Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive Helensvale,
QLD, 4212 Australia
PO Box 3330 Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987 *Email *carl.ballant...@guvera.com
<mailto:carl.ballant...@guvera.com> *Web *www.guveralimited.com
<http://www.guveralimited.com/>
