Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Carl Ballantyne 
8080 is just the normal web UI. Which is the information I want, ie 
Running Applications, but in HTML format. I want it in JSON so I don't 
have to be scraping and parsing HTML.


From my understanding api/v1/applications should do the trick ... 
except it doesn't.


Ah well.

On 1/12/2016 4:00 PM, Miguel Morales wrote:

Don't have a Spark cluster up to verify this, but try port 8080.

http://spark-master-ip:8080/api/v1/applications.

But glad to hear you're getting somewhere, best of luck.

On Wed, Nov 30, 2016 at 9:59 PM, Carl Ballantyne 
mailto:carl.ballant...@guvera.com>> wrote:


Hmmm getting closer I think.

I thought this was only for Mesos and Yarn clusters (from reading
the documentation). I tried anyway and initially received
Connection Refused. So I ran ./start-history-server.sh. This was
on the Spark Master instance.

I now get 404 not found.

Nothing in the log file for the history server indicates there was
a problem.

I will keep digging around. Thanks for your help so far Miguel.


On 1/12/2016 3:33 PM, Miguel Morales wrote:

Try hitting:http://:18080/api/v1

Then hit /applications.

That should give you a list of running spark jobs on a given server.

On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
   wrote:

Yes I was looking at this. But it says I need to access the driver 
-http://:4040.

I don't have a running driver Spark instance since I am submitting jobs to 
Spark using the SparkLauncher class. Or maybe I am missing something obvious. 
Apologies if so.




On 1/12/2016 3:21 PM, Miguel Morales wrote:

Check the Monitoring and Instrumentation 
API:http://spark.apache.org/docs/latest/monitoring.html


On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne 
  wrote:

Hi All,

I want to get the running applications for my Spark Standalone cluster in 
JSON format. The same information displayed on the web UI on port 8080 ... but 
in JSON.

Is there an easy way to do this? It seems I need to scrap the HTML page in 
order to get this information.

The reason I want to know this information is so I can ensure the Spark 
cluster does not get too many jobs submitted at once. A Stand alone cluster 
processes jobs FIFO. I would prefer to just send back a message to the user 
telling them to try later then submit a job which has to wait for other jobs to 
finish before starting.

Any help appreciated. Thanks.

Cheers,
Carl

-
To unsubscribe e-mail:user-unsubscr...@spark.apache.org



--
Carl Ballantyne
Lead Reporting Developer
Guvera Operations Pty Ltd.
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
Phone+61 (0) 7 5578 8987 
emailcarl.ballant...@guvera.com 
Webwww.guveralimited.com 


-- 
*Carl Ballantyne* Lead Reporting Developer

*Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212 Australia
PO Box 3330 Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987  *Email
*carl.ballant...@guvera.com 
*Web *www.guveralimited.com 


--
*Carl Ballantyne* Lead Reporting Developer
*Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive Helensvale, 
QLD, 4212 Australia

PO Box 3330 Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987 *Email *carl.ballant...@guvera.com 
 *Web *www.guveralimited.com

Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Miguel Morales 
Don't have a Spark cluster up to verify this, but try port 8080.

http://spark-master-ip:8080/api/v1/applications.

But glad to hear you're getting somewhere, best of luck.

On Wed, Nov 30, 2016 at 9:59 PM, Carl Ballantyne  wrote:

> Hmmm getting closer I think.
>
> I thought this was only for Mesos and Yarn clusters (from reading the
> documentation). I tried anyway and initially received Connection Refused.
> So I ran ./start-history-server.sh. This was on the Spark Master instance.
>
> I now get 404 not found.
>
> Nothing in the log file for the history server indicates there was a
> problem.
>
> I will keep digging around. Thanks for your help so far Miguel.
>
> On 1/12/2016 3:33 PM, Miguel Morales wrote:
>
> Try hitting:  http://:18080/api/v1
>
> Then hit /applications.
>
> That should give you a list of running spark jobs on a given server.
>
> On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne 
>  wrote:
>
> Yes I was looking at this. But it says I need to access the driver - 
> http://:4040.
>
> I don't have a running driver Spark instance since I am submitting jobs to 
> Spark using the SparkLauncher class. Or maybe I am missing something obvious. 
> Apologies if so.
>
>
>
>
> On 1/12/2016 3:21 PM, Miguel Morales wrote:
>
> Check the Monitoring and Instrumentation API: 
> http://spark.apache.org/docs/latest/monitoring.html
>
> On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne  
>  wrote:
>
> Hi All,
>
> I want to get the running applications for my Spark Standalone cluster in 
> JSON format. The same information displayed on the web UI on port 8080 ... 
> but in JSON.
>
> Is there an easy way to do this? It seems I need to scrap the HTML page in 
> order to get this information.
>
> The reason I want to know this information is so I can ensure the Spark 
> cluster does not get too many jobs submitted at once. A Stand alone cluster 
> processes jobs FIFO. I would prefer to just send back a message to the user 
> telling them to try later then submit a job which has to wait for other jobs 
> to finish before starting.
>
> Any help appreciated. Thanks.
>
> Cheers,
> Carl
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
> --
> Carl Ballantyne
> Lead Reporting Developer
> Guvera Operations Pty Ltd.
> Suite 1b, 58 Kingston Drive
> Helensvale, QLD, 4212
> Australia
> PO Box 3330
> Helensvale Town Centre, QLD, 4212
> Phone +61 (0) 7 5578 8987 <+61%207%205578%208987>
> Email carl.ballant...@guvera.com
> Web www.guveralimited.com
>
>
> --
> *Carl Ballantyne*
> Lead Reporting Developer
> *Guvera Operations Pty Ltd.*
> Suite 1b, 58 Kingston Drive
> Helensvale, QLD, 4212
> Australia
> PO Box 3330
> Helensvale Town Centre, QLD, 4212
> *Phone *+61 (0) 7 5578 8987 <+61%207%205578%208987>
> *Email *carl.ballant...@guvera.com
> *Web *www.guveralimited.com
>
>
>

Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Carl Ballantyne 

Hmmm getting closer I think.

I thought this was only for Mesos and Yarn clusters (from reading the 
documentation). I tried anyway and initially received Connection 
Refused. So I ran ./start-history-server.sh. This was on the Spark 
Master instance.


I now get 404 not found.

Nothing in the log file for the history server indicates there was a 
problem.


I will keep digging around. Thanks for your help so far Miguel.


On 1/12/2016 3:33 PM, Miguel Morales wrote:

Try hitting:  http://:18080/api/v1

Then hit /applications.

That should give you a list of running spark jobs on a given server.

On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
 wrote:

Yes I was looking at this. But it says I need to access the driver - 
http://:4040.

I don't have a running driver Spark instance since I am submitting jobs to 
Spark using the SparkLauncher class. Or maybe I am missing something obvious. 
Apologies if so.




On 1/12/2016 3:21 PM, Miguel Morales wrote:

Check the Monitoring and Instrumentation API: 
http://spark.apache.org/docs/latest/monitoring.html

On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne  
wrote:

Hi All,

I want to get the running applications for my Spark Standalone cluster in JSON 
format. The same information displayed on the web UI on port 8080 ... but in 
JSON.

Is there an easy way to do this? It seems I need to scrap the HTML page in 
order to get this information.

The reason I want to know this information is so I can ensure the Spark cluster 
does not get too many jobs submitted at once. A Stand alone cluster processes 
jobs FIFO. I would prefer to just send back a message to the user telling them 
to try later then submit a job which has to wait for other jobs to finish 
before starting.

Any help appreciated. Thanks.

Cheers,
Carl

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org



--
Carl Ballantyne
Lead Reporting Developer
Guvera Operations Pty Ltd.
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
Phone +61 (0) 7 5578 8987
Email carl.ballant...@guvera.com
Web www.guveralimited.com




--
*Carl Ballantyne*
Lead Reporting Developer
*Guvera Operations Pty Ltd.*
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987
*Email *carl.ballant...@guvera.com 
*Web *www.guveralimited.com

Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Miguel Morales 
Try hitting:  http://:18080/api/v1

Then hit /applications.

That should give you a list of running spark jobs on a given server.

On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
 wrote:
>
> Yes I was looking at this. But it says I need to access the driver - 
> http://:4040.
>
> I don't have a running driver Spark instance since I am submitting jobs to 
> Spark using the SparkLauncher class. Or maybe I am missing something obvious. 
> Apologies if so.
>
>
>
>
> On 1/12/2016 3:21 PM, Miguel Morales wrote:
>
> Check the Monitoring and Instrumentation API: 
> http://spark.apache.org/docs/latest/monitoring.html
>
> On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne  
> wrote:
>>
>> Hi All,
>>
>> I want to get the running applications for my Spark Standalone cluster in 
>> JSON format. The same information displayed on the web UI on port 8080 ... 
>> but in JSON.
>>
>> Is there an easy way to do this? It seems I need to scrap the HTML page in 
>> order to get this information.
>>
>> The reason I want to know this information is so I can ensure the Spark 
>> cluster does not get too many jobs submitted at once. A Stand alone cluster 
>> processes jobs FIFO. I would prefer to just send back a message to the user 
>> telling them to try later then submit a job which has to wait for other jobs 
>> to finish before starting.
>>
>> Any help appreciated. Thanks.
>>
>> Cheers,
>> Carl
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>
>
> --
> Carl Ballantyne
> Lead Reporting Developer
> Guvera Operations Pty Ltd.
> Suite 1b, 58 Kingston Drive
> Helensvale, QLD, 4212
> Australia
> PO Box 3330
> Helensvale Town Centre, QLD, 4212
> Phone +61 (0) 7 5578 8987
> Email carl.ballant...@guvera.com
> Web www.guveralimited.com
>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Carl Ballantyne 
Yes I was looking at this. But it says I need to access the driver - 
|http://:4040.|


I don't have a running driver Spark instance since I am submitting jobs 
to Spark using the SparkLauncher class. Or maybe I am missing something 
obvious. Apologies if so.




On 1/12/2016 3:21 PM, Miguel Morales wrote:
Check the Monitoring and Instrumentation API: 
http://spark.apache.org/docs/latest/monitoring.html


On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne 
mailto:carl.ballant...@guvera.com>> wrote:


Hi All,

I want to get the running applications for my Spark Standalone
cluster in JSON format. The same information displayed on the web
UI on port 8080 ... but in JSON.

Is there an easy way to do this? It seems I need to scrap the HTML
page in order to get this information.

The reason I want to know this information is so I can ensure the
Spark cluster does not get too many jobs submitted at once. A
Stand alone cluster processes jobs FIFO. I would prefer to just
send back a message to the user telling them to try later then
submit a job which has to wait for other jobs to finish before
starting.

Any help appreciated. Thanks.

Cheers,
Carl

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org





--
*Carl Ballantyne*
Lead Reporting Developer
*Guvera Operations Pty Ltd.*
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987
*Email *carl.ballant...@guvera.com 
*Web *www.guveralimited.com

Re: Spark Standalone Cluster - Running applications in JSON format

2016-11-30 Thread Miguel Morales 
Check the Monitoring and Instrumentation API:
http://spark.apache.org/docs/latest/monitoring.html

On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne  wrote:

> Hi All,
>
> I want to get the running applications for my Spark Standalone cluster in
> JSON format. The same information displayed on the web UI on port 8080 ...
> but in JSON.
>
> Is there an easy way to do this? It seems I need to scrap the HTML page in
> order to get this information.
>
> The reason I want to know this information is so I can ensure the Spark
> cluster does not get too many jobs submitted at once. A Stand alone cluster
> processes jobs FIFO. I would prefer to just send back a message to the user
> telling them to try later then submit a job which has to wait for other
> jobs to finish before starting.
>
> Any help appreciated. Thanks.
>
> Cheers,
> Carl
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>