How to spark-submit using python subprocess module?

I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server. This is what I have so far in my SubmitJob.py script #!/usr/bin/python #

Re: Spark REST API shows Error 503 Service Unavailable

Hi Prateek, Were you able to figure why this is happening? I'm seeing the same error on my spark standalone cluster. Any pointers anyone? On Fri, Dec 11, 2015 at 2:05 PM, prateek arora wrote: > > > Hi > > I am trying to access Spark Using REST API but got below

Re: Spark REST API shows Error 503 Service Unavailable

No we are using standard spark w/ datastax cassandra. I'm able to see some json when I do http://10.1.40.16:7080/json/v1/applications but getting the following errors when I do http://10.1.40.16:7080/api/v1/applications HTTP ERROR 503 Problem accessing /api/v1/applications. Reason: Service

Re: How to kill spark applications submitted using spark-submit reliably?

s > > On Nov 20, 2015, at 6:46 PM, Vikram Kone <vikramk...@gmail.com> wrote: > > Hi, > I'm seeing a strange problem. I have a spark cluster in standalone mode. I > submit spark jobs from a remote node as follows from the terminal > > spark-submit --master spark://10.1

Re: How to kill spark applications submitted using spark-submit reliably?

Spark 1.4.1 On Friday, November 20, 2015, Ted Yu <yuzhih...@gmail.com> wrote: > Which Spark release are you using ? > > Can you pastebin the stack trace of the process running on your machine ? > > Thanks > > On Nov 20, 2015, at 6:46 PM, Vikram Kone <vikramk...@gma

Re: How to kill spark applications submitted using spark-submit reliably?

ntee that the > thread pool was shutdown when spark stopped > > I hopes this help > > Stephane > ​ > > On Fri, Nov 20, 2015 at 7:46 PM, Vikram Kone <vikramk...@gmail.com > <javascript:_e(%7B%7D,'cvml','vikramk...@gmail.com');>> wrote: > >> Hi, >

Re: Spark job workflow engine recommendations

Hi Feng, Does airflow allow remote submissions of spark jobs via spark-submit? On Wed, Nov 18, 2015 at 6:01 PM, Fengdong Yu <fengdo...@everstring.com> wrote: > Hi, > > we use ‘Airflow' as our job workflow scheduler. > > > > > On Nov 19, 2015, at 9:47 AM, Vikram Kon

Re: Spark job workflow engine recommendations

h oozie >> >> Le ven. 7 août 2015 à 17:51, Hien Luu <h...@linkedin.com.invalid> a >> écrit : >> >>> Looks like Oozie can satisfy most of your requirements. >>> >>> >>> >>> On Fri, Aug 7, 2015 at 8:43 AM, Vikram Kone <vikramk...@gma

Re: Spark job workflow engine recommendations

ins/pull/195. You can leverage > the SLA feature to kill a job if it ran longer than expected. > > BTW, we just solved the scalability issue by supporting multiple > executors. Within a week or two, the code for that should be merged in the > main trunk. > > Hien > > On

Re: Notification on Spark Streaming job failure

We are using Monit to kick off spark streaming jobs n seems to work fine. On Monday, September 28, 2015, Chen Song wrote: > I am also interested specifically in monitoring and alerting on Spark > streaming jobs. It will be helpful to get some general guidelines or advice

Re: Spark job workflow engine recommendations

Does Azkaban support scheduling long running jobs like spark steaming jobs? Will Azkaban kill a job if it's running for a long time. On Friday, August 7, 2015, Vikram Kone <vikramk...@gmail.com> wrote: > Hien, > Is Azkaban being phased out at linkedin as rumored? If so, what's link

How to run spark in standalone mode on cassandra with high availability?

Hi, We are planning to install Spark in stand alone mode on cassandra cluster. The problem, is since Cassandra has a no-SPOF architecture ie any node can become the master for the cluster, it creates the problem for Spark master since it's not a peer-peer architecture where any node can become the

Re: Spark job workflow engine recommendations

satisfy most of your requirements. On Fri, Aug 7, 2015 at 8:43 AM, Vikram Kone vikramk...@gmail.com wrote: Hi, I'm looking for open source workflow tools/engines that allow us to schedule spark jobs on a datastax cassandra cluster. Since there are tonnes of alternatives out there like Ozzie

Spark job workflow engine recommendations

Hi, I'm looking for open source workflow tools/engines that allow us to schedule spark jobs on a datastax cassandra cluster. Since there are tonnes of alternatives out there like Ozzie, Azkaban, Luigi , Chronos etc, I wanted to check with people here to see what they are using today. Some of the

Re: Spark job workflow engine recommendations

full feature set On Friday, August 7, 2015, Hien Luu h...@linkedin.com wrote: Looks like Oozie can satisfy most of your requirements. On Fri, Aug 7, 2015 at 8:43 AM, Vikram Kone vikramk...@gmail.com javascript:_e(%7B%7D,'cvml','vikramk...@gmail.com'); wrote: Hi, I'm looking for open

Re: Spark job workflow engine recommendations

,'cvml','jornfra...@gmail.com'); wrote: Check also falcon in combination with oozie Le ven. 7 août 2015 à 17:51, Hien Luu h...@linkedin.com.invalid a écrit : Looks like Oozie can satisfy most of your requirements. On Fri, Aug 7, 2015 at 8:43 AM, Vikram Kone vikramk...@gmail.com javascript:_e