run multiple spark applications in parallel

2014-10-28 Thread Josh J 
Hi,

How do I run multiple spark applications in parallel? I tried to run on
yarn cluster, though the second application submitted does not run.

Thanks,
Josh

Re: run multiple spark applications in parallel

2014-10-28 Thread Soumya Simanta 
Try reducing the resources (cores and memory) of each application. 



 On Oct 28, 2014, at 7:05 PM, Josh J joshjd...@gmail.com wrote:
 
 Hi,
 
 How do I run multiple spark applications in parallel? I tried to run on yarn 
 cluster, though the second application submitted does not run.
 
 Thanks,
 Josh

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: run multiple spark applications in parallel

2014-10-28 Thread Josh J 
Sorry, I should've included some stats with my email

I execute each job in the following manner

./bin/spark-submit --class CLASSNAME --master yarn-cluster --driver-memory
1g --executor-memory 1g --executor-cores 1 UBER.JAR
${ZK_PORT_2181_TCP_ADDR} my-consumer-group1 1


The box has

24 CPUs, Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz

32 GB RAM


Thanks,

Josh

On Tue, Oct 28, 2014 at 4:15 PM, Soumya Simanta soumya.sima...@gmail.com
wrote:

 Try reducing the resources (cores and memory) of each application.



  On Oct 28, 2014, at 7:05 PM, Josh J joshjd...@gmail.com wrote:
 
  Hi,
 
  How do I run multiple spark applications in parallel? I tried to run on
 yarn cluster, though the second application submitted does not run.
 
  Thanks,
  Josh

Re: run multiple spark applications in parallel

2014-10-28 Thread Soumya Simanta 
Maybe changing --master yarn-cluster to --master yarn-client help.


On Tue, Oct 28, 2014 at 7:25 PM, Josh J joshjd...@gmail.com wrote:

 Sorry, I should've included some stats with my email

 I execute each job in the following manner

 ./bin/spark-submit --class CLASSNAME --master yarn-cluster --driver-memory
 1g --executor-memory 1g --executor-cores 1 UBER.JAR
 ${ZK_PORT_2181_TCP_ADDR} my-consumer-group1 1


 The box has

 24 CPUs, Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz

 32 GB RAM


 Thanks,

 Josh

 On Tue, Oct 28, 2014 at 4:15 PM, Soumya Simanta soumya.sima...@gmail.com
 wrote:

 Try reducing the resources (cores and memory) of each application.



  On Oct 28, 2014, at 7:05 PM, Josh J joshjd...@gmail.com wrote:
 
  Hi,
 
  How do I run multiple spark applications in parallel? I tried to run on
 yarn cluster, though the second application submitted does not run.
 
  Thanks,
  Josh

Re: run multiple spark applications in parallel

2014-10-28 Thread Zhan Zhang 
You can set your executor number with --num-executors. Also changing 
yarn-client save you one container for driver. Then check your yarn resource 
manager to make sure there are more containers available to serve your extra 
apps.

Thanks.

Zhan Zhang

On Oct 28, 2014, at 5:31 PM, Soumya Simanta soumya.sima...@gmail.com wrote:

 Maybe changing --master yarn-cluster to --master yarn-client help. 
 
 
 On Tue, Oct 28, 2014 at 7:25 PM, Josh J joshjd...@gmail.com wrote:
 Sorry, I should've included some stats with my email
 
 I execute each job in the following manner
 
 ./bin/spark-submit --class CLASSNAME --master yarn-cluster --driver-memory 1g 
 --executor-memory 1g --executor-cores 1 UBER.JAR ${ZK_PORT_2181_TCP_ADDR} 
 my-consumer-group1 1
 
 
 
 The box has 
 
 24 CPUs, Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz
 
 32 GB RAM
 
 
 
 Thanks,
 
 Josh
 
 
 On Tue, Oct 28, 2014 at 4:15 PM, Soumya Simanta soumya.sima...@gmail.com 
 wrote:
 Try reducing the resources (cores and memory) of each application.
 
 
 
  On Oct 28, 2014, at 7:05 PM, Josh J joshjd...@gmail.com wrote:
 
  Hi,
 
  How do I run multiple spark applications in parallel? I tried to run on 
  yarn cluster, though the second application submitted does not run.
 
  Thanks,
  Josh
 
 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.