How to increase parallelism in Yarn

2014-12-18 Thread Suman Somasundar 
Hi,

 

I am using Spark 1.1.1 on Yarn. When I try to run K-Means, I see from the Yarn 
dashboard that only 3 containers are being used. How do I increase the number 
of containers used?

 

P.S: When I run K-Means on Mahout with the same settings, I see that there are 
25-30 containers being used.

 

Thanks,
Suman.

Re: How to increase parallelism in Yarn

2014-12-18 Thread Andrew Or 
Hi Suman,

I'll assume that you are using spark submit to run your application. You
can pass the --num-executors flag to ask for more containers. If you want
to allocate more memory for each executor, you may also pass in the
--executor-memory flag (this accepts a string in the format 1g, 512m etc.).

-Andrew

2014-12-18 14:37 GMT-08:00 Suman Somasundar suman.somasun...@oracle.com:

 Hi,



 I am using Spark 1.1.1 on Yarn. When I try to run K-Means, I see from the
 Yarn dashboard that only 3 containers are being used. How do I increase the
 number of containers used?



 P.S: When I run K-Means on Mahout with the same settings, I see that there
 are 25-30 containers being used.



 Thanks,
 Suman.