Re: Num of executors and cores

2016-07-26 Thread Mail.com
>>> >>>> On Tue, Jul 26, 2016 at 2:18 AM, Mail.com <pradeep.mi...@mail.com> wrote: >>>> Hi All, >>>> >>>> I have a directory which has 12 files. I want to read the entire file so I >>>> am reading it as wholeTextFiles(dirpat

Re: Num of executors and cores

2016-07-26 Thread Jacek Laskowski
com> wrote: >>> Hi All, >>> >>> I have a directory which has 12 files. I want to read the entire file so I >>> am reading it as wholeTextFiles(dirpath, numPartitions). >>> >>> I run spark-submit as --num-executors 12 --executor-cores &

Re: Num of executors and cores

2016-07-26 Thread Mail.com
have a directory which has 12 files. I want to read the entire file so I >> am reading it as wholeTextFiles(dirpath, numPartitions). >> >> I run spark-submit as --num-executors 12 --executor-cores >> 1 and numPartitions 12. >> >> However, when I run the jo

Re: Num of executors and cores

2016-07-26 Thread Jacek Laskowski
il.com> wrote: > Hi All, > > I have a directory which has 12 files. I want to read the entire file so I am > reading it as wholeTextFiles(dirpath, numPartitions). > > I run spark-submit as --num-executors 12 --executor-cores 1 > and numPartitions 12. > > However, wh

Num of executors and cores

2016-07-25 Thread Mail.com
Hi All, I have a directory which has 12 files. I want to read the entire file so I am reading it as wholeTextFiles(dirpath, numPartitions). I run spark-submit as --num-executors 12 --executor-cores 1 and numPartitions 12. However, when I run the job I see that the stage which reads