Re: question on SPARK_WORKER_CORES
Hi, kodali. SPARK_WORKER_CORES is designed for cluster resource manager, see http://spark.apache.org/docs/latest/cluster-overview.html if interested. For standalone mode, you should use the following 3 arguments to allocate resource for normal spark tasks: - --executor-memory - --executor-cores - --total-executor-cores and the meaning is as below: - Executor memory: --executor-memory - Executor cores: --executor-cores - Number of executors: --total-executor-cores/--executor-cores more details see http://spark.apache.org/docs/latest/submitting-applications.html. On Sat, Feb 18, 2017 at 9:20 AM, kant kodali wrote: > Hi Satish, > > I am using spark 2.0.2. And no I have not passed those variables because > I didn't want to shoot in the dark. According to the documentation it looks > like SPARK_WORKER_CORES is the one which should do it. If not, can you > please explain how these variables inter play together? > > --num-executors > --executor-cores > –total-executor-cores > SPARK_WORKER_CORES > > Thanks! > > > On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam > wrote: > >> Have you tried passing --executor-cores or –total-executor-cores as >> arguments, , depending on the spark version? >> >> >> >> >> >> *From:* kant kodali [mailto:kanth...@gmail.com] >> *Sent:* Friday, February 17, 2017 5:03 PM >> *To:* Alex Kozlov >> *Cc:* user @spark >> *Subject:* Re: question on SPARK_WORKER_CORES >> >> >> >> Standalone. >> >> >> >> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov wrote: >> >> What Spark mode are you running the program in? >> >> >> >> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: >> >> when I submit a job using spark shell I get something like this >> >> >> >> [Stage 0:>(36814 + 4) / 220129] >> >> >> >> Now all I want is I want to increase number of parallel tasks running >> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in >> conf/spark-env.sh. I though that should do it but it doesn't. It still >> shows me 4. any idea? >> >> >> >> Thanks much! >> >> >> >> >> >> -- >> >> Alex Kozlov >> (408) 507-4987 >> (650) 887-2135 efax >> ale...@gmail.com >> >> >> > >
Re: question on SPARK_WORKER_CORES
one executor per Spark slave should be fine right I am not really sure what benefit one would get by starting more executors (jvm's) on one node? End of the day JVM creates native/kernel threads through system calls so if those threads are spawned by one or multiple processes I dont see much benefit (In theory it should be the same). With different processes one would get different address spaces in the kernel but memory isn't an issue so far. On Fri, Feb 17, 2017 at 5:32 PM, Alex Kozlov wrote: > I found in some previous CDH versions that Spark starts only one executor > per Spark slave, and DECREASING the executor-cores in standalone makes > the total # of executors go up. Just my 2¢. > > On Fri, Feb 17, 2017 at 5:20 PM, kant kodali wrote: > >> Hi Satish, >> >> I am using spark 2.0.2. And no I have not passed those variables because >> I didn't want to shoot in the dark. According to the documentation it looks >> like SPARK_WORKER_CORES is the one which should do it. If not, can you >> please explain how these variables inter play together? >> >> --num-executors >> --executor-cores >> –total-executor-cores >> SPARK_WORKER_CORES >> >> Thanks! >> >> >> On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam >> wrote: >> >>> Have you tried passing --executor-cores or –total-executor-cores as >>> arguments, , depending on the spark version? >>> >>> >>> >>> >>> >>> *From:* kant kodali [mailto:kanth...@gmail.com] >>> *Sent:* Friday, February 17, 2017 5:03 PM >>> *To:* Alex Kozlov >>> *Cc:* user @spark >>> *Subject:* Re: question on SPARK_WORKER_CORES >>> >>> >>> >>> Standalone. >>> >>> >>> >>> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov wrote: >>> >>> What Spark mode are you running the program in? >>> >>> >>> >>> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: >>> >>> when I submit a job using spark shell I get something like this >>> >>> >>> >>> [Stage 0:>(36814 + 4) / 220129] >>> >>> >>> >>> Now all I want is I want to increase number of parallel tasks running >>> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in >>> conf/spark-env.sh. I though that should do it but it doesn't. It still >>> shows me 4. any idea? >>> >>> >>> >>> Thanks much! >>> >>> >>> >>> >>> >>> -- >>> >>> Alex Kozlov >>> (408) 507-4987 >>> (650) 887-2135 efax >>> ale...@gmail.com >>> >>> >>> >> >> > > > -- > Alex Kozlov > (408) 507-4987 > (650) 887-2135 efax > ale...@gmail.com >
Re: question on SPARK_WORKER_CORES
I found in some previous CDH versions that Spark starts only one executor per Spark slave, and DECREASING the executor-cores in standalone makes the total # of executors go up. Just my 2¢. On Fri, Feb 17, 2017 at 5:20 PM, kant kodali wrote: > Hi Satish, > > I am using spark 2.0.2. And no I have not passed those variables because > I didn't want to shoot in the dark. According to the documentation it looks > like SPARK_WORKER_CORES is the one which should do it. If not, can you > please explain how these variables inter play together? > > --num-executors > --executor-cores > –total-executor-cores > SPARK_WORKER_CORES > > Thanks! > > > On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam > wrote: > >> Have you tried passing --executor-cores or –total-executor-cores as >> arguments, , depending on the spark version? >> >> >> >> >> >> *From:* kant kodali [mailto:kanth...@gmail.com] >> *Sent:* Friday, February 17, 2017 5:03 PM >> *To:* Alex Kozlov >> *Cc:* user @spark >> *Subject:* Re: question on SPARK_WORKER_CORES >> >> >> >> Standalone. >> >> >> >> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov wrote: >> >> What Spark mode are you running the program in? >> >> >> >> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: >> >> when I submit a job using spark shell I get something like this >> >> >> >> [Stage 0:>(36814 + 4) / 220129] >> >> >> >> Now all I want is I want to increase number of parallel tasks running >> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in >> conf/spark-env.sh. I though that should do it but it doesn't. It still >> shows me 4. any idea? >> >> >> >> Thanks much! >> >> >> >> >> >> -- >> >> Alex Kozlov >> (408) 507-4987 >> (650) 887-2135 efax >> ale...@gmail.com >> >> >> > > -- Alex Kozlov (408) 507-4987 (650) 887-2135 efax ale...@gmail.com
Re: question on SPARK_WORKER_CORES
Hi Satish, I am using spark 2.0.2. And no I have not passed those variables because I didn't want to shoot in the dark. According to the documentation it looks like SPARK_WORKER_CORES is the one which should do it. If not, can you please explain how these variables inter play together? --num-executors --executor-cores –total-executor-cores SPARK_WORKER_CORES Thanks! On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam wrote: > Have you tried passing --executor-cores or –total-executor-cores as > arguments, , depending on the spark version? > > > > > > *From:* kant kodali [mailto:kanth...@gmail.com] > *Sent:* Friday, February 17, 2017 5:03 PM > *To:* Alex Kozlov > *Cc:* user @spark > *Subject:* Re: question on SPARK_WORKER_CORES > > > > Standalone. > > > > On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov wrote: > > What Spark mode are you running the program in? > > > > On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: > > when I submit a job using spark shell I get something like this > > > > [Stage 0:>(36814 + 4) / 220129] > > > > Now all I want is I want to increase number of parallel tasks running from > 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in > conf/spark-env.sh. I though that should do it but it doesn't. It still > shows me 4. any idea? > > > > Thanks much! > > > > > > -- > > Alex Kozlov > (408) 507-4987 > (650) 887-2135 efax > ale...@gmail.com > > >
RE: question on SPARK_WORKER_CORES
Have you tried passing --executor-cores or –total-executor-cores as arguments, , depending on the spark version? From: kant kodali [mailto:kanth...@gmail.com] Sent: Friday, February 17, 2017 5:03 PM To: Alex Kozlov Cc: user @spark Subject: Re: question on SPARK_WORKER_CORES Standalone. On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov mailto:ale...@gmail.com>> wrote: What Spark mode are you running the program in? On Fri, Feb 17, 2017 at 4:55 PM, kant kodali mailto:kanth...@gmail.com>> wrote: when I submit a job using spark shell I get something like this [Stage 0:>(36814 + 4) / 220129] Now all I want is I want to increase number of parallel tasks running from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in conf/spark-env.sh. I though that should do it but it doesn't. It still shows me 4. any idea? Thanks much! -- Alex Kozlov (408) 507-4987 (650) 887-2135 efax ale...@gmail.com<mailto:ale...@gmail.com>
Re: question on SPARK_WORKER_CORES
Standalone. On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov wrote: > What Spark mode are you running the program in? > > On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: > >> when I submit a job using spark shell I get something like this >> >> [Stage 0:>(36814 + 4) / 220129] >> >> >> Now all I want is I want to increase number of parallel tasks running >> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in >> conf/spark-env.sh. I though that should do it but it doesn't. It still >> shows me 4. any idea? >> >> >> Thanks much! >> >> >> > > > -- > Alex Kozlov > (408) 507-4987 > (650) 887-2135 efax > ale...@gmail.com >
Re: question on SPARK_WORKER_CORES
What Spark mode are you running the program in? On Fri, Feb 17, 2017 at 4:55 PM, kant kodali wrote: > when I submit a job using spark shell I get something like this > > [Stage 0:>(36814 + 4) / 220129] > > > Now all I want is I want to increase number of parallel tasks running from > 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in > conf/spark-env.sh. I though that should do it but it doesn't. It still > shows me 4. any idea? > > > Thanks much! > > > -- Alex Kozlov (408) 507-4987 (650) 887-2135 efax ale...@gmail.com
question on SPARK_WORKER_CORES
when I submit a job using spark shell I get something like this [Stage 0:>(36814 + 4) / 220129] Now all I want is I want to increase number of parallel tasks running from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in conf/spark-env.sh. I though that should do it but it doesn't. It still shows me 4. any idea? Thanks much!