Hi, kodali. SPARK_WORKER_CORES is designed for cluster resource manager, see http://spark.apache.org/docs/latest/cluster-overview.html if interested.
For standalone mode, you should use the following 3 arguments to allocate resource for normal spark tasks: - --executor-memory - --executor-cores - --total-executor-cores and the meaning is as below: - Executor memory: --executor-memory - Executor cores: --executor-cores - Number of executors: --total-executor-cores/--executor-cores more details see http://spark.apache.org/docs/latest/submitting-applications.html. On Sat, Feb 18, 2017 at 9:20 AM, kant kodali <[email protected]> wrote: > Hi Satish, > > I am using spark 2.0.2. And no I have not passed those variables because > I didn't want to shoot in the dark. According to the documentation it looks > like SPARK_WORKER_CORES is the one which should do it. If not, can you > please explain how these variables inter play together? > > --num-executors > --executor-cores > –total-executor-cores > SPARK_WORKER_CORES > > Thanks! > > > On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam <[email protected]> > wrote: > >> Have you tried passing --executor-cores or –total-executor-cores as >> arguments, , depending on the spark version? >> >> >> >> >> >> *From:* kant kodali [mailto:[email protected]] >> *Sent:* Friday, February 17, 2017 5:03 PM >> *To:* Alex Kozlov <[email protected]> >> *Cc:* user @spark <[email protected]> >> *Subject:* Re: question on SPARK_WORKER_CORES >> >> >> >> Standalone. >> >> >> >> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <[email protected]> wrote: >> >> What Spark mode are you running the program in? >> >> >> >> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <[email protected]> wrote: >> >> when I submit a job using spark shell I get something like this >> >> >> >> [Stage 0:========> (36814 + 4) / 220129] >> >> >> >> Now all I want is I want to increase number of parallel tasks running >> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in >> conf/spark-env.sh. I though that should do it but it doesn't. It still >> shows me 4. any idea? >> >> >> >> Thanks much! >> >> >> >> >> >> -- >> >> Alex Kozlov >> (408) 507-4987 >> (650) 887-2135 efax >> [email protected] >> >> >> > >
