It depends on what you want to do: If, on any given server, you don't want Spark to use more than one core, use this to start the workers: SPARK_HOME/sbin/start-slave.sh --cores=1
If you have a bunch of servers dedicated to Spark, but you don't want a driver to use more than one core per server, then: spark.executor.cores=1 tells it not to use more than 1 core per server. However, it seems it will start as many pyspark as there are cores, but maybe not use them. On Mon, Jul 4, 2016 at 10:44 AM Ashwin Raaghav <ashraag...@gmail.com> wrote: > Hi Mathieu, > > Isn't that the same as setting "spark.executor.cores" to 1? And how can I > specify "--cores=1" from the application? > > On Mon, Jul 4, 2016 at 8:06 PM, Mathieu Longtin <math...@closetwork.org> > wrote: > >> When running the executor, put --cores=1. We use this and I only see 2 >> pyspark process, one seem to be the parent of the other and is idle. >> >> In your case, are all pyspark process working? >> >> On Mon, Jul 4, 2016 at 3:15 AM ar7 <ashraag...@gmail.com> wrote: >> >>> Hi, >>> >>> I am currently using PySpark 1.6.1 in my cluster. When a pyspark >>> application >>> is run, the load on the workers seems to go more than what was given. >>> When I >>> ran top, I noticed that there were too many Pyspark.daemons processes >>> running. There was another mail thread regarding the same: >>> >>> >>> https://mail-archives.apache.org/mod_mbox/spark-user/201606.mbox/%3ccao429hvi3drc-ojemue3x4q1vdzt61htbyeacagtre9yrhs...@mail.gmail.com%3E >>> >>> I followed what was mentioned there, i.e. reduced the number of executor >>> cores and number of executors in one node to 1. But the number of >>> pyspark.daemons process is still not coming down. It looks like initially >>> there is one Pyspark.daemons process and this in turn spawns as many >>> pyspark.daemons processes as the number of cores in the machine. >>> >>> Any help is appreciated :) >>> >>> Thanks, >>> Ashwin Raaghav. >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Limiting-Pyspark-daemons-tp27272.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> -- >> Mathieu Longtin >> 1-514-803-8977 >> > > > > -- > Regards, > > Ashwin Raaghav > -- Mathieu Longtin 1-514-803-8977
