Thanks for the reply. Yeah I found the same doc and am able to use multiple
cores in spark-shell, however, when I use pyspark, it appears only to use
one core, I am wondering if this is something I did't configure correctly
or something supported in pyspark.
On Fri, Mar 24, 2017 at 3:52 PM, Kadam, Gangadhar (GE Aviation, Non-GE) <
gangadhar.ka...@ge.com> wrote:
> In Local Mode all processes are executed inside a single JVM.
> Application is started in a local mode by setting master to local,
> local[*] or local[n].
> spark.executor.cores and spark.executor.cores are not applicable in the
> local mode because there is only one embedded executor.
>
>
> In Standalone mode, you need standalone Spark cluster<
> https://spark.apache.org/docs/latest/spark-standalone.html>.
>
> It requires a master node (can be started using
> SPARK_HOME/sbin/start-master.sh script) and at least one worker node (can
> be started using SPARK_HOME/sbin/start-slave.sh script).SparkConf should
> use master node address to create (spark://host:port)
>
> Thanks!
>
> Gangadhar
> From: Li Jin <ice.xell...@gmail.com<mailto:ice.xell...@gmail.com>>
> Date: Friday, March 24, 2017 at 3:43 PM
> To: "user@spark.apache.org<mailto:user@spark.apache.org>" <
> user@spark.apache.org<mailto:user@spark.apache.org>>
> Subject: EXT: Multiple cores/executors in Pyspark standalone mode
>
> Hi,
>
> I am wondering does pyspark standalone (local) mode support multi
> cores/executors?
>
> Thanks,
> Li
>
