You can set the following entry inside the conf/spark-defaults.conf file
spark.cores.max 16
If you want to read the default value, then you can use the following api
call
*sc*.defaultParallelism
where *sc* is your sparkContext object.
Thanks
Best Regards
On Sun, Nov 9, 2014 at 6:48 PM, ReticulatedPython <person.of.b...@gmail.com>
wrote:
> So, I'm running this simple program on a 16 core multicore system. I run it
> by issuing the following.
>
> spark-submit --master local[*] pi.py
>
> And the code of that program is the following. When I use top to see CPU
> consumption, only 1 core is being utilized. Why is it so? Seconldy, spark
> documentation says that the default parallelism is contained in property
> spark.default.parallelism. How can I read this property from within my
> python program?
>
> #"""pi.py"""
> from pyspark import SparkContext
> import random
>
> NUM_SAMPLES = 12500000
>
> def sample(p):
> x, y = random.random(), random.random()
> return 1 if x*x + y*y < 1 else 0
>
> sc = SparkContext("local", "Test App")
> count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a,
> b: a + b)
> print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Why-does-this-siimple-spark-program-uses-only-one-core-tp18434.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
