Hi All,
I have a machine with the following configuration:
32 GB RAM
500 GB HDD
8 CPUs
Following are the parameters i'm starting my Spark context with:
val conf = new
SparkConf().setAppName("MasterApp").setMaster("local[1]").set("spark.executor.memory",
"20g")
I'm reading a 4.3 GB file and counting the no. of characters in it.
When i run my program with:
local[1], the count is returned in 1.8 minutes
local[8], the count is returned in 4.2 minutes
Both the times, Spark uses Storage Memory of 3.7 GB.
Why would my program take more time with local[8]?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-master-takes-more-time-with-local-8-than-local-1-tp26052.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
