Hi, I have a four single core machines as slaves in my cluster. I set the spark.default.parallelism to 4 and ran SparkTC given in examples. It took around 26 sec. Now, I increased the spark.default.parallelism to 8, but my performance deteriorates. The same application takes 32 sec now. I have read that usually the best performance is obtained when the parallelism is set to 2X of the number of cores available. I do not quite understand this. Could anyone please tell me?
Thank You Regards, Deep
