I have two nodes with 96G ram 16 cores, my setup is as follows: conf = (SparkConf() .setMaster("yarn-cluster") .set("spark.executor.memory", "30G") .set("spark.cores.max", 32) .set("spark.executor.instances", 2) .set("spark.executor.cores", 8) .set("spark.akka.timeout", 10000) .set("spark.akka.askTimeout", 100) .set("spark.akka.frameSize", 500) .set("spark.cleaner.ttl", 86400) .set("spark.tast.maxFailures", 16) .set("spark.worker.timeout", 150)
thanks a lot, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Any-issues-with-repartition-tp13462p15674.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org