Hi All, I did worked on spark installed on Hadoop cluster but never worked on spark on standalone cluster.
My question how to set number of partitions in spark when it's running on spark standalone cluster? If spark on Hadoop I calculate my formula using hdfs block sizes but how I calculate without hdfs block size if spark running on standalone non Hadoop cluster. Partition formula for 100gb file:- Hdfs block size:-256 100*1024 =400 partitions /256 Executors:- 100/4= 25 Executor memory:- 160gb/25=7 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-partition-formula-on-standalone-mode-tp27237.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org