Hi Mich, I have only 32 cores, I have tested with 2 GB of memory per worker to force spills to disk. My application had 12 cores and 3 cores per executor.
Thank you very much. Luca Da: Mich Talebzadeh [mailto:[email protected]] Inviato: venerdì 15 aprile 2016 18:56 A: Luca Guerra <[email protected]> Cc: user @spark <[email protected]> Oggetto: Re: How many disks for spark_local_dirs? Is that 32 CPUs or 32 cores? So in this configuration assuming 32 cores you have I worker with how much memory (deducting memory for OS etc) and 32 cores. What is the ratio of memory per core in this case? HTH Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/> On 15 April 2016 at 16:15, luca_guerra <[email protected]<mailto:[email protected]>> wrote: Hi, I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html: "We recommend having 4-8 disks per node", I have tried both with one and two disks but I have seen that with 2 disks the execution time is doubled. Any explanations about this? This is my configuration: 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual configuration) and on this I have a standalone Spark cluster with 1 Worker. Thank you very much for the help. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for-spark-local-dirs-tp26790.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected]<mailto:[email protected]> For additional commands, e-mail: [email protected]<mailto:[email protected]>
