Hi Jan,
It's a physical server, I have launched the application with:
- "spark.cores.max": "12",
- "spark.executor.cores": "3"
- 2 GB RAM per worker
Spark version is 1.6.0, I don't use Hadoop.
Thanks,
Luca
-----Messaggio originale-----
Da: Jan Rock [mailto:[email protected]]
Inviato: venerdì 15 aprile 2016 18:04
A: Luca Guerra <[email protected]>
Cc: [email protected]
Oggetto: Re: How many disks for spark_local_dirs?
Hi,
is it physical server or AWS/Azure? What are the executed parameters for
spark-shell command? Hadoop distro/version and Spark version?
Kind Regards,
Jan
> On 15 Apr 2016, at 16:15, luca_guerra <[email protected]> wrote:
>
> Hi,
> I'm looking for a solution to improve my Spark cluster performances, I
> have read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
> "We recommend having 4-8 disks per node", I have tried both with one
> and two disks but I have seen that with 2 disks the execution time is
> doubled. Any explanations about this?
>
> This is my configuration:
> 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an
> unusual
> configuration) and on this I have a standalone Spark cluster with 1 Worker.
>
> Thank you very much for the help.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-many-disks-for
> -spark-local-dirs-tp26790.html Sent from the Apache Spark User List
> mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected] For
> additional commands, e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
