Assume your block size is 128MB. Thanks Best Regards
On Mon, May 4, 2015 at 2:38 PM, ayan guha <guha.a...@gmail.com> wrote: > Hi > > How do you figure out 500gig~3900 partitions? I am trying to do the math. > If I assume 64mb block size then 1G~16 blocks and 500g~8000 blocks. If we > assume split and block sizes are same, shouldn't we end up with 8k > partitions? > On 4 May 2015 17:49, "Akhil Das" <ak...@sigmoidanalytics.com> wrote: > >> 500GB of data will have nearly 3900 partitions and if you can have nearly >> that many number of cores and around 500GB of memory then things will be >> lightening fast. :) >> >> Thanks >> Best Regards >> >> On Sun, May 3, 2015 at 12:49 PM, sherine ahmed < >> sherine.sha...@hotmail.com> wrote: >> >>> I need to use spark to upload a 500 GB data from hadoop on standalone >>> mode >>> cluster what are the minimum hardware requirements if it's known that it >>> will be used for advanced analysis (social network analysis)? >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >>
