The latter would be faster. With S3, you want to maximize number of concurrent readers until you hit your network throughput limits.
On Wed, Feb 4, 2015 at 6:20 AM, Peter Rudenko <petro.rude...@gmail.com> wrote: > Hi if i have a 10GB file on s3 and set 10 partitions, would it be > download whole file on master first and broadcast it or each worker would > just read it's range from the file? > > Thanks, > Peter > > On 2015-02-03 23:30, Sven Krasser wrote: > > Hey Joe, > > With the ephemeral HDFS, you get the instance store of your worker nodes. > For m3.xlarge that will be two 40 GB SSDs local to each instance, which are > very fast. > > For the persistent HDFS, you get whatever EBS volumes the launch script > configured. EBS volumes are always network drives, so the usual limitations > apply. To optimize throughput, you can use EBS volumes with provisioned > IOPS and you can use EBS optimized instances. I don't have hard numbers at > hand, but I'd expect this to be noticeably slower than using local SSDs. > > As far as only using S3 goes, it depends on your use case (i.e. what you > plan on doing with the data while it is there). If you store it there in > between running different applications, you can likely work around > consistency issues. > > Also, if you use Amazon's EMRFS to access data in S3, you can use their > new consistency feature ( > https://aws.amazon.com/blogs/aws/emr-consistent-file-system/). > > Hope this helps! > -Sven > > > On Tue, Feb 3, 2015 at 9:32 AM, Joe Wass <jw...@crossref.org> wrote: > >> The data is coming from S3 in the first place, and the results will be >> uploaded back there. But even in the same availability zone, fetching 170 >> GB (that's gzipped) is slow. From what I understand of the pipelines, >> multiple transforms on the same RDD might involve re-reading the input, >> which very quickly add up in comparison to having the data locally. Unless >> I persisted the data (which I am in fact doing) but that would involve >> storing approximately the same amount of data in HDFS, which wouldn't fit. >> >> Also, I understood that S3 was unsuitable for practical? See "Why you >> cannot use S3 as a replacement for HDFS"[0]. I'd love to be proved wrong, >> though, that would make things a lot easier. >> >> [0] http://wiki.apache.org/hadoop/AmazonS3 >> >> >> >> On 3 February 2015 at 16:45, David Rosenstrauch <dar...@darose.net> >> wrote: >> >>> You could also just push the data to Amazon S3, which would un-link the >>> size of the cluster needed to process the data from the size of the data. >>> >>> DR >>> >>> >>> On 02/03/2015 11:43 AM, Joe Wass wrote: >>> >>>> I want to process about 800 GB of data on an Amazon EC2 cluster. So, I >>>> need >>>> to store the input in HDFS somehow. >>>> >>>> I currently have a cluster of 5 x m3.xlarge, each of which has 80GB >>>> disk. >>>> Each HDFS node reports 73 GB, and the total capacity is ~370 GB. >>>> >>>> If I want to process 800 GB of data (assuming I can't split the jobs >>>> up), >>>> I'm guessing I need to get persistent-hdfs involved. >>>> >>>> 1 - Does persistent-hdfs have noticeably different performance than >>>> ephemeral-hdfs? >>>> 2 - If so, is there a recommended configuration (like storing input and >>>> output on persistent, but persisted RDDs on ephemeral?) >>>> >>>> This seems like a common use-case, so sorry if this has already been >>>> covered. >>>> >>>> Joe >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> > > > -- > http://sites.google.com/site/krasser/?utm_source=sig > > >
