EMR's S3 does extra things, which is why netflix used injection tricks to add theirs on top.
For blobstores, key use cases are 1. -general source of low-rate-of-change artifacts 2. -input for analysis jobs 3. -output from them 4. -chained operations 5. storage of data to outlive the EMR cluster #1 isn't a problem assuming the velocity of the artifacts is pretty low. #2 -OK for data written "a while" earlier, provided there isn't an ongoing partition. #3 -speculation relies on atomic rename that fails if dest dir exists. Blobstores don't have this and do rename as (i): check (iii) create root path (iii) copy of individual items below path (iv) delete of source. The race between (i) and (ii) exists, and if the object store doesn't even do create consistency (e.g AWS S3 US-East, but not the others [1]). This means there's a risk of two committing reducers mixing outputs (risk low, requires both processes to commit simultaneously) #4 is trouble -anything waiting for one MR job to finish may start when it finishes, but when job #2 kicks off and does an of the dir/path listing methods, it may get an incomplete list of children -and hence, incomplete list of output files. That's the trouble. If people follow the best practise -HDFS for intermediate work, S3 for final output, all is well. Netflix use S3 as the output of all work, so they can schedule analytics on any Hadoop cluster they have, and at the scale they run at they hit this problem. Other people may have -just not noticed. " I fear that a lot of applications are not ready for eventual consistency, and may never be" Exactly: i have code that uses HDFS to co-ordinate, and will never work on an object store that doesn't have atomic/consistent ops "leading to the feeling that Hadoop on S3 is buggy" https://issues.apache.org/jira/browse/HADOOP-9577 -filed by someone @amazon -Steve HADOOP-9565 says "add a marker" : https://issues.apache.org/jira/browse/HADOOP-9565 HADOOP-10373 goes further and says "move the s3 & s3n code into hadoop-tools/hadoop-aws https://issues.apache.org/jira/browse/HADOOP-10373 This will make it possible to swap in versions compiled against the same Hadoop release, without having to build your own hadoop JARs steve (who learned too much about object stores and the FileSystem class while doing the swift:// coding) [1] http://aws.amazon.com/s3/faqs/ On 6 March 2014 18:47, Colin McCabe <cmcc...@alumni.cmu.edu> wrote: > NetFlix's Apache-licensed S3mper system provides consistency for an > S3-backed store. > http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html > > It would be nice to see this or something like it integrated with > Hadoop. I fear that a lot of applications are not ready for eventual > consistency, and may never be, leading to the feeling that Hadoop on > S3 is buggy. > > Colin > > On Thu, Mar 6, 2014 at 10:42 AM, Jay Vyas <jayunit...@gmail.com> wrote: > > do you consider that native S3 FS a real "reference implementation" for > > blob stores? or just something that , by mere chance, we are able to use > as > > a ref. impl. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
