On 7 March 2014 01:35, Jay Vyas jayunit...@gmail.com wrote:
Thanks steve. So i guess the conclusion is
1) Wait on HADOOP-9361.
help with is a better plan. I really don't look at it that often. I'll
try and get it ready to review this weekend
2) There definitively cannot be a strict
As part of HADOOP-9361, im visioning this.
1) - We create In Memory FS implementation of different Reference
FileSystems, each of which specifies appropriate tests , and passes those
tests , i.e.
InMemStrictlyConsistentFS (i.e. hdfs)
InMemEventuallyConsistentFS (blob stores)
On 6 March 2014 16:37, Jay Vyas jayunit...@gmail.com wrote:
As part of HADOOP-9361, im visioning this.
1) - We create In Memory FS implementation of different Reference
FileSystems, each of which specifies appropriate tests , and passes those
tests , i.e.
InMemStrictlyConsistentFS (i.e.
do you consider that native S3 FS a real reference implementation for
blob stores? or just something that , by mere chance, we are able to use as
a ref. impl.
NetFlix's Apache-licensed S3mper system provides consistency for an
S3-backed store.
http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html
It would be nice to see this or something like it integrated with
Hadoop. I fear that a lot of applications are not ready for eventual
Thanks Colin: that's a good example of why we want To unify the hcfs test
profile. So how can hcfs implementations use current hadoop-common tests?
In mind there are three ways.
- one solution is to manually cobble together and copy tests , running them one
by one and seeing which ones apply
EMR's S3 does extra things, which is why netflix used injection tricks to
add theirs on top.
For blobstores, key use cases are
1. -general source of low-rate-of-change artifacts
2. -input for analysis jobs
3. -output from them
4. -chained operations
5. storage of data to outlive
Lets get the HADOOP-9361 stuff in (it lives alongside
FileSystemContractBaseTest) and you can work off that.
On 6 March 2014 18:57, Jay Vyas jayunit...@gmail.com wrote:
Thanks Colin: that's a good example of why we want To unify the hcfs test
profile. So how can hcfs implementations use
Thanks steve. So i guess the conclusion is
1) Wait on HADOOP-9361.
2) There definitively cannot be a strict contract for a single HCFS, based
on your examples shown.
In the meantime ill audit existing test coverage, and let me know if i can
lend a hand in the cleanup process.
On Thu, Mar
