On 7 March 2014 01:35, Jay Vyas <jayunit...@gmail.com> wrote: > Thanks steve. So i guess the conclusion is > > 1) Wait on HADOOP-9361. >
"help" with is a better plan. I really don't look at it that often. I'll try and get it ready to review this weekend > 2) There definitively cannot be a strict contract for a single HCFS, based > on your examples shown. > > HDFS effectively defines the behaviour, though it is a defacto specification. All the tests and docs we can do can formalise the behaviour, and show other filesystems (including file://) where they are inconsistent. Failure modes are intractable -it is everyone's right to fail differently. What worries me is that there are some unintentional behaviours -epiphenomena- that we aren't aware of, but which downstream code depends on. mkdirs() being an atomic is an example -nobody made a decision to do that, it just fell out of holding locks efficiently in the NN. Does anything depend on int? Hopefully not -as if it does, that's something HDFS needs to keep forever -so there'd better be a test for it. Any others? I don't know -and that worries me. Filename, dir size and file size assumptions are things that caused problems in the swift object store, code also assumes that rmdir is O(1) and not O(n), so teardown of tests on directories with many small files caused timeouts against throttled Swift endpoints. > In the meantime ill audit existing test coverage, and let me know if i can > lend a hand in the cleanup process. > That'll be great -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
