Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "HCFS/Progress" page has been changed by SteveWatt: https://wiki.apache.org/hadoop/HCFS/Progress?action=diff&rev1=10&rev2=11 - '''Proposal for organizing the work''' + '''HCFS Workstream Definition''' - If our eventual goal is to create a test suite that comprehensively vets adherence of a given Hadoop FileSystem implementation to the FileSystem contract and the expected behavior of its operations, then I think we need to define what "comprehensive" means. We should also decide on whether these tests are going to be against the FileSystem 1.0 or 2.0 interface. + As agreed to by June 10th Meeting participants: - My proposal is to organize the work into the following activities: + * Focus on Hadoop 2.0 FS Interface. If possible, create a work stream that would allow testing and validation of the FS 1.0 Interface. * An audit of the Hadoop FileSystem 1.0 Test Coverage - [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests | I have already created a first pass at this]] @@ -12, +12 @@ * An audit of the new Hadoop FS Tests added by Steve Loughran for his [[https://issues.apache.org/jira/browse/HADOOP-8545 | Hadoop FS Plugin for SWIFT]] - * A document codifying the expected semantics/behavior of the FileSystem 2.0 Operations - [[ https://issues.apache.org/jira/browse/HADOOP-9371 | Steve Loughran has started this already]] + * Create JavaDocs reflecting a FileSystem 2.0 Spec that codifying the expected semantics/behavior of the FileSystem 2.0 Operations and all the FS operations - [[ https://issues.apache.org/jira/browse/HADOOP-9371 | Steve Loughran has started this already]] - * A gap analysis that examines the FileSystem 2.0 Class, the expected behavior of the Operations and the respective Test Coverage available. + * Create a gap analysis that examines the FileSystem 2.0 Class, the expected behavior of the Operations and the respective Test Coverage available. * Create tests to fill in the gaps + + - Create a workstream to identify if Object/Blob stores have unique properties that make them a special case for Test Coverage as a Hadoop FS. Create a strategy for handling Object/Block Stores. + + * Validation that a given Hadoop FileSystem implementation is compatible would involve: + + - Functional Validation: Successfully passing the test library that will be created (described above) + + - Ecosystem Validation: Successfully passing the Hadoop Integration Tests from Apache BigTop + ---- Next Meeting - June 10th 9AM PST - Google Hangout - https://plus.google.com/hangouts/_/6b3ac2351f2a794717677be7c29a21599c0433ab?authuser=0&hl=en + '''June 25th''' a Red Hat in Mountain View. The day before Hadoop Summmit. More details to follow. + + ---- + ''Work thus far'' + + ---- + '''June 10th''' 9AM PST via Google Hangout + + Attendees: Tim St. Clair, Matt Farrellee, Steve Watt, Jay Vyas, Steve Loughran, Sanjay Radia, Andrew Purtell, Joe Buck, Roman Shaposhnik, Nathan (?) Agenda: @@ -32, +48 @@ - Discussion on where people would like to participate + Outcome: + + - Validation of the current goals, plus the addition of: + + * Leveraging BigTop in order to make use of the Hadoop Ecosystem Tests as additional Test Coverage for a FileSystem Implementation + + * Create a workstream to identify if Object/Blob stores have unique properties that make them a special case for Test Coverage as a Hadoop FS. Create a strategy for handling Object/Block Stores. + + * Focus tests against the AbstractFileSystem class rather than the FileSystem Class (which is an abstract class). Yes, this can be confusing. + + * Create a Hadoop 2.0 FileSystem Interface Specification for developers creating plugins as well as additional background for interested users. This should be created as a JavaDoc and managed in JIRA so that it supports proper governance. + + The workstream definition at the top of this page has been updated to reflect the new additions to the initiative. + ---- + '''June 4th''' - ''Work thus far'' - ---- - June 4th Created a [[https://github.com/wattsteve/HCFS/blob/master/jdiff/Report.txt | diff report]] contrasting Hadoop FileSystem 1.0 and 2.0 Next step is to evaluate how comprehensive the unit test case coverage is for FileSystem 1.0 and 2.0. This is a work in progress [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests | Audit of the FileSystem 1.0 Test Library ]] ---- - May 23rd - A broader call for participation was made to the hadoop-core dev proposing: + '''May 23rd''' - A broader call for participation was made to the hadoop-core dev proposing: * broader participation in [[ https://issues.apache.org/jira/browse/HADOOP-9371 | defining the expected behavior of Hadoop FileSystem operations]]
