[
https://issues.apache.org/jira/browse/HADOOP-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10461:
------------------------------
Summary: Create an extensible single point entry for of HCFS tests - using
RawLocalFileSystem as example impl. (was: Create an extensible single point of
HCFS tests - using RawLocalFileSystem as example impl.)
> Create an extensible single point entry for of HCFS tests - using
> RawLocalFileSystem as example impl.
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-10461
> URL: https://issues.apache.org/jira/browse/HADOOP-10461
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: jay vyas
> Priority: Minor
>
> Currently alot of manual inheritance and stub classes are required in order
> to run the FileSystemBaseContract and FSMainOperations tests. This means
> that the HCFS Test suite for any given FileSystem spans many classes, and
> makes it confusing and cumbersome for anyone to create new tests for a file
> system, and also very hard to audit and update the tests for a particular
> file system.
> *If each FileSystem had a single point of entry for its tests, we would be
> much better off.*
> There are two ways to do this.
> * Have a folder or package for each file system (i.e.
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal ) Or
> * Wrap all the HCFS Test implementation classes into a single class (i.e.
> RawLocalFileSystemHCFSTestSuite.java).
> *The purpose of this JIRA is to implement a wrapper class for the
> RawLocalFileSystem, which serves as a single point of entry to ALL
> RawLocalFileSystem tests. This will cleanup the RawLocalFileSystem tests
> while providing other HCFS implementors a template they can use for their own
> filesystems (i.e. they can just copy the code and customize it however they
> want).*
> For example: This is a sampling of the classes necessary to glue Local FS
> implementations into the Existing generic FileSystem test classes:
> {noformat}
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
> {noformat}
> Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any
> once can copy that suite to build tests for their own file system.
> Using the "Enclosed" Junit
> (http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html
> ) class, we can combine all of the RawLocalFileSystem tests into a single
> class. That way
> * Can save on boiler plate for setup and declaring the same FileSystem stubs
> over and over again, making the test "logic" more easy to focus on. For
> example, here are some repeat file context declarations for ViewFS tests:
> {noformat}
> 01:42:08 {TRUNK} $ git grep
> "ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper)"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java:
> fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java:
> fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java:
> return ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> 01:45:19 {TRUNK} $ git grep "
> FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsURIs.java:
> FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java:
> fcView = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java:
> FileContext fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> {noformat}
> * Can easily be adopted for other FileSystems.
> * Can read in System properties to skip certain tests , thus providing
> support for the type of variability that we know FileSystem tests require.
> These can then be used to implement the semantics of HADOOP-9361.
> Ideally, we could replace RawLocalFileSystem tests with this injector as a
> second follow up patch to this, it would reduce the overall amount of code
> required, probably.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
