[
https://issues.apache.org/jira/browse/HADOOP-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated HADOOP-10461:
------------------------------
Description:
Currently alot of manual inheritance and stub classes are required in order to
run the FileSystemBaseContract and FSMainOperations tests. This means that
the HCFS Test suite for any given FileSystem spans many classes, and makes it
confusing and cumbersome for anyone to create new tests for a file system, and
also very hard to audit and update the tests for a particular file system.
*If each FileSystem had a single point of entry for its tests, we would be much
better off.*
There are two ways to do this.
* Have a folder or package for each file system (i.e.
.//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal ) Or
* Wrap all the HCFS Test implementation classes into a single class (i.e.
RawLocalFileSystemHCFSTestSuite.java).
*The purpose of this JIRA is to implement (2) for the RawLocalFileSystem. This
will cleanup the RawLocalFileSystem tests while providing other HCFS
implementors a template they can use for their own filesystems (i.e. they can
just copy the code and customize it however they want).*
For example: This is a sampling of the classes necessary to glue Local FS
implementations into the Existing generic FileSystem test classes:
{noformat}
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
{noformat}
Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any
once can copy that suite to build tests for their own file system.
Using the "Enclosed" Junit
(http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html
) class, we can combine all of the RawLocalFileSystem tests into a single
class. That way
* Can save on boiler plate for setup and declaring the same FileSystem stubs
over and over again, making the test "logic" more easy to focus on. For
example, here are some repeat file context declarations for ViewFS tests:
{noformat}
01:42:08 {TRUNK} $ git grep
"ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper)"
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java:
fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java:
fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java:
return ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
01:45:19 {TRUNK} $ git grep "
FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);"
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsURIs.java:
FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java:
fcView = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java:
FileContext fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
{noformat}
* Can easily be adopted for other FileSystems.
* Can read in System properties to skip certain tests , thus providing support
for the type of variability that we know FileSystem tests require. These can
then be used to implement the semantics of HADOOP-9361.
Ideally, we could replace RawLocalFileSystem tests with this injector as a
second follow up patch to this, it would reduce the overall amount of code
required, probably.
was:
Currently alot of manual inheritance and stub classes are required in order to
run the FileSystemBaseContract and FSMainOperations tests. This means that
the HCFS Test suite for any given FileSystem spans many classes, and makes it
confusing and cumbersome for anyone to create new tests for a file system, and
also very hard to audit and update the tests for a particular file system.
*If each FileSystem had a single point of entry for its tests, we would be much
better off.*
There are two ways to do this.
* Have a folder or package for each file system (i.e.
.//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal ) Or
* Wrap all the HCFS Test implementation classes into a single class (i.e.
RawLocalFileSystemHCFSTestSuite.java).
*The purpose of this JIRA is to implement (2) for the RawLocalFileSystem. This
will cleanup the RawLocalFileSystem tests while providing other HCFS
implementors a template they can use for their own filesystems (i.e. they can
just copy the code and customize it however they want).*
For example: This is a sampling of the classes necessary to glue Local FS
implementations into the Existing generic FileSystem test classes:
{noformat}
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
.//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
{noformat}
Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any
once can copy that suite to build tests for their own file system.
Using the "Enclosed" Junit
(http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html
) class, we can combine all of the RawLocalFileSystem tests into a single
class. That way
* Injects the file system at runtime.
* Can easily be adopted for other FileSystems.
* Can read in System properties to skip certain tests , thus providing support
for the type of variability that we know FileSystem tests require. These can
then be used to implement the semantics of HADOOP-9361.
Ideally, we could replace RawLocalFileSystem tests with this injector as a
second follow up patch to this, it would reduce the overall amount of code
required, probably.
> Runtime HCFS injector for FileSystem tests
> ------------------------------------------
>
> Key: HADOOP-10461
> URL: https://issues.apache.org/jira/browse/HADOOP-10461
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: jay vyas
> Priority: Minor
>
> Currently alot of manual inheritance and stub classes are required in order
> to run the FileSystemBaseContract and FSMainOperations tests. This means
> that the HCFS Test suite for any given FileSystem spans many classes, and
> makes it confusing and cumbersome for anyone to create new tests for a file
> system, and also very hard to audit and update the tests for a particular
> file system.
> *If each FileSystem had a single point of entry for its tests, we would be
> much better off.*
> There are two ways to do this.
> * Have a folder or package for each file system (i.e.
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal ) Or
> * Wrap all the HCFS Test implementation classes into a single class (i.e.
> RawLocalFileSystemHCFSTestSuite.java).
> *The purpose of this JIRA is to implement (2) for the RawLocalFileSystem.
> This will cleanup the RawLocalFileSystem tests while providing other HCFS
> implementors a template they can use for their own filesystems (i.e. they can
> just copy the code and customize it however they want).*
> For example: This is a sampling of the classes necessary to glue Local FS
> implementations into the Existing generic FileSystem test classes:
> {noformat}
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
> {noformat}
> Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any
> once can copy that suite to build tests for their own file system.
> Using the "Enclosed" Junit
> (http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html
> ) class, we can combine all of the RawLocalFileSystem tests into a single
> class. That way
> * Can save on boiler plate for setup and declaring the same FileSystem stubs
> over and over again, making the test "logic" more easy to focus on. For
> example, here are some repeat file context declarations for ViewFS tests:
> {noformat}
> 01:42:08 {TRUNK} $ git grep
> "ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper)"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java:
> fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java:
> fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java:
> return ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> 01:45:19 {TRUNK} $ git grep "
> FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsURIs.java:
> FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java:
> fcView = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java:
> FileContext fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> {noformat}
> * Can easily be adopted for other FileSystems.
> * Can read in System properties to skip certain tests , thus providing
> support for the type of variability that we know FileSystem tests require.
> These can then be used to implement the semantics of HADOOP-9361.
> Ideally, we could replace RawLocalFileSystem tests with this injector as a
> second follow up patch to this, it would reduce the overall amount of code
> required, probably.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
