[
https://issues.apache.org/jira/browse/BIGTOP-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913849#comment-13913849
]
jay vyas edited comment on BIGTOP-1221 at 2/27/14 1:10 AM:
-----------------------------------------------------------
OKay here it is ! Some hardcore Fuse tests for HDFS (oops did i say that ... i
meant HCFS :) :) :) ). Also the framework is easy to expand. To add a new
test:
1) Add a shell setup command (i.e. "touch a")
2) Add a test command (i.e. "ls *")
3) Add a labmda to evalueate the test (i.e.
"sh.getOut().toString().contains("a"))"
This will pave the way for a much more comprehensive smoke test, and will work
for any HCFS as long as they create a fuse mount. For HDFS, ive confirmed it
works as-is :
{noformat}
[root@vagrant bigtop]# groovy -classpath
/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/root/.m2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/etc/hadoop/conf/
/vagrant/TestFuseDFS2.groovy
mounting dfs://vagrant.bigtop1:17020 on /tmp/hcfs-test
[/tmp/hcfs-test/user/root/TestFuseDFS-testDir/dir1]
[file1]
/bin/cp -rf /tmp/FUSETEST_bigtop
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cf2 out : []
NOW IN THE VALIDATOR FUNCTION .....................
cat output = [hi_bigtop, hi_bigtop] [] 0
After cd, pwd=/tmp/hcfs-test/user/root/TestFuseDFS-testDir
mkdir /tmp/hcfs-test/user/root/TestFuseDFS-testDir/targetdir && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cp1 && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cp2 out : []
touch /tmp/hcfs-test/user/root/TestFuseDFS-testDir/a out : []
[total 8.0K, -rw-r--r-- 1 root hadoop 0 Feb 27 01:10 a, drwxrwxrwx 10 root
hadoop 4.0K Feb 27 01:10 .., drwxr-xr-x 3 root hadoop 4.0K Feb 27 01:10 .]
mkdir -p /tmp/hcfs-test/user/root/TestFuseDFS-testDir/subdir1 && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/subdir1/innerfile out : []
touch /tmp/hcfs-test/user/root/TestFuseDFS-testDir/file-removed out : []
JUnit 4 Runner, Tests: 8, Failures: 0, Time: 5533
{noformat}
was (Author: jayunit100):
OKay here it is ! Some hardcore Fuse tests for HDFS. Also the framework is
easy to expand. To add a new test:
1) Add a shell setup command (i.e. "touch a")
2) Add a test command (i.e. "ls *")
3) Add a labmda to evalueate the test (i.e.
"sh.getOut().toString().contains("a"))"
This will pave the way for a much more comprehensive smoke test, and will work
for any HCFS as long as they create a fuse mount. For HDFS, ive confirmed it
works as-is :
{noformat}
[root@vagrant bigtop]# groovy -classpath
/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/root/.m2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/etc/hadoop/conf/
/vagrant/TestFuseDFS2.groovy
mounting dfs://vagrant.bigtop1:17020 on /tmp/hcfs-test
[/tmp/hcfs-test/user/root/TestFuseDFS-testDir/dir1]
[file1]
/bin/cp -rf /tmp/FUSETEST_bigtop
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cf2 out : []
NOW IN THE VALIDATOR FUNCTION .....................
cat output = [hi_bigtop, hi_bigtop] [] 0
After cd, pwd=/tmp/hcfs-test/user/root/TestFuseDFS-testDir
mkdir /tmp/hcfs-test/user/root/TestFuseDFS-testDir/targetdir && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cp1 && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/cp2 out : []
touch /tmp/hcfs-test/user/root/TestFuseDFS-testDir/a out : []
[total 8.0K, -rw-r--r-- 1 root hadoop 0 Feb 27 01:10 a, drwxrwxrwx 10 root
hadoop 4.0K Feb 27 01:10 .., drwxr-xr-x 3 root hadoop 4.0K Feb 27 01:10 .]
mkdir -p /tmp/hcfs-test/user/root/TestFuseDFS-testDir/subdir1 && touch
/tmp/hcfs-test/user/root/TestFuseDFS-testDir/subdir1/innerfile out : []
touch /tmp/hcfs-test/user/root/TestFuseDFS-testDir/file-removed out : []
JUnit 4 Runner, Tests: 8, Failures: 0, Time: 5533
{noformat}
> Expand and updated FUSE tests
> -----------------------------
>
> Key: BIGTOP-1221
> URL: https://issues.apache.org/jira/browse/BIGTOP-1221
> Project: Bigtop
> Issue Type: Bug
> Reporter: jay vyas
> Attachments: BIGTOP-1221.patch
>
>
> The BigTop TestFuseDFS need
> - tests for consistency after individual writes : consistency in FUSE mounts
> is important since there are often associated multitenant workloads. Writes
> should result in immediate recognition of a file.
> - Should use HCFS semantics (That way we can confidently use and maintain
> this for alternative file system implementations).
> - should use logging instead of System.out. That way all the logs can be
> picked up from the same place in case we change the logging framework.
> Logging in trace mode, in any case, will allow us to see everything that the
> ITest shell does, and so might as well use it for other logs.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
