[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542956#comment-13542956 ] Gary Gregory commented on VFS-442: -- Got it: I found the hdfs Maven profile. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch, VFS-442-5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542559#comment-13542559 ] Gary Gregory commented on VFS-442: -- I am reviewing the patch. So far, so good :) Thank you. It's been a long day, and I do not see it: Can you explain way the tests run on Linux and are ignored on Windows? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch, VFS-442-5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542635#comment-13542635 ] Dave Marion commented on VFS-442: - Gary, Take a look at the 4th comment. Hadoop doesn't really support Windows unless using Cygwin. I think you tried it and it did not work. You asked me to disable the tests when building on Windows. Dave HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch, VFS-442-5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542736#comment-13542736 ] Gary Gregory commented on VFS-442: -- Hi Dave, Yes, this is all good. I want to understand how the code works. When I run mvn test, what is the magic that causes the HDFS tests to be ignored on Windows and run on Linux? I can see the test fail on Windows with m3 test -Dtest=org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest; I just do not see the code that says don't run me on Windows. Thank you, Gary HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch, VFS-442-5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540986#comment-13540986 ] Dave Marion commented on VFS-442: - Uploaded Patch #5. Includes a fix for #1. I built the site and looked at the reports, I think #2 and #3 are also done. Please let me know if you see anything. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch, VFS-442-5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539744#comment-13539744
]
Dave Marion commented on VFS-442:
-
Started working on #1. I am running into a couple of issues. One seems to be
with the the doAttach method. The javadoc and code seem to indicate that it
should thrown an error, but the test checks to see if the file type is
IMAGINARY.
{noformat}
org.apache.commons.vfs2.FileSystemException: Could not determine the type of
file hdfs://localhost:8020/test-data/read-tests/unknown-child.
at
org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:1755)
at
org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:566)
at
org.apache.commons.vfs2.test.ProviderReadTests.testType(ProviderReadTests.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.commons.vfs2.test.AbstractProviderTestCase.runTest(AbstractProviderTestCase.java:218)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:23)
at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.extensions.TestSetup.run(TestSetup.java:27)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.io.FileNotFoundException: File does not exist:
/test-data/read-tests/unknown-child
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:528)
at
org.apache.commons.vfs2.provider.hdfs.HdfsFileObject.doAttach(HdfsFileObject.java:80)
at
org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:1738)
... 25 more
{noformat}
HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
Labels: accumulo, hdfs
Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch,
VFS-442-4.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539763#comment-13539763 ] Gary Gregory commented on VFS-442: -- You might want to compare with other providers and their implementations. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502492#comment-13502492 ] Dave Marion commented on VFS-442: - I attached patch 4. It includes changes that we made in ACCUMULO-708. These changes include getting the umask and setting the MiniDFSCluster to use the target directory instead of the build directory. I also added an hdfs profile in the core/pom.xml file that is supposed to exclude the hdfs test on Windows. Maybe someone building on Windows can verify that this works. Remaining TO-DOs: 1. It is still important to provide a test that fits in this framework in addition to the current test. The current test may then become obsolete. 2. Some Javadoc comments are missing for public members. 3. Only make public what must be public. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502538#comment-13502538 ] Gary Gregory commented on VFS-442: -- Ok, this is getting better :) Your 1,2,3s are what should be done next in order of importance. (1) will help you flush out pieces that may be missing for a full VFS implementation. Steps (2) and (3) can come once we see what happens with a test class that fits in the VFS framework. Thank you and keep up the good work! Gary HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502558#comment-13502558 ] Dave Marion commented on VFS-442: - Gary, On a side note, is there a way to change the directory that is used for the cache? Some of the Accumulo folks were asking, we see it using /tmp/vfs_cache by default. Thanks, Dave HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502559#comment-13502559 ] Dave Marion commented on VFS-442: - I think I found it, DefaultFileReplicator(String dir). Thanks anyway. Dave HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch, VFS-442-4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492485#comment-13492485 ] Gary Gregory commented on VFS-442: -- The patch works for me on Linux. I'll have to disable these tests on Windows and review the patch again. Thank you. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch, VFS-442-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492536#comment-13492536 ] Gary Gregory commented on VFS-442: -- Hello Dave, I've attached version 3 of the patch that: - IMPORTANT: adds HDFS as a provider in providers.xml. - adds missing @Overrides annotations, - makes some ivars final, - makes some vars private, - Edited the Javadoc on the test to say Linux is required. As my previous comment shows, Cygwin on Windows does not work. I cannot run most of the VFS tests in Eclipse due to the way the VFS test framework was coded a long time ago. That's unfortunate, but that's what we have today. I often resort to debugging VFS from Eclipse by connecting to the JVM started by a local Maven run. TO-DOs: # It is still important to provide a test that fits in this framework in addition to the current test. The current test may then become obsolete. # The test needs to be disabled on Windows. It might be best to do that from the POM, which I don't know how to do off the top of my head. JUnit's Assume may be the solution of last resort. # The Maven build creates a build directory in the root of the project, that's not good because someone may end up committing it by mistake or including it in a patch (like I almost did). Is this an HDFS store? If so, put it in a folder where the system propoerty java.io.tmpdir points to and remove it at the end of the test. # Some Javadoc comments are missing for public members. # Only make public what must be public. Thank you for your time and contribution! Gary HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492865#comment-13492865 ] Dave Marion commented on VFS-442: - Gary, When I get a few minutes I'll check out the patch. Regarding the build directory, its an HDFS store that is created by the embedded HDFS test cluster. I'd have to look to see if that directory location is configurable. Can we just add this directory to the svn-ignore list? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492870#comment-13492870 ] Gary Gregory commented on VFS-442: -- Yes, we could/should add build to the SVN ignore list. But the test should still clean up after itself. It would also help if the directory name were more obvious like test-hdfs but like you mentioned, you'll see if the location can be configured. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: vfs-422-3.diff, VFS-442-1.patch, VFS-442-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492044#comment-13492044
]
Dave Marion commented on VFS-442:
-
bq. 1. Get coverage as close to 100% as possible.
I'm having an issue running code coverage in Eclipse. However, I think its
close based on what I did in ACCUMULO-708
bq. 2. No need to have read-only in the class names. That's just the
capabilities of the provider today. If someone makes it read-write then there
is no need for a class rename or a new class name.
Done.
bq. 3. Always use blocks, even for one-liners: if (expr) { stmnt; }
Fixed most checkstyle issues that made sense.
bq. 4. Only make public what will be set in stone. You can make something
public tomorrow, but you cannot make it have less visibility and keep binary
compatibility.
bq. 5. Javadoc everything public.
Fixed most issues.
bq. 6. Use Java 5 enhanced for loops where possible.
bq. 7. Don't initialize ivars to their default values, for example do this:
private FileSystem fs;, not this: private FileSystem fs = null;
bq. 8. Make ivars final where possible
Fixed most issues.
bq. 9. Don't make the code platform dependent, for example, on Windows 7
Added comments in test that it needs to run on Hadoop compatible
environments, need Cygwin for Windows.
Upgraded to Hadoop dependency to version 1.1.0. I will upload a new patch in
a few moments to take the place of the first patch.
HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
Labels: accumulo, hdfs
Attachments: VFS-442-1.patch, VFS-442-2.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492086#comment-13492086
]
Gary Gregory commented on VFS-442:
--
Sigh, Cygwin is no panacea here:
{noformat}
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed:
1600 sec ERROR!
java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:425)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:283)
at
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.clinit(HdfsFileProviderTest.java:95)
... 22 more
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time elapsed:
1600 sec ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
What is your set up? What does mvn -version tell you?
HDFS FileSystem Provider
Key: VFS-442
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487159#comment-13487159 ] Dave Marion commented on VFS-442: - Before I start fixing the issues that you identified above, adding Hadoop does make this somewhat non-portable. Do you think this is still a good candidate for including in VFS? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487167#comment-13487167
]
Dave Marion commented on VFS-442:
-
bq. Is there a reason the test does not live in the VFS test hierarchy?
Not sure I follow, the test class is in the correct package:
{noformat}
package org.apache.commons.vfs2.provider.hdfs.test;
import java.io.IOException;
...
public class ReadOnlyHdfsFileProviderTest {
{noformat}
I checked the patch that I submitted, and it also appears in the correct
location:
{noformat}
Index:
core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java
===
---
core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java
(revision 0)
+++
core/src/test/java/org/apache/commons/vfs2/provider/hdfs/test/ReadOnlyHdfsFileProviderTest.java
(revision 0)
@@ -0,0 +1,344 @@
{noformat}
What am I missing?
HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
Labels: accumulo, hdfs
Attachments: VFS-442-1.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487177#comment-13487177
]
Gary Gregory commented on VFS-442:
--
Hello Dave,
Yes, a HDFS makes sense, if there is NO way to get Hadoop running on Windows
(is there? Cygwin?), then the test MUST be coded such that it is a no-op on
Windows. The test Javadoc MUST make a note of this and suggest that Windows
users build it under VirtualBox/Linux/Ubuntu (if that works).
Did you look at the other provider test cases? Most extend
{{AbstractProviderTestConfig}} and/or implement {{ProviderTestConfig}}.
HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
Labels: accumulo, hdfs
Attachments: VFS-442-1.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487178#comment-13487178 ] Gary Gregory commented on VFS-442: -- Also, is it possible to talk to Hadoop remotely? Over HTTP, sockets or SSH? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487182#comment-13487182 ] Dave Marion commented on VFS-442: - Hey Gary, I believe that you can run Hadoop on Windows with Cygwin. Programmatic access to the file system is typically done using Hadoop client software. It does all of the communication with the NameNode and DataNodes to access the blocks on the nodes that make up the file you are interacting with. I did look at the other test cases, but I didn't see a reason to extend AbstractProviderTestConfig. If it is required, then I will do it. Dave Marion HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487205#comment-13487205 ] Gary Gregory commented on VFS-442: -- If you subclass AbstractProviderTestConfig, your provider will run against some predefined tests. It's all twisty code (JUnit 3, not 4) but that how providers are tested, which avoids a lot of code duplication. If you are getting 100% code coverage with your test, then good. But you might not find out what methods or behaviors your provider is supposed to implement unless you fit in the test framework. I would not dump your existing test though. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487204#comment-13487204 ] Gary Gregory commented on VFS-442: -- If you subclass AbstractProviderTestConfig, your provider will run against some predefined tests. It's all twisty code (JUnit 3, not 4) but that how providers are tested, which avoids a lot of code duplication. If you are getting 100% code coverage with your test, then good. But you might not find out what methods or behaviors your provider is supposed to implement unless you fit in the test framework. I would not dump your existing test though. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486227#comment-13486227
]
Gary Gregory commented on VFS-442:
--
The first thing to check is that you can build with Maven, I use Maven 3.0.4.
For example: mvn clean site.
I get an error:
{noformat}
[INFO] --- maven-bundle-plugin:2.3.7:manifest (bundle-manifest) @ commons-vfs2
---
[ERROR] Manifest org.apache.commons:commons-vfs2:jar:2.1-SNAPSHOT : The default
package '.' is not permitted by the Import-Package syntax.
This can be caused by compile errors in Eclipse because Eclipse creates
valid class files regardless of compile errors.
The following package(s) import from the default package
[org.apache.commons.vfs2.provider.hdfs]
[ERROR] Error(s) found in manifest configuration
{noformat}
HDFS FileSystem Provider
Key: VFS-442
URL: https://issues.apache.org/jira/browse/VFS-442
Project: Commons VFS
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Dave Marion
Priority: Minor
Labels: accumulo, hdfs
Attachments: VFS-442-1.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486450#comment-13486450 ] Dave Marion commented on VFS-442: - mvn clean site completed successfully. I looked around the site and noticed that I have a bunch of issues with checkstyle. Is there a code formatter template somewhere? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[
https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486562#comment-13486562
]
Gary Gregory commented on VFS-442:
--
I do not know of code format templates for different IDEs. You might want to
google around, someone might have created one for the version of your IDE.
Otherwise, look at the existing code and adjust your IDE formatter to match.
Even though it does not build for me with Maven. I can give you some pointers:
# Get coverage as close to 100% as possible.
# No need to have read-only in the class names. That's just the capabilities of
the provider today. If someone makes it read-write then there is no need for a
class rename or a new class name.
# Always use blocks, even for one-liners: if (expr) { stmnt; }
# Only make public what will be set in stone. You can make something public
tomorrow, but you cannot make it have less visibility and keep binary
compatibility.
# Javadoc everything public.
# Use Java 5 enhanced for loops where possible.
# Don't initialize ivars to their default values, for example do this: private
FileSystem fs;, not this: private FileSystem fs = null;
# Make ivars final where possible
# Don't make the code platform dependent, for example, on Windows 7, I get:
{noformat}
java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.RuntimeException: Error while running command to get file
permissions : java.io.IOException: Cannot run program ls: CreateProcess
error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:418)
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:393)
at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1537)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1484)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:280)
at
org.apache.commons.vfs2.provider.hdfs.test.ReadOnlyHdfsFileProviderTest.clinit(ReadOnlyHdfsFileProviderTest.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486571#comment-13486571 ] Gary Gregory commented on VFS-442: -- We can put my Maven build issue aside now. It seems to have resolved itself. Maven can be black magic. I still suffer from what seems to be Hadoop's snobbish attitude towards Windows: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program ls: CreateProcess error=2, The system cannot find the file specified Unless that is configurable? HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-442) HDFS FileSystem Provider
[ https://issues.apache.org/jira/browse/VFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486574#comment-13486574 ] Gary Gregory commented on VFS-442: -- Also: # Fix Checkstyle issues as appropriate. # Fix PMD issues as appropriate. The PMD plugin is broken for VFS until version 2.8 comes out shortly I hope. Until then you should run PMD from your IDE, there are plugins for many IDEs. HDFS FileSystem Provider Key: VFS-442 URL: https://issues.apache.org/jira/browse/VFS-442 Project: Commons VFS Issue Type: New Feature Affects Versions: 2.0 Reporter: Dave Marion Priority: Minor Labels: accumulo, hdfs Attachments: VFS-442-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
