Hello Chris, thanks for answering. I am not happy about the fact, that it might be not possible to have generic fallbacks in all cases.
For the actual method in question I think File#isWriteable is a perfectly fine fallback especially as it is used in Posix systems anyway (and is never really reliable). I think the method is also used without a real need in this place. I hade the hope that other places look similiar. Of course I can understand that you dont want to allow generic fallbacks in production environments. Would you suggest to have a system property? I have seen you comments on the VFS bugs. I am not so sure about the binaries policy for testing in apache commons, I will need to check this. However I would simply disable the tests on Windows again. There are some comments talking about the need for fallbacks in the source. If this is not the actual desire it should be removed. Gruss Bernd Am Fri, 20 Jun 2014 10:02:49 -0700 schrieb Chris Nauroth <cnaur...@hortonworks.com>: > Hello Bernd, > > This is my first time looking at Commons VFS. It looks like a very > useful project, and I'm sorry to hear that we've caused some grief > for your tests. > > I can provide some background on these changes. Traditionally, 1.2 > and earlier versions didn't officially support Windows beyond a best > effort to keep some approximation of the functionality running on > developer desktops. As we worked on enabling real production > deployments on Windows, we discovered multiple bugs caused by those > approximations. > > In some cases, the Cygwin and GnuWin32 tools had incomplete > implementations or bugs. Examples of the buggy areas include finding > the groups for a user and process management. Basically, anything in > org.apache.hadoop.util.Shell was problematic. For this category of > bugs, the solution was winutils.exe, which delegates straight to the > necessary Windows API calls. > > Other problems arose from bugs or missing functionality in the > Windows JDK implementation. For example, the canRead/canWrite > methods in java.io.File don't work correctly on Windows. (This is > the code path shown in your stack trace.) For this category of bugs, > the solution was hadoop.dll, which is a JNI layer that delegates to > the Windows API. > > Without these workarounds, production deployment on Windows wouldn't > have been feasible. There would have been too many dangerous bugs. > > Unfortunately, we have a weakness right now in that the Windows native > build artifacts aren't bundled into the Apache releases. We also > don't have a fallback mode that works without the native code. (Even > if there was such a fallback mode, it would be very dangerous to run > in this mode in production, so I'd want to treat it as a test-only > setting.) > > I expect you'll run into a lot of challenges while fully implementing > a Java-only fallback. I'd like to help by checking out the Commons > VFS code and experimenting a bit. I'll try to put some time into > this next week and then comment on the VFS jira issues with my > findings. Does this sound good? > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Tue, Jun 17, 2014 at 7:46 PM, Bernd Eckenfels > <e...@zusammenkunft.net> wrote: > > > Hello, > > > > in the Commons VFS project we have a Provider for HDFS. This > > Provider is unit-tested with the help of the excelent MiniDFSCluster > > component. > > > > A while back I enabled automated testing on Windows (cause it worked > > for me), but I did not realize it only works with cygwin (namely > > ls.exe) in the Path. This was some 1.2 version and it used the ls > > command to read the owner. > > > > We currently switch the dependency to 2.4.0 and had the hope that it > > better (i.e. > > without binary dependencies) works in this scenario - as we want to > > run the maven tests without pre-installing a tool chain on Windows. > > > > However it got actually worse, it warns about a missing winutils.exe > > and it requires the hadoop.dll. The first location where it fails is > > testing writeability for the name node. And unlike all the comments > > in FileUtil/NativeIO it does not cleanly fall back to java portable > > methods. > > > > I think the access/access0 method can easily be fixed, but before I > > work on this patch, does it make sense to go in this direction or > > will I have to fix some other, more complicated platform > > dependencies. My goal would be to start MiniDFSCluster inside > > surefire from maven without the need for complicated external > > toolchain setup. > > > > Is actually anybody caring about making this possible? > > > > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest > > Time elapsed: 2.291 sec <<< ERROR! > > java.lang.UnsatisfiedLinkError: > > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z > > at > > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native > > Method) at > > org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570) > > at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996) > > at > > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484) > > at > > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292) > > at > > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202) > > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879) > > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304) > > at > > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975) > > at > > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856) > > at > > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702) > > at > > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640) > > at > > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95) > > > > Greetings > > Bernd > > > > PS - in case you want to follow the VFS progress: > > * https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4 > > migration > > * https://issues.apache.org/jira/browse/VFS-529 - the broken > > windows tests > > * https://issues.apache.org/jira/browse/VFS-526 - failed enable of > > win tests > > > > >
