Hello Bernd, This is my first time looking at Commons VFS. It looks like a very useful project, and I'm sorry to hear that we've caused some grief for your tests.
I can provide some background on these changes. Traditionally, 1.2 and earlier versions didn't officially support Windows beyond a best effort to keep some approximation of the functionality running on developer desktops. As we worked on enabling real production deployments on Windows, we discovered multiple bugs caused by those approximations. In some cases, the Cygwin and GnuWin32 tools had incomplete implementations or bugs. Examples of the buggy areas include finding the groups for a user and process management. Basically, anything in org.apache.hadoop.util.Shell was problematic. For this category of bugs, the solution was winutils.exe, which delegates straight to the necessary Windows API calls. Other problems arose from bugs or missing functionality in the Windows JDK implementation. For example, the canRead/canWrite methods in java.io.File don't work correctly on Windows. (This is the code path shown in your stack trace.) For this category of bugs, the solution was hadoop.dll, which is a JNI layer that delegates to the Windows API. Without these workarounds, production deployment on Windows wouldn't have been feasible. There would have been too many dangerous bugs. Unfortunately, we have a weakness right now in that the Windows native build artifacts aren't bundled into the Apache releases. We also don't have a fallback mode that works without the native code. (Even if there was such a fallback mode, it would be very dangerous to run in this mode in production, so I'd want to treat it as a test-only setting.) I expect you'll run into a lot of challenges while fully implementing a Java-only fallback. I'd like to help by checking out the Commons VFS code and experimenting a bit. I'll try to put some time into this next week and then comment on the VFS jira issues with my findings. Does this sound good? Chris Nauroth Hortonworks http://hortonworks.com/ On Tue, Jun 17, 2014 at 7:46 PM, Bernd Eckenfels <e...@zusammenkunft.net> wrote: > Hello, > > in the Commons VFS project we have a Provider for HDFS. This > Provider is unit-tested with the help of the excelent MiniDFSCluster > component. > > A while back I enabled automated testing on Windows (cause it worked > for me), but I did not realize it only works with cygwin (namely > ls.exe) in the Path. This was some 1.2 version and it used the ls > command to read the owner. > > We currently switch the dependency to 2.4.0 and had the hope that it > better (i.e. > without binary dependencies) works in this scenario - as we want to run > the maven tests without pre-installing a tool chain on Windows. > > However it got actually worse, it warns about a missing winutils.exe > and it requires the hadoop.dll. The first location where it fails is > testing writeability for the name node. And unlike all the comments in > FileUtil/NativeIO it does not cleanly fall back to java portable > methods. > > I think the access/access0 method can easily be fixed, but before I work > on this patch, does it make sense to go in this direction or will I have to > fix some other, more complicated platform dependencies. My goal would be to > start MiniDFSCluster inside surefire from maven without the need for > complicated external toolchain setup. > > Is actually anybody caring about making this possible? > > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Time > elapsed: 2.291 sec <<< ERROR! > java.lang.UnsatisfiedLinkError: > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native > Method) > at > org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570) > at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996) > at > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702) > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640) > at > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95) > > Greetings > Bernd > > PS - in case you want to follow the VFS progress: > * https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4 migration > * https://issues.apache.org/jira/browse/VFS-529 - the broken windows tests > * https://issues.apache.org/jira/browse/VFS-526 - failed enable of win > tests > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
