[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257676#comment-13257676 ]
Dave Latham commented on HADOOP-7682: ------------------------------------- Here's my understanding of this issue from digging around a bit in case it's helpful for others. I could be mistaken on some of the details. Some time ago, RawLocalFileSystem.setPermission used to use a shell exec command to fork a process to alter permissions of a file. Somewhere along the 0.20 branch (I believe in the 0.20.security branch) it was decided that this was too slow, and instead java.io.File.set{Readable|Writable|Executable} should be used (see HADOOP-6304 for one such issue. It has a patch with the logic that wound up in FileUtil though perhaps committed from a different patch). However the java.io.File methods don't allow one to directly set all the permissions so the code first has to clear permissions, then build them up again. This resulted in two problems. First, there is a race condition when the file briefly has no permissions even for the owner (see MAPREDUCE-2238 for more detail). Second, Windows doesn't support clearing all permissions on the file (this JIRA). The first problem was worked around in HADOOP-7110 (HADOOP-7432 backported it to this branch) by using JNI native code instead. However, if the native code is not available, then it falls back to the java.io.File methods. So, the second problem still remains, that FileUtil.setPermission (and thus the RawLocalFileSystem setPermission) does not work on Windows because Windows does not have the native code implementation and also fails the java.io.File fallback. The issues FKorning ran into in a comment above appear to be wider than this particular JIRA, though I may have misunderstood what led to his shorn yak. Windows is listed as a supported platform for Hadoop, and some of our developers use Windows as a development environment, so it's important for us that hadoop at least functions on Windows, even if it's not as performant as our production clusters on Linux. I noted that this is currently the highest voted open JIRA for hadoop. In order for it to function on Windows, I added a final fallback in FileUtil.setPermission to revert to the older behavior of using a shell exec fork for setPermission when the other methods fail. Perhaps it will be helpful for others: {code} public static void setPermission(File f, FsPermission permission ) throws IOException { FsAction user = permission.getUserAction(); FsAction group = permission.getGroupAction(); FsAction other = permission.getOtherAction(); // use the native/fork if the group/other permissions are different // or if the native is available if (group != other || NativeIO.isAvailable()) { execSetPermission(f, permission); return; } try { boolean rv = true; // read perms rv = f.setReadable(group.implies(FsAction.READ), false); checkReturnValue(rv, f, permission); if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) { f.setReadable(user.implies(FsAction.READ), true); checkReturnValue(rv, f, permission); } // write perms rv = f.setWritable(group.implies(FsAction.WRITE), false); checkReturnValue(rv, f, permission); if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) { f.setWritable(user.implies(FsAction.WRITE), true); checkReturnValue(rv, f, permission); } // exec perms rv = f.setExecutable(group.implies(FsAction.EXECUTE), false); checkReturnValue(rv, f, permission); if (group.implies(FsAction.EXECUTE) != user.implies(FsAction.EXECUTE)) { f.setExecutable(user.implies(FsAction.EXECUTE), true); checkReturnValue(rv, f, permission); } } catch (IOException ioe) { LOG.warn("Java file permissions failed to set " + f + " to " + permission + " falling back to fork"); execSetPermission(f, permission); } } {code} > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -------------------------------------------------------------------------------------- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 > Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira