[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795321#comment-13795321 ] Jason Lowe commented on HADOOP-9652: This change can cause ShuffleHandlers to become unresponsive when the cluster is under load because {{exists}} is now a very expensive call in RawLocalFileSystem. See MAPREDUCE-5584. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798818#comment-13798818 ] Arun C Murthy commented on HADOOP-9652: --- Should we revert this? It's caused a major regression. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798828#comment-13798828 ] Arun C Murthy commented on HADOOP-9652: --- [~andrew.wang]? [~cmccabe]? > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799167#comment-13799167 ] Suresh Srinivas commented on HADOOP-9652: - +1 for reverting this. Lets revisit this later with other symlink issues. [~sanjay.radia], please consider this in symlink discussions. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799361#comment-13799361 ] Colin Patrick McCabe commented on HADOOP-9652: -- I think the easiest way to handle this is to enable the useLegacy thing. We don't need a full revert. Will post a patch shortly. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799407#comment-13799407 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12609182/0001-temporarily-disable-HADOOP-9652.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem org.apache.hadoop.fs.TestSymlinkLocalFSFileContext {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799408#comment-13799408 ] Andrew Wang commented on HADOOP-9652: - Fixup patch LGTM, +1. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799409#comment-13799409 ] Colin Patrick McCabe commented on HADOOP-9652: -- I'm going to update it to keep using the new {{getFileLinkStatus}} in {{TestSymlinkLocalFSFileSystem}}. We don't want those tests to bitrot. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799537#comment-13799537 ] Andrew Wang commented on HADOOP-9652: - I figured we were applying this fix only to branch-2 as a stopgap for a proper fix using JNI later. Regardless, fixing the unit tests sounds like a good idea. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799708#comment-13799708 ] Colin Patrick McCabe commented on HADOOP-9652: -- I'm having some problems running the unit tests on trunk as well. Have the symlink unit tests bitrotted already? Will take a look later. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880138#comment-13880138 ] Jason Lowe commented on HADOOP-9652: Any update on this? I see branch-2 is still doing a ton of fork-n-exec overhead for fs.exists() on local files. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.4.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880283#comment-13880283 ] Colin Patrick McCabe commented on HADOOP-9652: -- +1 for the workaround. thanks, Andrew. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.4.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch, > hadoop-9652-workaround.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880487#comment-13880487 ] Jason Lowe commented on HADOOP-9652: +1, thanks Andrew! I plan on committing this but need to clear up something about the state of this JIRA. It looks like the reported problem will still exist, since getFileLinkStatus will not fill in the link owner and mode by default after the workaround patch. In that sense the cleanest thing would be to simply revert and commit a real fix later, but it looks like the last committed patch did more than just fix the described problem (e.g.: also fixing HADOOP-9693). So I'm thinking we need to rename this JIRA to reflect it's laying the groundwork for the eventual fix and file a followup JIRA to track completing the fix. Does that make sense? > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.4.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch, > hadoop-9652-workaround.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880505#comment-13880505 ] Andrew Wang commented on HADOOP-9652: - Sounds good to me, thanks Jason. I'm honestly okay with reverting symlink support for local filesystem entirely since they seem to be more trouble than they're worth, but that's a discussion we can have at a later time. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.4.0 > > Attachments: 0001-temporarily-disable-HADOOP-9652.patch, > hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch, > hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch, > hadoop-9652-workaround.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706652#comment-13706652 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12591965/hadoop-9452-1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707262#comment-13707262 ] Colin Patrick McCabe commented on HADOOP-9652: -- Looks good. I am glad to see that {{DelegateToFileSystem}} is now delegating the symlinks methods-- good idea. {code} -assertEquals(FsPermission.getDefault(), fsd.getPermission()); {code} This test case should be still applicable, right? Symlinks created by the test user should be owned by him. {code} + public static final OSType osType; + + static { +osType = getOSType(); + } {code} This can al be combined into one line. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707472#comment-13707472 ] Andrew Wang commented on HADOOP-9652: - Thanks Colin. bq. Symlinks created by the test user should be owned by him. True, I added the username check back in. The other tests are unfortunately harder; for instance, symlink permissions are inconsistent across operating systems, and it's unclear to me which of the user's groups is set. Fixed your other comment too. I also realized the earlier patch removed Windows support, so I brought the old functions back as a fallback. I think it'd be better to eventually add Windows support to the Stat class so we can remove the old code. Though it should be fine, I haven't tested this on windows either, so maybe [~cnauroth] or [~ivanmi] could verify since we'd like to get this into 2.1. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707506#comment-13707506 ] Hadoop QA commented on HADOOP-9652: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12592092/hadoop-9652-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2778//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2778//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708054#comment-13708054 ] Chris Nauroth commented on HADOOP-9652: --- These tests have not been working correctly on a Windows local file system, even before this patch. [~arpitagarwal] has been doing most of the coding to get them working in the context of HADOOP-9527. The HADOOP-9527 patch needs a rebase though to reconcile it against recent refactoring work. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13715668#comment-13715668 ] Andrew Wang commented on HADOOP-9652: - Cool, thanks for taking a peek Chris. Just to further clarify, the v1 patch would have broken getFileStatus and getFileLinkStatus for local Windows filesystems, so I figure the v2 still makes sense even if the local Windows symlink tests are currently broken. I think the v2 patch is thus ready for further review or commit (still applies to trunk for me). Already has a +1 from Jenkins above. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718943#comment-13718943 ] Colin Patrick McCabe commented on HADOOP-9652: -- bq. True, I added the username check back in. The other tests are unfortunately harder; for instance, symlink permissions are inconsistent across operating systems, and it's unclear to me which of the user's groups is set. Newly created files get set to the user's primary group on UNIX. {code} +FileStatus status = fsImpl.getFileLinkStatus(f); +// FileSystem#getFileLinkStatus qualifies the link target +// AbstractFileSystem needs to return it plain since it's qualified +// in FileContext, so re-get and set the plain target +if (status.isSymlink()) { + status.setSymlink(fsImpl.getLinkTarget(f)); +} +return status; {code} This actually highlight something that I think we might need to fix in the FileSystem symlink support-- the fact that it's impossible to view relative symlink paths once they are created. Let's say I create a symlink from a -> ./b in my local HDFS. With the new FSShell support in HDFS-4019, Hadoop displays this as: {code} lrwxrwxrwx - cmccabe supergroup 0 2013-07-23 15:59 /a -> hdfs://localhost:6000/b {code} It seems to me that we should display relative symlinks *as* relative, rather than absolute-izing them and adding a URI. Otherwise, if we use DistCp to copy a cluster's data to another cluster, all the symlink targets will be wrong. This is probably out of scope for this particular JIRA, but should we file a follow-up? {code} +if (osType == OSType.OS_TYPE_UNIX) { + return new String[] { + "bash", "-c", + "exec 'stat' '" + derefFlag + "c' '%s,%F,%Y,%X,%a,%U,%G,%N' '" + + path + "' 2>&1" }; {code} Shouldn't this be OS_TYPE_LINUX? I don't think, for example, OpenBSD comes with the GNU stat program. We should also really also file a follow-up bug to remove this and move to a more portable solution when we get the green light to drop JDK6 support. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719033#comment-13719033 ] Andrew Wang commented on HADOOP-9652: - Thanks for the review Colin, new patch attached. bq. Newly created files get set to the user's primary group on UNIX. TY, added that check back in. bq. impossible to view relative symlink paths once they are created I think FileSystem#getLinkTarget serves this purpose; targets are returned as set. bq. Shouldn't this be OS_TYPE_LINUX? Truth, changed it to LINUX. It's not actually a positive Linux detection though, since it's the default OSType. bq. We should also really also file a follow-up bug to remove this and move to a more portable solution when we get the green light to drop JDK6 support. Definitely agree on this, there are lots of places we shell out to workaround Java 6. I'll file a JIRA. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719097#comment-13719097 ] Hadoop QA commented on HADOOP-9652: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594059/hadoop-9652-3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719822#comment-13719822 ] Colin Patrick McCabe commented on HADOOP-9652: -- bq. I think FileSystem#getLinkTarget serves the purpose... [of returning targets as set]. Hmm. Not completely convinced, but that also isn't really relevant to this JIRA. Thanks for creating HADOOP-9769 as well. +1. thanks, Andrew. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721360#comment-13721360 ] Colin Patrick McCabe commented on HADOOP-9652: -- I filed https://issues.apache.org/jira/browse/HADOOP-9780 to discuss the {{FileSystem#getLinkTarget}} issue, which is unrelated to this stuff > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721813#comment-13721813 ] Vinod Kumar Vavilapalli commented on HADOOP-9652: - Tests also broke also on MacOSX because of this. Hoping HDFS-5039/HADOOP-9782 can be addressed quickly. Tx. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722038#comment-13722038 ] Andrew Wang commented on HADOOP-9652: - Hey folks, sorry about the breakage. I filed a fixup patch at HADOOP-9783, would appreciate if someone could test on their Mac environment. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722870#comment-13722870 ] Jason Lowe commented on HADOOP-9652: It looks like this may have also broken distributed cache archive processing, as TestMRApps#testSetClasspathWithArchives is failing after this change. See MAPREDUCE-5430. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723041#comment-13723041 ] Robert Parker commented on HADOOP-9652: --- It appears to have also broken TestDFSShell#testCount, TestDFSShell#testText, and TestDFSShell#testFilePermissions. Checked out the checkin just prior (git:700817) and these test passed. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723066#comment-13723066 ] Daryn Sharp commented on HADOOP-9652: - Whoa. Anytime you think you need to call "bash -c" you need rethink the change. User input should only be passed a direct arguments to execve. As this patch stands, any shell metachars in a path is going to produce unexpected and possible security issues. I was originally led to this jira because my dev env broke and we've got people spinning cycles trying to debug new test failures. Given all issues, I think we should consider reverting the change. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723070#comment-13723070 ] Robert Parker commented on HADOOP-9652: --- Also breaks TestHttpFSFileSystemLocalFileSystem#testOperationDoAs and TestHttpFSFileSystemLocalFileSystem#testOperation. Concur that we should consider reverting. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723094#comment-13723094 ] Andrew Wang commented on HADOOP-9652: - Let's just revert, this has caused way more fallout than I thought. I've tracked down most of the issues so far, but it seems cleaner to just do a single fixed patch than all these little JIRAs. I'll also kick off full-project test builds on my own for future patches to this JIRA, because apparently the common tests are lacking here. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723149#comment-13723149 ] Colin Patrick McCabe commented on HADOOP-9652: -- Daryn, that is a good catch. We definitely *don't* need to call {{bash -c}} here. It adds no value, and certainly could be a security vulnerability. bq. Let's just revert, this has caused way more fallout than I thought. I've tracked down most of the issues so far, but it seems cleaner to just do a single fixed patch than all these little JIRAs. OK. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723161#comment-13723161 ] Colin Patrick McCabe commented on HADOOP-9652: -- OK. I have reverted the patch. Here is a summary of the issues we know of with the original patch: * Fallback path should handle MacOS (HADOOP-9783) * Issue with URI fragment handling and DistCp (MAPREDUCE-5430) * Stray {{System.println}} caused some TestDFSShell failures * must not use "{{bash -c}}" So we need a new version of the patch which addresses all of these. Additionally, can you touch a file in each project (i.e., hadoop-hdfs-project, hadoop-mapreduce-project, hadoop-yarn-project), so that we get the full Jenkins run, rather than just a Jenkins run on common? > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723225#comment-13723225 ] Colin Patrick McCabe commented on HADOOP-9652: -- I have managed to isolate the problem we had with {{TestMRApps}}. The old code stripped the fragment in {{FileSystem#resolvePath}}; the new code does not. Here's a unit test which only passes on the previous code: {code} @Test (timeout = 12) public void testResolvePath() throws Exception { FileSystem fs = FileSystem.get(URI.create("file:///tmp/"), new Configuration()); URI preUri = new URI("file:///tmp#fragment"); Path pre = new Path(preUri); Path post = fs.resolvePath(pre); Assert.assertEquals("resolvePath did not strip the fragment", "file:/tmp", post.toString()); } {code} Ultimately this comes down to {{FileSystem#getFileStatus}}, since that's what {{resolvePath}} calls. Andrew, in your new {{getFileLinkStatusInternal}}, you currently just put the path passed by the user directly into the {{FileStatus}} object. In contrast, the old code returned {{new Path(new File(originalPath.toUri().getPath())).getPath()).makeQualified(fs.getUri(), fs.getWorkingDirectory())}}. After all those conversions back and forth, I think: * everything is lost except the "path string" -- i.e. "{{/tmp}}" * we fully qualify it For compatibility reasons, I think we probably need to do the same-- strip off everything but the "path string" and fully qualify. I took a quick look at the HDFS (as opposed to LocalFileSystem) code, and it goes through a very different path. As far as I can see, however, {{DistributedFileSystem#getFileInfo}} will also strip the "fragment." So at least we are consistent. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723430#comment-13723430 ] Chris Nauroth commented on HADOOP-9652: --- trunk is reverted, but does this still need to be reverted from branch-2? Thanks! > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723780#comment-13723780 ] Robert Parker commented on HADOOP-9652: --- [~cnauroth] Please revert branch-2 also. All the test that I reported were failing on branch-2. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726895#comment-13726895 ] Andrew Wang commented on HADOOP-9652: - Hi everyone, Here's a new patch that rolls up the previous issues. This includes: - Stripping out URI fragments from Paths, with a new test for this behavior. This was causing the MR failure. - Add sticky bit support and remove stray prints. This was breaking DFSShell tests. - Clean up OS detection to fix Mac OSX test runs - No more "bash -c". DF still does this, so I'll file another JIRA to improve this. I've run the failed tests on my end, but would of course appreciate any further testing. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726931#comment-13726931 ] Hadoop QA commented on HADOOP-9652: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595479/hadoop-9652-4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2905//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2905//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728130#comment-13728130 ] Colin Patrick McCabe commented on HADOOP-9652: -- I tested this on a Solaris derivative (OpenIndiana) and verified that the fallback path for LocalFileSystem worked. It works! {code} public static String[] getCheckProcessIsAliveCommand(String pid) { -return Shell.WINDOWS ? +return WINDOWS ? new String[] { Shell.WINUTILS, "task", "isAlive", pid } : {code} Let's make these cleanup changes in a separate JIRA to keep the diff down. In {{RawLocalFileSystem}}, I think we should have a static final boolean that caches the information about whether we are using the fallback path. That also would make the code a little clearer-- it's otherwise not terribly clear why we do something different for linux and freebsd. {code} + @Deprecated static class RawLocalFileStatus extends FileStatus { /* We can add extra fields here. It breaks at least CopyFiles.FilePair(). * We recognize if the information is already loaded by check if @@ -697,6 +706,7 @@ public void createSymlink(Path target, Path link, boolean createParent) * the given path does not refer to a symlink or there is an error * accessing the symlink. */ + @Deprecated private String readLink(Path p) { {code} Can we rename these something that indicates that they're part of the fallback path? I think {{RawLocalFileSystem#RawLocalFileStatus}} should be renamed to something like {{DeprecatedLocalFileStatus}}. In general, everything that's only part of the fallback path should be somehow indicated. I filed https://issues.apache.org/jira/browse/HDFS-5062 to address the existing {{bash -c}} in {{org.apache.hadoop.fs.Df}} > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735168#comment-13735168 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597141/hadoop-9652-5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735193#comment-13735193 ] Andrew Wang commented on HADOOP-9652: - These are the same {{metrics2}} mbean findbugs warnings that have been cropping up recently, I believe spurious. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735502#comment-13735502 ] Colin Patrick McCabe commented on HADOOP-9652: -- {code} @Override public Path getLinkTarget(Path f) throws IOException { FileStatus fi = getFileLinkStatusInternal(f, false); // return an unqualified symlink target return fi.getSymlink(); } {code} Isn't this going to unconditionally call the Stat() code path, even on platforms where that is not supported? This also suggests to me that {{getFileLinkStatusInternal}} should be named something like {{getNativeFileLinkStatus}}, to avoid confusion. Looks reasonable aside from that. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737254#comment-13737254 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597537/hadoop-9652-6.patch against trunk revision . {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2969//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739018#comment-13739018 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597537/hadoop-9652-6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739031#comment-13739031 ] Hadoop QA commented on HADOOP-9652: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597844/hadoop-9652-6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//console This message is automatically generated. > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739038#comment-13739038 ] Colin Patrick McCabe commented on HADOOP-9652: -- +1. will commit shortly > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740399#comment-13740399 ] Hudson commented on HADOOP-9652: SUCCESS: Integrated in Hadoop-trunk-Commit #4260 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4260/]) HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740865#comment-13740865 ] Hudson commented on HADOOP-9652: SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/302/]) HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740937#comment-13740937 ] Hudson commented on HADOOP-9652: FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/]) HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[ https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741008#comment-13741008 ] Hudson commented on HADOOP-9652: FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/]) HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java > RawLocalFs#getFileLinkStatus does not fill in the link owner and mode > - > > Key: HADOOP-9652 > URL: https://issues.apache.org/jira/browse/HADOOP-9652 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Fix For: 2.3.0 > > Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch, > hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch, > hadoop-9652-6.patch > > > {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of > the symlink, but instead uses the owner and mode of the symlink target. If > the target can't be found, it fills in bogus values (the empty string and > FsPermission.getDefault) for these. > Symlinks have an owner distinct from the owner of the target they point to, > and getFileLinkStatus ought to expose this. > In some operating systems, symlinks can have a permission other than 0777. > We ought to expose this in RawLocalFilesystem and other places, although we > don't necessarily have to support this behavior in HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira