[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795321#comment-13795321
]
Jason Lowe commented on HADOOP-9652:
This change can cause ShuffleHandlers to become unresponsive when the cluster
is under load because {{exists}} is now a very expensive call in
RawLocalFileSystem. See MAPREDUCE-5584.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798818#comment-13798818
]
Arun C Murthy commented on HADOOP-9652:
---
Should we revert this? It's caused a major regression.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798828#comment-13798828
]
Arun C Murthy commented on HADOOP-9652:
---
[~andrew.wang]? [~cmccabe]?
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799167#comment-13799167
]
Suresh Srinivas commented on HADOOP-9652:
-
+1 for reverting this. Lets revisit this later with other symlink issues.
[~sanjay.radia], please consider this in symlink discussions.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799361#comment-13799361
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I think the easiest way to handle this is to enable the useLegacy thing. We
don't need a full revert. Will post a patch shortly.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799407#comment-13799407
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609182/0001-temporarily-disable-HADOOP-9652.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3235//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799408#comment-13799408
]
Andrew Wang commented on HADOOP-9652:
-
Fixup patch LGTM, +1.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799409#comment-13799409
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I'm going to update it to keep using the new {{getFileLinkStatus}} in
{{TestSymlinkLocalFSFileSystem}}. We don't want those tests to bitrot.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799537#comment-13799537
]
Andrew Wang commented on HADOOP-9652:
-
I figured we were applying this fix only to branch-2 as a stopgap for a proper
fix using JNI later. Regardless, fixing the unit tests sounds like a good idea.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799708#comment-13799708
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I'm having some problems running the unit tests on trunk as well. Have the
symlink unit tests bitrotted already? Will take a look later.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880138#comment-13880138
]
Jason Lowe commented on HADOOP-9652:
Any update on this? I see branch-2 is still doing a ton of fork-n-exec
overhead for fs.exists() on local files.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.4.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880283#comment-13880283
]
Colin Patrick McCabe commented on HADOOP-9652:
--
+1 for the workaround. thanks, Andrew.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.4.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch,
> hadoop-9652-workaround.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880487#comment-13880487
]
Jason Lowe commented on HADOOP-9652:
+1, thanks Andrew! I plan on committing this but need to clear up something
about the state of this JIRA. It looks like the reported problem will still
exist, since getFileLinkStatus will not fill in the link owner and mode by
default after the workaround patch. In that sense the cleanest thing would be
to simply revert and commit a real fix later, but it looks like the last
committed patch did more than just fix the described problem (e.g.: also fixing
HADOOP-9693). So I'm thinking we need to rename this JIRA to reflect it's
laying the groundwork for the eventual fix and file a followup JIRA to track
completing the fix. Does that make sense?
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.4.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch,
> hadoop-9652-workaround.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880505#comment-13880505
]
Andrew Wang commented on HADOOP-9652:
-
Sounds good to me, thanks Jason. I'm honestly okay with reverting symlink
support for local filesystem entirely since they seem to be more trouble
than they're worth, but that's a discussion we can have at a later time.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.4.0
>
> Attachments: 0001-temporarily-disable-HADOOP-9652.patch,
> hadoop-9452-1.patch, hadoop-9652-2.patch, hadoop-9652-3.patch,
> hadoop-9652-4.patch, hadoop-9652-5.patch, hadoop-9652-6.patch,
> hadoop-9652-workaround.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706652#comment-13706652
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591965/hadoop-9452-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2774//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707262#comment-13707262
]
Colin Patrick McCabe commented on HADOOP-9652:
--
Looks good. I am glad to see that {{DelegateToFileSystem}} is now delegating
the symlinks methods-- good idea.
{code}
-assertEquals(FsPermission.getDefault(), fsd.getPermission());
{code}
This test case should be still applicable, right? Symlinks created by the test
user should be owned by him.
{code}
+ public static final OSType osType;
+
+ static {
+osType = getOSType();
+ }
{code}
This can al be combined into one line.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707472#comment-13707472
]
Andrew Wang commented on HADOOP-9652:
-
Thanks Colin.
bq. Symlinks created by the test user should be owned by him.
True, I added the username check back in. The other tests are unfortunately
harder; for instance, symlink permissions are inconsistent across operating
systems, and it's unclear to me which of the user's groups is set.
Fixed your other comment too.
I also realized the earlier patch removed Windows support, so I brought the old
functions back as a fallback. I think it'd be better to eventually add Windows
support to the Stat class so we can remove the old code. Though it should be
fine, I haven't tested this on windows either, so maybe [~cnauroth] or
[~ivanmi] could verify since we'd like to get this into 2.1.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707506#comment-13707506
]
Hadoop QA commented on HADOOP-9652:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12592092/hadoop-9652-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2778//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2778//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708054#comment-13708054
]
Chris Nauroth commented on HADOOP-9652:
---
These tests have not been working correctly on a Windows local file system,
even before this patch. [~arpitagarwal] has been doing most of the coding to
get them working in the context of HADOOP-9527. The HADOOP-9527 patch needs a
rebase though to reconcile it against recent refactoring work.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13715668#comment-13715668
]
Andrew Wang commented on HADOOP-9652:
-
Cool, thanks for taking a peek Chris. Just to further clarify, the v1 patch
would have broken getFileStatus and getFileLinkStatus for local Windows
filesystems, so I figure the v2 still makes sense even if the local Windows
symlink tests are currently broken.
I think the v2 patch is thus ready for further review or commit (still applies
to trunk for me). Already has a +1 from Jenkins above.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718943#comment-13718943
]
Colin Patrick McCabe commented on HADOOP-9652:
--
bq. True, I added the username check back in. The other tests are unfortunately
harder; for instance, symlink permissions are inconsistent across operating
systems, and it's unclear to me which of the user's groups is set.
Newly created files get set to the user's primary group on UNIX.
{code}
+FileStatus status = fsImpl.getFileLinkStatus(f);
+// FileSystem#getFileLinkStatus qualifies the link target
+// AbstractFileSystem needs to return it plain since it's qualified
+// in FileContext, so re-get and set the plain target
+if (status.isSymlink()) {
+ status.setSymlink(fsImpl.getLinkTarget(f));
+}
+return status;
{code}
This actually highlight something that I think we might need to fix in the
FileSystem symlink support-- the fact that it's impossible to view relative
symlink paths once they are created.
Let's say I create a symlink from a -> ./b in my local HDFS.
With the new FSShell support in HDFS-4019, Hadoop displays this as:
{code}
lrwxrwxrwx - cmccabe supergroup 0 2013-07-23 15:59 /a ->
hdfs://localhost:6000/b
{code}
It seems to me that we should display relative symlinks *as* relative, rather
than absolute-izing them and adding a URI. Otherwise, if we use DistCp to copy
a cluster's data to another cluster, all the symlink targets will be wrong.
This is probably out of scope for this particular JIRA, but should we file a
follow-up?
{code}
+if (osType == OSType.OS_TYPE_UNIX) {
+ return new String[] {
+ "bash", "-c",
+ "exec 'stat' '" + derefFlag + "c' '%s,%F,%Y,%X,%a,%U,%G,%N' '"
+ + path + "' 2>&1" };
{code}
Shouldn't this be OS_TYPE_LINUX? I don't think, for example, OpenBSD comes
with the GNU stat program. We should also really also file a follow-up bug to
remove this and move to a more portable solution when we get the green light to
drop JDK6 support.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719033#comment-13719033
]
Andrew Wang commented on HADOOP-9652:
-
Thanks for the review Colin, new patch attached.
bq. Newly created files get set to the user's primary group on UNIX.
TY, added that check back in.
bq. impossible to view relative symlink paths once they are created
I think FileSystem#getLinkTarget serves this purpose; targets are returned as
set.
bq. Shouldn't this be OS_TYPE_LINUX?
Truth, changed it to LINUX. It's not actually a positive Linux detection
though, since it's the default OSType.
bq. We should also really also file a follow-up bug to remove this and move to
a more portable solution when we get the green light to drop JDK6 support.
Definitely agree on this, there are lots of places we shell out to workaround
Java 6. I'll file a JIRA.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719097#comment-13719097
]
Hadoop QA commented on HADOOP-9652:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594059/hadoop-9652-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719822#comment-13719822
]
Colin Patrick McCabe commented on HADOOP-9652:
--
bq. I think FileSystem#getLinkTarget serves the purpose... [of returning
targets as set].
Hmm. Not completely convinced, but that also isn't really relevant to this
JIRA.
Thanks for creating HADOOP-9769 as well.
+1. thanks, Andrew.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721360#comment-13721360
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I filed https://issues.apache.org/jira/browse/HADOOP-9780 to discuss the
{{FileSystem#getLinkTarget}} issue, which is unrelated to this stuff
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721813#comment-13721813
]
Vinod Kumar Vavilapalli commented on HADOOP-9652:
-
Tests also broke also on MacOSX because of this. Hoping HDFS-5039/HADOOP-9782
can be addressed quickly. Tx.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722038#comment-13722038
]
Andrew Wang commented on HADOOP-9652:
-
Hey folks, sorry about the breakage. I filed a fixup patch at HADOOP-9783,
would appreciate if someone could test on their Mac environment.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722870#comment-13722870
]
Jason Lowe commented on HADOOP-9652:
It looks like this may have also broken distributed cache archive processing,
as TestMRApps#testSetClasspathWithArchives is failing after this change. See
MAPREDUCE-5430.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723041#comment-13723041
]
Robert Parker commented on HADOOP-9652:
---
It appears to have also broken TestDFSShell#testCount, TestDFSShell#testText,
and TestDFSShell#testFilePermissions. Checked out the checkin just prior
(git:700817) and these test passed.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723066#comment-13723066
]
Daryn Sharp commented on HADOOP-9652:
-
Whoa. Anytime you think you need to call "bash -c" you need rethink the
change. User input should only be passed a direct arguments to execve. As
this patch stands, any shell metachars in a path is going to produce unexpected
and possible security issues.
I was originally led to this jira because my dev env broke and we've got people
spinning cycles trying to debug new test failures.
Given all issues, I think we should consider reverting the change.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723070#comment-13723070
]
Robert Parker commented on HADOOP-9652:
---
Also breaks TestHttpFSFileSystemLocalFileSystem#testOperationDoAs and
TestHttpFSFileSystemLocalFileSystem#testOperation. Concur that we should
consider reverting.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723094#comment-13723094
]
Andrew Wang commented on HADOOP-9652:
-
Let's just revert, this has caused way more fallout than I thought. I've
tracked down most of the issues so far, but it seems cleaner to just do a
single fixed patch than all these little JIRAs.
I'll also kick off full-project test builds on my own for future patches to
this JIRA, because apparently the common tests are lacking here.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723149#comment-13723149
]
Colin Patrick McCabe commented on HADOOP-9652:
--
Daryn, that is a good catch. We definitely *don't* need to call {{bash -c}}
here. It adds no value, and certainly could be a security vulnerability.
bq. Let's just revert, this has caused way more fallout than I thought. I've
tracked down most of the issues so far, but it seems cleaner to just do a
single fixed patch than all these little JIRAs.
OK.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723161#comment-13723161
]
Colin Patrick McCabe commented on HADOOP-9652:
--
OK. I have reverted the patch.
Here is a summary of the issues we know of with the original patch:
* Fallback path should handle MacOS (HADOOP-9783)
* Issue with URI fragment handling and DistCp (MAPREDUCE-5430)
* Stray {{System.println}} caused some TestDFSShell failures
* must not use "{{bash -c}}"
So we need a new version of the patch which addresses all of these.
Additionally, can you touch a file in each project (i.e., hadoop-hdfs-project,
hadoop-mapreduce-project, hadoop-yarn-project), so that we get the full Jenkins
run, rather than just a Jenkins run on common?
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723225#comment-13723225
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I have managed to isolate the problem we had with {{TestMRApps}}. The old code
stripped the fragment in {{FileSystem#resolvePath}}; the new code does not.
Here's a unit test which only passes on the previous code:
{code}
@Test (timeout = 12)
public void testResolvePath() throws Exception {
FileSystem fs = FileSystem.get(URI.create("file:///tmp/"), new
Configuration());
URI preUri = new URI("file:///tmp#fragment");
Path pre = new Path(preUri);
Path post = fs.resolvePath(pre);
Assert.assertEquals("resolvePath did not strip the fragment",
"file:/tmp", post.toString());
}
{code}
Ultimately this comes down to {{FileSystem#getFileStatus}}, since that's what
{{resolvePath}} calls. Andrew, in your new {{getFileLinkStatusInternal}}, you
currently just put the path passed by the user directly into the {{FileStatus}}
object.
In contrast, the old code returned {{new Path(new
File(originalPath.toUri().getPath())).getPath()).makeQualified(fs.getUri(),
fs.getWorkingDirectory())}}. After all those conversions back and forth, I
think:
* everything is lost except the "path string" -- i.e. "{{/tmp}}"
* we fully qualify it
For compatibility reasons, I think we probably need to do the same-- strip off
everything but the "path string" and fully qualify. I took a quick look at the
HDFS (as opposed to LocalFileSystem) code, and it goes through a very different
path. As far as I can see, however, {{DistributedFileSystem#getFileInfo}} will
also strip the "fragment." So at least we are consistent.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723430#comment-13723430
]
Chris Nauroth commented on HADOOP-9652:
---
trunk is reverted, but does this still need to be reverted from branch-2?
Thanks!
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723780#comment-13723780
]
Robert Parker commented on HADOOP-9652:
---
[~cnauroth] Please revert branch-2 also. All the test that I reported were
failing on branch-2.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726895#comment-13726895
]
Andrew Wang commented on HADOOP-9652:
-
Hi everyone,
Here's a new patch that rolls up the previous issues. This includes:
- Stripping out URI fragments from Paths, with a new test for this behavior.
This was causing the MR failure.
- Add sticky bit support and remove stray prints. This was breaking DFSShell
tests.
- Clean up OS detection to fix Mac OSX test runs
- No more "bash -c". DF still does this, so I'll file another JIRA to improve
this.
I've run the failed tests on my end, but would of course appreciate any further
testing.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726931#comment-13726931
]
Hadoop QA commented on HADOOP-9652:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12595479/hadoop-9652-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2905//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2905//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728130#comment-13728130
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I tested this on a Solaris derivative (OpenIndiana) and verified that the
fallback path for LocalFileSystem worked. It works!
{code}
public static String[] getCheckProcessIsAliveCommand(String pid) {
-return Shell.WINDOWS ?
+return WINDOWS ?
new String[] { Shell.WINUTILS, "task", "isAlive", pid } :
{code}
Let's make these cleanup changes in a separate JIRA to keep the diff down.
In {{RawLocalFileSystem}}, I think we should have a static final boolean that
caches the information about whether we are using the fallback path. That also
would make the code a little clearer-- it's otherwise not terribly clear why we
do something different for linux and freebsd.
{code}
+ @Deprecated
static class RawLocalFileStatus extends FileStatus {
/* We can add extra fields here. It breaks at least CopyFiles.FilePair().
* We recognize if the information is already loaded by check if
@@ -697,6 +706,7 @@ public void createSymlink(Path target, Path link, boolean
createParent)
* the given path does not refer to a symlink or there is an error
* accessing the symlink.
*/
+ @Deprecated
private String readLink(Path p) {
{code}
Can we rename these something that indicates that they're part of the fallback
path? I think {{RawLocalFileSystem#RawLocalFileStatus}} should be renamed to
something like {{DeprecatedLocalFileStatus}}.
In general, everything that's only part of the fallback path should be somehow
indicated.
I filed https://issues.apache.org/jira/browse/HDFS-5062 to address the existing
{{bash -c}} in {{org.apache.hadoop.fs.Df}}
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735168#comment-13735168
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12597141/hadoop-9652-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2954//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735193#comment-13735193
]
Andrew Wang commented on HADOOP-9652:
-
These are the same {{metrics2}} mbean findbugs warnings that have been cropping
up recently, I believe spurious.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735502#comment-13735502
]
Colin Patrick McCabe commented on HADOOP-9652:
--
{code}
@Override
public Path getLinkTarget(Path f) throws IOException {
FileStatus fi = getFileLinkStatusInternal(f, false);
// return an unqualified symlink target
return fi.getSymlink();
}
{code}
Isn't this going to unconditionally call the Stat() code path, even on
platforms where that is not supported? This also suggests to me that
{{getFileLinkStatusInternal}} should be named something like
{{getNativeFileLinkStatus}}, to avoid confusion.
Looks reasonable aside from that.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737254#comment-13737254
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12597537/hadoop-9652-6.patch
against trunk revision .
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2969//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739018#comment-13739018
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12597537/hadoop-9652-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2977//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739031#comment-13739031
]
Hadoop QA commented on HADOOP-9652:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12597844/hadoop-9652-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2978//console
This message is automatically generated.
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739038#comment-13739038
]
Colin Patrick McCabe commented on HADOOP-9652:
--
+1. will commit shortly
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740399#comment-13740399
]
Hudson commented on HADOOP-9652:
SUCCESS: Integrated in Hadoop-trunk-Commit #4260 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4260/])
HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and
mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740865#comment-13740865
]
Hudson commented on HADOOP-9652:
SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/302/])
HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and
mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740937#comment-13740937
]
Hudson commented on HADOOP-9652:
FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/])
HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and
mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741008#comment-13741008
]
Hudson commented on HADOOP-9652:
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/])
HADOOP-9652. RawLocalFs#getFileLinkStatus does not fill in the link owner and
mode. (Andrew Wang via Colin Patrick McCabe) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514088)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Stat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestStat.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch, hadoop-9652-4.patch, hadoop-9652-5.patch,
> hadoop-9652-6.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
