[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647918#comment-13647918 ] Suresh Srinivas commented on HADOOP-9043: - +1 for the patch. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.6.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.4.patch, > HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.6.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647786#comment-13647786 ] Arpit Agarwal commented on HADOOP-9043: --- Thanks for addressing that. +1 Verified tests pass on Windows and are skipped on OS X. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.6.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.4.patch, > HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.6.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646712#comment-13646712
]
Arpit Agarwal commented on HADOOP-9043:
---
{code}
+ @Test (timeout = 3)
+ public void testSymlinkRejectsForwardSlashesInTarget() throws IOException {
+if (!Shell.WINDOWS) {
+ // Not supported on non-Windows platforms
+ return;
+}
{code}
Hi Chris - would you consider replacing the if clause with
{{assumeTrue(Shell.WINDOWS)}} instead? I am fixing the remaining instances
TestWinUtils.java with HADOOP-9483. If your patch gets in first I can fix it as
I will have to rebase.
+1 otherwise.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.patch,
> HADOOP-9043.trunk.2.patch, HADOOP-9043.trunk.3.patch,
> HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642207#comment-13642207 ] Chris Nauroth commented on HADOOP-9043: --- To the committer who picks this up: would you please co-credit the patch to both me and Arpit Agarwal? We both wrote code for this patch. Thanks! > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.patch, > HADOOP-9043.trunk.2.patch, HADOOP-9043.trunk.3.patch, > HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634939#comment-13634939 ] Ivan Mitic commented on HADOOP-9043: Looks good, +1 on both trunk and branch-1-win patches. Thanks! > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.patch, > HADOOP-9043.trunk.2.patch, HADOOP-9043.trunk.3.patch, > HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634891#comment-13634891
]
Hadoop QA commented on HADOOP-9043:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579273/HADOOP-9043.trunk.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2454//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2454//console
This message is automatically generated.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.patch,
> HADOOP-9043.trunk.2.patch, HADOOP-9043.trunk.3.patch,
> HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634803#comment-13634803
]
Ivan Mitic commented on HADOOP-9043:
Thanks Chris, patch looks really good! One minor comment below, otherwise, +1
- Should we also disable
{{TestLocalFSFileContextSymlink#testCreateDanglingLink}} on Windows?
I verified that TestWinUtils passes on Windows, thanks for the new tests.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634601#comment-13634601
]
Hadoop QA commented on HADOOP-9043:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579223/HADOOP-9043.trunk.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2453//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2453//console
This message is automatically generated.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634257#comment-13634257 ] Ivan Mitic commented on HADOOP-9043: Thanks Chris for addressing the comments. bq. Could you elaborate on this? There are 2 call sites for this method, and they both need to convert the result to string, which is why I kept the method returning string. I just meant to have the method explicitly return a File instead of a String. This way you can remove the Windows specific comment in code and everything implicitly works. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634249#comment-13634249
]
Chris Nauroth commented on HADOOP-9043:
---
{quote}
Should we open a new JIRA to investigate readlink problem?
{quote}
+1 on this idea. I've filed HADOOP-9483 to address winutils readlink
separately. I've transferred the relevant design discussion and watchers to
the new ticket.
I'll work on one more version of the current HADOOP-9043 patch to address
Ivan's last round of feedback, but there was one piece of feedback that I
didn't quite follow:
{quote}
RawLocalFs: An incremental improvement would be to have
getPathWithoutSchemeAndAuthority return a File instead of a string.
{quote}
Could you elaborate on this? There are 2 call sites for this method, and they
both need to convert the result to string, which is why I kept the method
returning string.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632419#comment-13632419 ] Chuan Liu commented on HADOOP-9043: --- I am +1 on the existing patches to fix the unusable symlinks. Should we open a new JIRA to investigate readlink problem? On the readlink implementation I agree with the general approach Chris has described. Some logic is already implemented in libwinutils as Ivan mentioned. I also think GetFinalPathNameByHandle() may be easier to use than the DeviceIoControl(). > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631428#comment-13631428 ] Ivan Mitic commented on HADOOP-9043: Thanks Chris for the writeup on readLink! The approach overall looks good. I have a few comments that should help with further progress: 1. There is no need to support junction points. Let's only focus on symlinks. We've seen problems with junction points on Windows for paths longer than 126 chars (IIRC) and finally decided to stay out of supporting them. 2. In winutils we already have SymbolicLinkCheck function which checks if the given path is a symlink. See if you can reconcile this with the new code. 3. DeviceIoControl is one way to retrieve the symlink target path. GetFinalPathNameByHandle provides the similar functionality and might be easier to use. See which one suits you better. (it is fine to assume that we’re running on Vista or later Windows OS). Also a few comments on the latest patch: 1. RawLocalFs: An incremental improvement would be to have getPathWithoutSchemeAndAuthority return a File instead of a string. 2. Would you mind adding a unittest to TestWinUtils that targets the scenario where input paths have forward slashes (for both the target and the link). 3. TestLocalFSFileContextSymlink: Can you add a short comment in the first assert saying that dangling links are currently not supported on Windows for LocalFS. 4. Please also include a few winutils unittests for the new readLink API. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631413#comment-13631413 ] Ivan Mitic commented on HADOOP-9043: Thanks Chris for the patch! I am reviewing this now... > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630513#comment-13630513
]
Chris Nauroth commented on HADOOP-9043:
---
I was just trying to resume progress on this to fix the remaining test failures
related to Windows symlinks on the local file system. I discovered a new
problem. The trunk code still has a Unix command dependency on the readlink
command for determining the target of a symlink. (See
{{RawLocalFs#readLink}}.) This is the source of some of the test failures.
I propose the addition of a new "winutils readlink" command. The logic of this
command would be:
# Call {{CreateFile}} with {{OPEN_EXISTING}} and
{{FILE_FLAG_OPEN_REPARSE_POINT}}.
**
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx
# Call {{GetFileInformationByHandle}}.
**
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364952(v=vs.85).aspx
# Check the {{BY_HANDLE_FILE_INFORMATION}} for the presence of
{{FILE_ATTRIBUTE_REPARSE_POINT}}.
**
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363788(v=vs.85).aspx
**
http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117(v=vs.85).aspx
# If not a reparse point, exit with code 1 and print nothing to stdout. (This
is what Unix readlink does.)
# Call {{DeviceIoControl}} with {{FSCTL_GET_REPARSE_POINT}}.
** http://msdn.microsoft.com/en-us/library/aa363216(v=VS.85).aspx
** http://msdn.microsoft.com/en-us/library/aa364571.aspx
# Get the {{REPARSE_DATA_BUFFER}} structure.
** http://msdn.microsoft.com/en-us/library/ff552012.aspx
# Check if {{ReparseTag}} is {{IO_REPARSE_TAG_SYMLINK}}.
**
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365511(v=vs.85).aspx
# If not {{IO_REPARSE_TAG_SYMLINK}}, then...?
# Get target from {{SymbolicLinkReparseBuffer}}.
# Print target to stdout and exit with code 0. (This is what Unix readlink
does.)
Could someone with more Windows expertise review this and comment on whether or
not the logic looks correct? There are a few edge cases that I'm not sure how
to handle. What should we do if the reparse point is not a symlink (i.e.
junction point)? MSDN also mentions that some reparse points may have a
different data structure associated with them, a {{REPARSE_GUID_DATA_BUFFER}},
and I'm not sure what special handling is required around that.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13605439#comment-13605439 ] Chuan Liu commented on HADOOP-9043: --- Sorry for the typo: symlink -path- should be symlink patch. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13605438#comment-13605438 ] Chuan Liu commented on HADOOP-9043: --- +1 on the symlink path, and agree with Chris to disable the test on Windows. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599006#comment-13599006
]
Chris Nauroth commented on HADOOP-9043:
---
Tightening up the validation is great, but we still have the problem that
{{TestLocalFSFileContextSymlink}} passes on Linux but fails on Windows. The
current state of the discussion here is that winutils will enforce validation
rules to reject creation of invalid/dangling symlinks. Since this diverges
intentionally from the Linux behavior, do we want to update
{{TestLocalFSFileContextSymlink}} to add {{assumeTrue(!Shell.WINDOWS)}} on the
invalid tests?
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595350#comment-13595350 ] Ivan Mitic commented on HADOOP-9043: bq. Forward paths in the source already appear to be rejected. Do you see otherwise? Do you see this rejected by winutils? Please add a test anyways to make sure this is the case. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595345#comment-13595345 ] Arpit Agarwal commented on HADOOP-9043: --- Forward paths in the source already appear to be rejected. Do you see otherwise? Looking into the test case. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595338#comment-13595338 ] Ivan Mitic commented on HADOOP-9043: Thanks Arpit, this is good. A few comments, mainly minor: 1. Can we do the same check for both source and target? 2. Please log additional information about the problem to stderr. This will help see what went wrong on the Java side. 3. Please add a unittest to TestWinUtils that validates the fix > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 1-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Fix For: 3.0.0, 1-win > > Attachments: HADOOP-9043.branch-1.2.patch, > HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, > HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595319#comment-13595319
]
Hadoop QA commented on HADOOP-9043:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572422/HADOOP-9043.trunk.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.util.TestWinUtils
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2283//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2283//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2283//console
This message is automatically generated.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 3.0.0, 1-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 1-win
>
> Attachments: HADOOP-9043.branch-1.2.patch,
> HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch,
> HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584877#comment-13584877 ] Chuan Liu commented on HADOOP-9043: --- To summary the cause of the problem: Windows symlink creation API - CreateSymbolicLink() - needs to know if the destination is a file or directory. We called DirectoryCheck() to tell if the given path is a file or directory. This leads to two issues: 1) The path needs exist in order for the symlink to be created successfully. This diverges from the Unix behavior, where invalid link can be created regardless existence of the destination. 2) We have an inconsistency in Windows API in treating forward slashes and backslashes. In this case, DirectoryCheck() will accept forward slashes and backslashes when checking if a path is directory or file, while the link contains forward slashes will become invalid symlinks even if the destination exists as indicated by DirectoryCheck(). We will not fix the first problem as this roots in how OS handle symlinks and API differences in symlink creation. For 2), I think both Arpit's and Ivan's fixes should work. I think Ivan's fix is safer in general by not considering paths containing forward slashes. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583766#comment-13583766 ] Ivan Mitic commented on HADOOP-9043: Thanks Arpit. I am fine with failing early in winutils#symlink in case we detect a forward slash. Seems useful for this scenario given that the symlink creation succeeds, but the link is not working. However, I would not recommend trying to support both forward and backward slashes in wintuils. Java is meant to solve this problem for us, so let's just build on top of it. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583384#comment-13583384 ] Arpit Agarwal commented on HADOOP-9043: --- Hi Ivan, Chris, I think the behavior of winutils is orthogonal to what the Java code invoking it does. If we are shipping winutils with the distribution it should do the right thing, which is to either fail the creation of unusable symlinks or handle the path conversion. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583336#comment-13583336
]
Chris Nauroth commented on HADOOP-9043:
---
{quote}
To fix this specific problem, we would want to change FileUtil#symlink to
"normalize the slashes". If you take a look at the branch-1-win code, it
already does this so you can just forward port the patch.
{quote}
This code was already ported to branch-trunk-win several months ago:
{code}
public static int symLink(String target, String linkname) throws IOException{
// Run the input paths through Java's File so that they are converted to the
// native OS form
File targetFile = new File(target);
File linkFile = new File(linkname);
{code}
I believe this jira is no longer valid, at least under its current description.
When I filed it, I didn't realize that Windows requires slightly different API
calls for creating a symlink that targets a directory vs. a file. Therefore,
winutils really does need to call {{DirectoryCheck}} to determine the type of
target. As a consequence, winutils differs from Unix ln in that it cannot
create a dangling symlink. (It has no way of knowing whether the caller is
trying to create a "dangling file symlink" or a "dangling directory symlink".)
I believe that both the Java code and the C code are doing the right thing for
us now, without further changes.
The remaining issue is the failure of {{TestLocalFSFileContextSymlink}} on
Windows, which is what prompted me to file this jira initially. We now know
that this is an inevitable platform difference, so let's use
{{Assert.assumeTrue(!Shell.WINDOWS)}} to skip the tests that can't possibly
pass on Windows. If needed, we could also add more tests to cover Windows
behavior, guarded with {{Assert.assumeTrue(Shell.WINDOWS)}}. AFAIK, Hadoop
product code does not actually require the ability to create a dangling
symlink, and the test suite is just trying to cover ln functionality
exhaustively.
I propose that we close this jira as invalid and create a new one to fix the
tests.
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 1-win, trunk-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch
>
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13582746#comment-13582746 ] Ivan Mitic commented on HADOOP-9043: Thanks Arpit! I just looked at the patch, and I would prefer we go a slightly different route. We should always assume that paths that enter winutils have only backshashes. The translation (/ -> \) should be happening on the Java side. To fix this specific problem, we would want to change FileUtil#symlink to "normalize the slashes". If you take a look at the branch-1-win code, it already does this so you can just forward port the patch. Does this make sense, or am I missing something? FileUtil APIs and APIs that work with local file system in general should be accepting java.io.File for params that are local files (instead of Strings or Paths), so that we can let Java handle cross platform path "normalization". Deprecating existing APIs would be an overhead especially for projects that tool a dependency, so we should just keep this in mind going forward for the new APIs we add. > winutils can create unusable symlinks > - > > Key: HADOOP-9043 > URL: https://issues.apache.org/jira/browse/HADOOP-9043 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch > > > In general, the winutils symlink command rejects attempts to create symlinks > targeting a destination file that does not exist. However, if given a > symlink destination with forward slashes pointing at a file that does exist, > then it creates the symlink with the forward slashes, and then attempts to > open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497813#comment-13497813
]
Chris Nauroth commented on HADOOP-9043:
---
This is best illustrated through an example. See below. Suppose there is a
file at foo\bar\hello.txt, and you want to create a symlink to it named
hello.txt in the current directory. Running "winutils symlink hello.txt
foo\bar\hello.txt" (using back slashes) works correctly. However, running
"winutils symlink hello.txt foo/bar/hello.txt" (using forward slashes) appears
to succeed, but then the file isn't really accessible through the symlink.
This is a minor winutils bug. The Java codebase is supposed to take care of
calling winutils with the correct path separator. This does make it more
challenging to troubleshoot problems, because the failure doesn't occur at the
source of the problem (symlink creation). Instead, the failure occurs later
when a process tries to access a file through the symlink.
This shows that winutils generally rejects attempts to create dangling symlinks.
{code}
C:\test2>\dep\hadoop-3.0.0-SNAPSHOT\bin\winutils.exe symlink baz doesNotExist
DirectoryCheck error (2): The system cannot find the file specified.
C:\test2>\dep\hadoop-3.0.0-SNAPSHOT\bin\winutils.exe symlink baz
subdir\doesNotExist
DirectoryCheck error (3): The system cannot find the path specified.
{code}
This shows that passing forward slashes can create a bad symlink.
{code}
C:\test2>dir
Volume in drive C has no label.
Volume Serial Number is 9898-B272
Directory of C:\test2
11/14/2012 12:24 PM .
11/14/2012 12:24 PM ..
0 File(s) 0 bytes
2 Dir(s) 5,014,097,920 bytes free
C:\test2>mkdir foo\bar
C:\test2>echo hello > foo\bar\hello.txt
C:\test2>\dep\hadoop-3.0.0-SNAPSHOT\bin\winutils.exe symlink hello.txt foo\bar\h
ello.txt
C:\test2>dir
Volume in drive C has no label.
Volume Serial Number is 9898-B272
Directory of C:\test2
11/14/2012 12:24 PM .
11/14/2012 12:24 PM ..
11/14/2012 12:24 PM foo
11/14/2012 12:24 PM hello.txt [foo\bar\hello.txt]
1 File(s) 0 bytes
3 Dir(s) 5,014,097,920 bytes free
C:\test2>type hello.txt
hello
C:\test2>del hello.txt
C:\test2>\dep\hadoop-3.0.0-SNAPSHOT\bin\winutils.exe symlink hello.txt foo/bar/h
ello.txt
C:\test2>dir
Volume in drive C has no label.
Volume Serial Number is 9898-B272
Directory of C:\test2
11/14/2012 12:24 PM .
11/14/2012 12:24 PM ..
11/14/2012 12:24 PM foo
11/14/2012 12:24 PM hello.txt [foo/bar/hello.txt]
1 File(s) 0 bytes
3 Dir(s) 5,014,097,920 bytes free
C:\test2>type hello.txt
The system cannot find the file specified.
{code}
> winutils can create unusable symlinks
> -
>
> Key: HADOOP-9043
> URL: https://issues.apache.org/jira/browse/HADOOP-9043
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 1-win, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>
> In general, the winutils symlink command rejects attempts to create symlinks
> targeting a destination file that does not exist. However, if given a
> symlink destination with forward slashes pointing at a file that does exist,
> then it creates the symlink with the forward slashes, and then attempts to
> open the file through the symlink will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
