[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687988#comment-13687988
]
Hudson commented on HADOOP-9637:
Integrated in Hadoop-Mapreduce-trunk #1462 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/])
HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed
by Chuan Liu. (Revision 1494341)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494341
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
> HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687943#comment-13687943
]
Hudson commented on HADOOP-9637:
Integrated in Hadoop-Hdfs-trunk #1435 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/])
HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed
by Chuan Liu. (Revision 1494341)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494341
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
> HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687842#comment-13687842
]
Hudson commented on HADOOP-9637:
Integrated in Hadoop-Yarn-trunk #245 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/245/])
HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed
by Chuan Liu. (Revision 1494341)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494341
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
> HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687292#comment-13687292
]
Hudson commented on HADOOP-9637:
Integrated in Hadoop-trunk-Commit #3971 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3971/])
HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. Contributed
by Chuan Liu. (Revision 1494341)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494341
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chmod.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/chown.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/ls.c
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
> HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687168#comment-13687168
]
Hadoop QA commented on HADOOP-9637:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588442/HADOOP-9637-trunk.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2665//console
This message is automatically generated.
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.3.patch,
> HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686983#comment-13686983
]
Chris Nauroth commented on HADOOP-9637:
---
Thanks for addressing my questions. It all makes sense. I have just one more
comment about the new patch.
{code}
if (Shell.WINDOWS) {
int i = owner.indexOf('\\');
if (i != -1)
this.owner = owner.substring(i + 1);
}
{code}
This code is identical for username and group, so let's refactor it behind a
helper function. We also can add comments stating that the reason for this is
to remove domain.
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686432#comment-13686432
]
Hadoop QA commented on HADOOP-9637:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588295/HADOOP-9637-trunk.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2663//console
This message is automatically generated.
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.2.patch, HADOOP-9637-trunk.patch
>
>
> In the YARN, nodemanager need to enforce the log file can only be accessed by
> the owner. At various places, {{SecureIOUtils.openForRead()}} was called to
> enforce this check. We don't have {{NativeIO.Posix.getFstat()}} used by
> {{SecureIOUtils.openForRead()}} on Windows, and this make the check fail on
> Windows. The YARN unit tests
> TestAggregatedLogFormat.testContainerLogsFileAccess and
> TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of
> this.
> The JIRA try to provide a Windows implementation of
> {{NativeIO.Posix.getFstat()}}.
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686247#comment-13686247
]
Chuan Liu commented on HADOOP-9637:
---
Thanks for reviewing, Chris!
>If I understand correctly, this is intended to remove the domain portion from
>a username. (Am I understanding it correctly?) Does Windows ever prepend
>domain to the group name too? Do we need similar logic for initializing
>this.group?
You are right. We also need this for group name. I will fix this in a new patch.
>It seems that this logic would treat user DOMAIN1\cnauroth the same as
>DOMAIN2\cnauroth, when in reality the same username in 2 different domains
>might not be the same person. Consumers of fstat would end up seeing
>"cnauroth" as the owner without being able to differentiate between the 2
>different users in 2 different domains. Is this acceptable?
This is by design right now. UserGroupInformation, NativeIO.Windows.getOwner(),
and a few other places all assume this pattern, i.e. username without domain.
(UserGroupInformation does not explicitly strip the domain, however, the Java
API it used only returns username.) There are some discussions on this in
HADOOP-8455. The main scenario we want to support (for unsecure Hadoop on
Windows) is to allow local users of the same name to run Hadoop cluster without
a domain controller. For example, users can create local user 'Alex' on the two
machines 'Win1' and 'Win2', and run Hadoop under the local user 'Alex'. I think
it is acceptable for now; I have outlined some thoughts on how we can improve
this in HADOOP-8455.
>Minor nit: use 2-space indentation in the above code.
Thanks for pointing this out; will fix in new patch.
>Does this mean that we need to subtract 1 from cchPathLen earlier, such as
>when we assign cchPathLen = dwRtnCode?
Not necessary. I have similar thoughts when I wrote the code. I intentionally
make the buffer size 1 char larger; so the last character is the NULL
terminator. Because without NULL terminator, some functions, e.g. wcslen(),
will fail on the string, including some Windows function like
GetNamedSecurityInfo() that is called on the path later. I think the MSDN
document means the buffer only need to be as large as string length without
null terminator. However, it does not require the buffer to be exact size. You
may notice the example given at the MSDN page also used a larger buffer to save
the path.
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.patch
>
>
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9637) Adding Native Fstat for Windows as needed by YARN
[
https://issues.apache.org/jira/browse/HADOOP-9637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685978#comment-13685978
]
Chuan Liu commented on HADOOP-9637:
---
>May be you should edit the title to be something in the line of "Adding Native
>Fstat for Windows as needed by YARN".
Thanks for the suggestion! I have changed the title; will update the
description shortly.
> Adding Native Fstat for Windows as needed by YARN
> -
>
> Key: HADOOP-9637
> URL: https://issues.apache.org/jira/browse/HADOOP-9637
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
> Attachments: HADOOP-9637-trunk.patch
>
>
> TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on
> Windows. The test case try to simulate a situation where first log file is
> owned by different user (probably symlink) and second one by the user itself.
> In this situation, the attempt to try to aggregate the logs should fail with
> the error message "Owner ... for path ... did not match expected owner ...".
> The check on file owner happens at {{AggregatedLogFormat.write()}} method.
> The method calls {{SecureIOUtils.openForRead()}} to read the log files before
> writing out to the OutputStream.
> {{SecureIOUtils.openForRead()}} use {{NativeIO.Posix.getFstat()}} to get the
> file owner and group. We don't have {{NativeIO.Posix.getFstat()}}
> implementation on Windows; thus, the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
