[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645586#comment-13645586 ] Hudson commented on HADOOP-9413: Integrated in Hadoop-Mapreduce-trunk #1415 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1415/]) HADOOP-9413. Add common utils for File#setReadable/Writable/Executable & File#canRead/Write/Execute that work cross-platform. Contributed by Ivan Mitic. (Revision 1477376) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477376 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TemporarySocketDirectory.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645547#comment-13645547 ] Hudson commented on HADOOP-9413: Integrated in Hadoop-Hdfs-trunk #1388 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1388/]) HADOOP-9413. Add common utils for File#setReadable/Writable/Executable & File#canRead/Write/Execute that work cross-platform. Contributed by Ivan Mitic. (Revision 1477376) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477376 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TemporarySocketDirectory.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645448#comment-13645448 ] Hudson commented on HADOOP-9413: Integrated in Hadoop-Yarn-trunk #199 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/199/]) HADOOP-9413. Add common utils for File#setReadable/Writable/Executable & File#canRead/Write/Execute that work cross-platform. Contributed by Ivan Mitic. (Revision 1477376) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477376 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TemporarySocketDirectory.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644985#comment-13644985 ] Hudson commented on HADOOP-9413: Integrated in Hadoop-trunk-Commit #3692 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3692/]) HADOOP-9413. Add common utils for File#setReadable/Writable/Executable & File#canRead/Write/Execute that work cross-platform. Contributed by Ivan Mitic. (Revision 1477376) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477376 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TemporarySocketDirectory.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShell.java > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644935#comment-13644935 ] Bikas Saha commented on HADOOP-9413: Looks good to me. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644927#comment-13644927 ] Suresh Srinivas commented on HADOOP-9413: - +1 for the patch. [~bikassaha] if you do not have further comments, I am going to commit this patch in a short while. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644654#comment-13644654
]
Chris Nauroth commented on HADOOP-9413:
---
Bikas, the best explanation of the issue mentioned by Ivan is in his latest
patch, in the comments for {{FileUtil#setExecutable}}. I've pasted that
comment below.
More details are discussed in HADOOP-9525, within the preliminary patch that
Ivan provided for adding more tests of chmod on Windows.
{code}
+ /**
+ * Platform independent implementation for {@link
File#setExecutable(boolean)}
+ * File#setExecutable does not work as expected on Windows.
+ * Note: revoking execute permission on folders does not have the same
+ * behavior on Windows as on Unix platforms. Creating, deleting or renaming
+ * a file within that folder will still succeed on Windows.
+ * @param f input file
+ * @param executable
+ * @return true on success, false otherwise
+ */
+ public static boolean setExecutable(File f, boolean executable) {
{code}
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch,
> HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644338#comment-13644338 ] Bikas Saha commented on HADOOP-9413: bq. Did only a minor update to the patch to note some subtle differences between setExecutable on Windows and Unix. Check HADOOP-9525 for additional details. Thanks Chris for the great comment on HDFS-4610 that made me realize this! Without a summary of the issue or links to exact explanatory comments in other jiras, its hard to ascertain what the issue is. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644247#comment-13644247
]
Hadoop QA commented on HADOOP-9413:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580926/HADOOP-9413.commonfileutils.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2490//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2490//console
This message is automatically generated.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch,
> HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644236#comment-13644236 ] Chris Nauroth commented on HADOOP-9413: --- +1 for the current patch. Thank you, Ivan, for the additional comment! > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643710#comment-13643710 ] Chris Nauroth commented on HADOOP-9413: --- HADOOP-9290 just got committed to fix the native library loading. Ivan, can we proceed with committing HADOOP-9413 now? (I am still +1 after the recent rebase.) > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, > HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640132#comment-13640132
]
Hadoop QA commented on HADOOP-9413:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580237/HADOOP-9413.commonfileutils.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2469//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2469//console
This message is automatically generated.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch,
> HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640077#comment-13640077 ] Ivan Mitic commented on HADOOP-9413: Wrong Jira, please ignore my previous comment :) > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640074#comment-13640074
]
Ivan Mitic commented on HADOOP-9413:
Thanks Arpit and Chris for preparing this patch! I kicked off a full test run
on Windows to validate the approach.
In parallel, I have a few questions/comments on the latest patch:
1. hadoop-project/pom.xml: It seems that the currently defined value for
LD_LIBRARY_PATH is invalid. I think it should have the following value:
{code}
${env.LD_LIBRARY_PATH}:${project.build.directory}/native/target/usr/local/lib:${basedir}/../../hadoop-common-project/hadoop-common/target/bin
{code}
2. hadoop-project/pom.xml: Assuming that we specified a valid value for
LD_LIBRARY_PATH above, the following section might not be needed:
{code}
${env.PATH};${basedir}/../../hadoop-common-project/hadoop-common/target/bin
${env.PATH};${hadoop.common.build.dir}/bin
{code}
3. Other pom.xml files: Can we just define a new value for LD_LIBRARY_PATH next
to HADOOP_HOME and point it to the appropriate /target/bin folder in every
subproject? This would provide a nice symmetry with how we locate winutils on
Windows.
4. Might make sense to either remove hadoop.common.build.dir or use it for both
winutils.exe and hadoop.dll
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639683#comment-13639683 ] Chris Nauroth commented on HADOOP-9413: --- Ivan, I just uploaded a new patch on HADOOP-9290. Can you try running with that to see if it resolves the test failures you saw with HADOOP-9413? > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639398#comment-13639398 ] Ivan Mitic commented on HADOOP-9413: bq. Could this be related to HADOOP-9290? On that one, we were hoping to find a graceful fix in one place. Perhaps at this point we need to fall back on the option of overriding with a different relative path per each sub-module pom.xml. Thanks Chris, I think this is exactly what is going on. Will review the patch for HADOOP-9290 and see if I can somehow help there. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639326#comment-13639326 ] Chris Nauroth commented on HADOOP-9413: --- +1 for the patch. This is great! The code looks good. I verified successful test runs on Mac, Windows, and Linux w/native. Just a reminder: we are still holding off on committing, pending resolution of some downstream test failures. (See prior comments from Ivan and me.) > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638580#comment-13638580
]
Chris Nauroth commented on HADOOP-9413:
---
{quote}
I noticed that hadoop.dll is not added to the java library path in the
downstream Hadoop projects, causing some tests to fail with
UnsatisfiedLinkError.
{quote}
Could this be related to HADOOP-9290? On that one, we were hoping to find a
graceful fix in one place. Perhaps at this point we need to fall back on the
option of overriding with a different relative path per each sub-module pom.xml.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638565#comment-13638565 ] Ivan Mitic commented on HADOOP-9413: Thanks Chuan for the review. I noticed that hadoop.dll is not added to the java library path in the downstream Hadoop projects, causing some tests to fail with UnsatisfiedLinkError. Let's hold on committing this Jira until I prepare a fix for that. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637790#comment-13637790
]
Hadoop QA commented on HADOOP-9413:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579778/HADOOP-9413.commonfileutils.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2464//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2464//console
This message is automatically generated.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637786#comment-13637786 ] Chuan Liu commented on HADOOP-9413: --- +1 Thanks for fixing the previous missed clean up as well! > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.2.patch, > HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637411#comment-13637411
]
Bikas Saha commented on HADOOP-9413:
Thanks for doing this. Much better approach and functionally correct!
Enums with defined value will make this unnecessary. Minor though but typesafe.
{code}
+ if (desiredAccess != ACCESS_READ && desiredAccess != ACCESS_WRITE
+ && desiredAccess != ACCESS_EXECUTE) {
{code}
The previous code was returning error when this failed. Not necessary anymore?
{code}
- if (!AuthzFreeContext(hAuthzClientContext))
+
+GetEffectiveRightsForSidEnd:
+ if (hManager != NULL)
{
-ret = GetLastError();
-goto GetEffectiveRightsForSidEnd;
+(void)AuthzFreeResourceManager(hManager);
+ }
+ if (hAuthzClientContext != NULL)
+ {
+(void)AuthzFreeContext(hAuthzClientContext);
}
{code}
How about changing these to setWritable()? Thus testing the other new methods
too. + end to end symmetry check also achieved.
{code}
+FileUtil.chmod(testFile.getAbsolutePath(), "u-r");
+assertFalse(NativeIO.Windows.access(testFile.getAbsolutePath(),
+NativeIO.Windows.ACCESS_READ));
{code}
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637312#comment-13637312
]
Hadoop QA commented on HADOOP-9413:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579683/HADOOP-9413.commonfileutils.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2462//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2462//console
This message is automatically generated.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.2.patch,
> HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618567#comment-13618567 ] Chris Nauroth commented on HADOOP-9413: --- The plan sounds great. Thanks, Ivan. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618560#comment-13618560 ] Ivan Mitic commented on HADOOP-9413: Thanks Chris and Bikas for the review comments! Bikas, the current patch gets things moving forward on Windows and there are no changes to the current behavior on non-Windows platforms. The patch just builds on top of the current design where Hadoop sees Windows as a POSIX compatible OS w.r.t. local file system access. Given Chris’ input from above where the previous logic from HADOOP-8973 works only on folders, my take is to proceed with one of the following: a. The approach from the current patch b. JNI implementation which will handle access checks on the ACLs level and provide a more natural integration with Windows Let me spend a few days investigating what would it take to implement b) and understanding the implications of this approach for Hadoop on Windows. It is definitely worth spending a bit of time on this. If this turns out to be strait forward, I’ll address it as part of this patch. If not, we can start with a new Jira and iterate thru problems. Let me know if this sounds good. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618472#comment-13618472 ] Bikas Saha commented on HADOOP-9413: I am in favor of any proposal that leaves the code and expected functionality consistent. Ideally, structurally similar and functionally correct. Structurally similar but functionally broken may be acceptable for the short term as long as non-Windows continues to run correctly. What I would be wary of is leaving the code in a situation where it works in A but not in B because that is a pain to read, debug and maintain. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615556#comment-13615556
]
Chris Nauroth commented on HADOOP-9413:
---
{quote}
However, as long as the abstractions are solid, I think we are good. Let me
know what you think.
{quote}
Yes, I am definitely in favor of the structure of this patch: providing common
utility wrappers for these methods. I think the only open question is whether
the implementation should revert the logic of HADOOP-8973 like this.
{quote}
Chris already has code that does the expected thing for the scenario in which
the running process is checking whether it has read/write/execute permissions
on a directory.
{quote}
A couple of caveats on this:
# The code I wrote for HADOOP-8973 works only for directories, and we need
these common APIs to handle both files and directories. See Ivan's patches on
HDFS-4610 and YARN-506 for some examples where the codebase needs to get/set
permissions on individual files.
# There was also the question of performance, because my code actually
performed file access and forked new processes to implement the access checks.
These were not problems for HADOOP-8973, because {{DiskChecker}} is used only
for directories and it's used rarely enough that the performance impact would
likely be unnoticeable.
I'm wondering if it's time for us to write a JNI call to {{AccessCheck}}:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa374815(v=vs.85).aspx
This would only be used if (Windows && JDKversion < 7). Otherwise, we expect
the JDK APIs to work.
Obviously, this will take longer to implement. A potential compromise would be
to go with the patch's current implementation of the new functions, but leave
{{DiskChecker}} as is temporarily, just until we get the JNI call written.
That way, the other permission checks in the codebase get pretty close to
working (not 100% correct, but certainly better than the current state), and we
still maintain full correctness for the very important {{DiskChecker}} piece.
Ivan and Bikas, do you have any follow-up thoughts?
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615506#comment-13615506 ] Bikas Saha commented on HADOOP-9413: Chris already has code that does the expected thing for the scenario in which the running process is checking whether it has read/write/execute permissions on a directory. We could move them into helper functions and use them. This is important because after this check is successful the process goes ahead and performs the action that depends on the check. So my preference would be to use the code that provides the expected functionality. We can improve that code later on. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614973#comment-13614973 ] Ivan Mitic commented on HADOOP-9413: Thanks for the review Chris! bq. This seems to revert the decisions made in HADOOP-8973 that checking for an owner match and FsAction#implies is insufficient in the presence of other permission mechanisms, such as ACLs. In this case though, your patch keeps this logic guarded behind an if (Windows) check, so it wouldn't sacrifice existing functionality for any existing Linux deployments that use POSIX ACLs. Just so I'm clear, is this a proposal that we accept it as a known limitation on Windows right now, with potential follow-up work to address it later (via JDK7 or possibly JNI calls)? That seems to be implicit in the patch, but I want to make sure I understand. I added a comment above asking for opinion on this one. This approach provides full symmetry between can* and set* functions on Windows, what I think is a good thing for now. I agree that we'll want to improve this eventually (this could be JDK7 or JNI). However, as long as the abstractions are solid, I think we are good. Let me know what you think. bq. Also, can you please clarify the second part of the if condition? This is checking if one of the user's group memberships has the same name as the file owner. Is it supposed to check the file group instead of the file owner? Thanks, will add a comment to clarify this part. On Windows, the owner can also be a group. One use-case is if I'm running as a member of the admins group (elevated), and create a file, the file will be owned by the admins group. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612385#comment-13612385
]
Chris Nauroth commented on HADOOP-9413:
---
Hi, Ivan.
{code}
+if (ugi.getShortUserName().equals(status.getOwner())
+|| Arrays.asList(ugi.getGroupNames()).contains(status.getOwner()))
{
+ if (user.implies(FsAction.READ)) {
+return true;
+ }
+}
{code}
This seems to revert the decisions made in HADOOP-8973 that checking for an
owner match and {{FsAction#implies}} is insufficient in the presence of other
permission mechanisms, such as ACLs. In this case though, your patch keeps
this logic guarded behind an if (Windows) check, so it wouldn't sacrifice
existing functionality for any existing Linux deployments that use POSIX ACLs.
Just so I'm clear, is this a proposal that we accept it as a known limitation
on Windows right now, with potential follow-up work to address it later (via
JDK7 or possibly JNI calls)? That seems to be implicit in the patch, but I
want to make sure I understand.
Also, can you please clarify the second part of the if condition? This is
checking if one of the user's group memberships has the same name as the file
owner. Is it supposed to check the file group instead of the file owner?
Thanks!
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612367#comment-13612367
]
Hadoop QA commented on HADOOP-9413:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574297/HADOOP-9413.commonfileutils.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2358//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2358//console
This message is automatically generated.
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612362#comment-13612362 ] Ivan Mitic commented on HADOOP-9413: Thanks Chris! bq. I've spotted another instance of this problem, in YARN nodemanager NodeHealthScriptRunner#shouldRun, which checks File#canExecute, and we've already seen that this is unreliable on Windows. The problem manifests as a test failure in TestNodeHealthService. I created YARN-506 and attached a patch that makes the appropriate changes in the yarn codebase. Btw, I noticed a single place in the mapreduce project which calls File#canExecute. Will file a separate Jira for this. bq. 1. At the moment, I did not expose File#set* APIs that accept the ownerOnly parameter. Adding this is strait forward, the only question is whether we need it. I didn't find use-cases in Common and HDFS projects, but I've seen some in Yarn (only on one or two places, and those could be changed to use chmod instead). To also respond to my earlier comment from above. I don't think we need to expose these APIs, at least not yet. I found only a single occurrence where these APIs are used, and using chmod as a replacement seems appropriate. > Introduce common utils for File#setReadable/Writable/Executable and > File#canRead/Write/Execute that work cross-platform > --- > > Key: HADOOP-9413 > URL: https://issues.apache.org/jira/browse/HADOOP-9413 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 3.0.0 > > Attachments: HADOOP-9413.commonfileutils.patch > > > So far, we've seen many unittest and product bugs in Hadoop on Windows > because Java's APIs that manipulate with permissions do not work as expected. > We've addressed many of these problems on one-by-one basis (by either > changing code a bit or disabling the test). While debugging the remaining > unittest failures we continue to run into the same patterns of problems, and > instead of addressing them one-by-one, I propose that we expose a set of > equivalent wrapper APIs that will work well for all platforms. > Scanning thru the codebase, this will actually be a simple change as there > are very few places that use File#setReadable/Writable/Executable and > File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). > HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[
https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609323#comment-13609323
]
Chris Nauroth commented on HADOOP-9413:
---
I've spotted another instance of this problem, in YARN nodemanager
{{NodeHealthScriptRunner#shouldRun}}, which checks {{File#canExecute}}, and
we've already seen that this is unreliable on Windows. The problem manifests
as a test failure in {{TestNodeHealthService}}.
I expect I'll get a chance to review this proposal within the next few days.
Thanks for investigating this, Ivan!
> Introduce common utils for File#setReadable/Writable/Executable and
> File#canRead/Write/Execute that work cross-platform
> ---
>
> Key: HADOOP-9413
> URL: https://issues.apache.org/jira/browse/HADOOP-9413
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: 3.0.0
>
> Attachments: HADOOP-9413.commonfileutils.patch
>
>
> So far, we've seen many unittest and product bugs in Hadoop on Windows
> because Java's APIs that manipulate with permissions do not work as expected.
> We've addressed many of these problems on one-by-one basis (by either
> changing code a bit or disabling the test). While debugging the remaining
> unittest failures we continue to run into the same patterns of problems, and
> instead of addressing them one-by-one, I propose that we expose a set of
> equivalent wrapper APIs that will work well for all platforms.
> Scanning thru the codebase, this will actually be a simple change as there
> are very few places that use File#setReadable/Writable/Executable and
> File#canRead/Write/Execute (5 files in Common, 9 files in HDFS).
> HADOOP-8973 contains additional context on the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
