[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598704#comment-13598704 ] Hudson commented on HADOOP-8973: Integrated in Hadoop-Yarn-trunk #152 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/152/]) HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs. Contributed by Chris Nauroth. (Revision 1454889) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1454889 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDiskChecker.java DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 1-win Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598783#comment-13598783 ] Hudson commented on HADOOP-8973: Integrated in Hadoop-Hdfs-trunk #1341 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1341/]) HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs. Contributed by Chris Nauroth. (Revision 1454889) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1454889 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDiskChecker.java DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 1-win Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598831#comment-13598831 ] Hudson commented on HADOOP-8973: Integrated in Hadoop-Mapreduce-trunk #1369 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1369/]) HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs. Contributed by Chris Nauroth. (Revision 1454889) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1454889 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDiskChecker.java DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 1-win Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599255#comment-13599255 ] Andrey Klochkov commented on HADOOP-8973: - Seems this patch has broken test TestDataDirs in trunk. On a side note, shouldn't we improve patch verifier to catch such issues? The reason the issue wasn't detected is that the robot doesn't run tests for modules which depend on the modified ones. Shouldn't we fix it? DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 1-win Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598340#comment-13598340 ] Hudson commented on HADOOP-8973: Integrated in Hadoop-trunk-Commit #3446 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3446/]) HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs. Contributed by Chris Nauroth. (Revision 1454889) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1454889 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDiskChecker.java DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598542#comment-13598542
]
Chris Nauroth commented on HADOOP-8973:
---
Thank you, Suresh. Thank you, everyone, for the code reviews and other
assistance with resolving this. I've filed HADOOP-9390 for follow-up on the
issue of tests failing when run as root/Administrator. I've file HADOOP-9391
for the follow-up work of refactoring the {{Shell}} methods to accept {{File}}
arguments instead of strings.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0, 1-win
Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch,
HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598016#comment-13598016
]
Chris Nauroth commented on HADOOP-8973:
---
Ivan, thanks for the help tracking this down. Very interesting find. I have
confirmed that the existing tests fail when running as root on Linux, just like
what you're experiencing when running as admin on Windows. It's unrelated to
this patch. This might explain a few complaints I've seen on the mailing lists
from new engineers trying to contribute to Hadoop, but finding some unit tests
failing immediately. I'll try to track down those threads and help them out.
I agree with sticking to the File APIs, because they give us the absolute
truth as you said. If the user is root/admin, then the disk is usable.
We're just left with a test issue. We could change the tests to detect
root/admin at runtime and then either skip the invalid tests with
{{assumeTrue}} or run different assertions. On Linux, we could check the
username against root. On Windows, would it be sufficient to check that user
is Administrator, or is it more complex than that?
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch,
HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598115#comment-13598115 ] Bikas Saha commented on HADOOP-8973: What is the issue? DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598122#comment-13598122
]
Ivan Mitic commented on HADOOP-8973:
{quote}
Ivan, thanks for the help tracking this down. Very interesting find. I have
confirmed that the existing tests fail when running as root on Linux, just like
what you're experiencing when running as admin on Windows. It's unrelated to
this patch. This might explain a few complaints I've seen on the mailing lists
from new engineers trying to contribute to Hadoop, but finding some unit tests
failing immediately. I'll try to track down those threads and help them out.
I agree with sticking to the File APIs, because they give us the absolute
truth as you said. If the user is root/admin, then the disk is usable.
{quote}
Thanks Chris for checking this out. Given that we have a symmetry with Unix I
am +1 on the current patch. Maybe just add a comment to the test saying that it
is expected to fail if run as root on Unix or in elevated context on Windows?
{quote}
We're just left with a test issue. We could change the tests to detect
root/admin at runtime and then either skip the invalid tests with assumeTrue or
run different assertions. On Linux, we could check the username against root.
On Windows, would it be sufficient to check that user is Administrator, or is
it more complex than that?
{quote}
For Windows, it won't be as simple as string compare. Even checking whether a
user is a member of the Administrators group is not the right behavior. We have
to check whether the process is running elevated or not. This could be done in
native code or via a cmd script as described
[here|http://stackoverflow.com/questions/7985755/how-to-detect-if-cmd-is-running-as-administrator-has-elevated-privileges].
We could expose something like TestUtils#IsRunningElevated and disable/relax
the test in that case, but this would be a separate Jira.
bq. What is the issue?
Bikas, the issue is that the test fails when executed in the elevated context
on Windows, or as root on Linux.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch,
HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598124#comment-13598124 ] Chris Nauroth commented on HADOOP-8973: --- Great, thanks Ivan. We know now that we've fixed the bug in this jira, so let's proceed with that. To the committer: just to clarify what needs to be committed, please commit HADOOP-8973.3.patch to trunk and HADOOP-8973-branch-1-win.3.patch to branch-1-win. Thank you! DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598151#comment-13598151 ] Bikas Saha commented on HADOOP-8973: bq. the issue is that the test fails when executed in the elevated context on Windows, or as root on Linux. Do we know why thats the case? DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598163#comment-13598163 ] Chris Nauroth commented on HADOOP-8973: --- Yes, please read the prior comments from Ivan and me for full explanation. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596936#comment-13596936 ] Bikas Saha commented on HADOOP-8973: Thanks for following up on my comments. Looks good. Minor comment. I think we added a check for java version and have used it to enable java api's for java 7, if available. If java 7 File.canWrite etc are going to work on Windows (like mentioned in some comments above) then we could enable the API path when (!Windows or Java6). Please do open a jira to fix this issue in branch-1-win too. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597336#comment-13597336
]
Chris Nauroth commented on HADOOP-8973:
---
{quote}
If java 7 File.canWrite etc are going to work on Windows (like mentioned in
some comments above) then we could enable the API path when (!Windows or
Java6).
{quote}
Sorry, let me clarify the Java 7 situation. Java 7 does not fix the bugs in
java.io.File canRead/canWrite/canExecute. Instead, it adds new methods that
are very similar in {{java.nio.file.Files}}, and these new methods are the ones
that correctly handle ACLs:
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html
I'm not sure why they didn't just fix the bugs in {{java.io.File}}, but that's
the way it is. This means that it wouldn't be correct to do if (JDK7) then use
File methods. We have to wait for Hadoop as a whole go to JDK7 (for compile
time, not just runtime) and migrate the {{DiskChecker}} code to use the new
APIs. Unfortunately, we can't do anything right now in this patch to prepare
for that.
The old Sun bug report has some relevant discussion:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6203387
{quote}
Please do open a jira to fix this issue in branch-1-win too.
{quote}
Actually, I was planning on doing this in scope of the current jira. I just
wanted to get buy-in on the trunk patch first. I expect to have the branch-1
patch ready later today.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973.3.patch,
HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597349#comment-13597349 ] Bikas Saha commented on HADOOP-8973: Makes sense DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597509#comment-13597509
]
Hadoop QA commented on HADOOP-8973:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572804/HADOOP-8973.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2299//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2299//console
This message is automatically generated.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch,
HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597619#comment-13597619 ] Arpit Agarwal commented on HADOOP-8973: --- +1 for the trunk patch. +1 for the branch-1-win patch. Thanks for adding the new tests on branch-1! I verified both patches on Windows. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597630#comment-13597630 ] Chris Nauroth commented on HADOOP-8973: --- Thanks for the important feedback everyone, and big thanks to Arpit for the final verification. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597646#comment-13597646
]
Ivan Mitic commented on HADOOP-8973:
Thanks Chris, this looks great!
Two comments:
- I noticed that testCheckDir_notListable and testCheckDir_notListable_local
fail on my machine (stacks are below, will take a look now) (hint: I am running
as admin)
- I would prefer that we make Shell#getSetPermissionCommand accept File
instead of a string (we should do the same for other Shell methods that accept
local paths, but this is orthogonal)
{code}
java.lang.AssertionError: checkDir success
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.util.TestDiskChecker._checkDirs(TestDiskChecker.java:126)
at
org.apache.hadoop.util.TestDiskChecker.testCheckDir_notListable(TestDiskChecker.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.call(FailOnTimeout.java:32)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
java.lang.AssertionError: checkDir success
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.util.TestDiskChecker._checkDirs(TestDiskChecker.java:174)
at
org.apache.hadoop.util.TestDiskChecker.testCheckDir_notListable_local(TestDiskChecker.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.call(FailOnTimeout.java:32)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch,
HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597647#comment-13597647 ] Bikas Saha commented on HADOOP-8973: +1 looks good! DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597650#comment-13597650 ] Bikas Saha commented on HADOOP-8973: bq. I would prefer that we make Shell#getSetPermissionCommand accept File instead of string +1 for this. I missed it but Ivan did not. Guess he has had to fix more cases of string-for-file issues than me :P DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597686#comment-13597686
]
Chris Nauroth commented on HADOOP-8973:
---
{quote}
I noticed that testCheckDir_notListable and testCheckDir_notListable_local fail
on my machine (stacks are below, will take a look now) (hint: I am running as
admin)
{quote}
Ivan, I would appreciate any help you can offer on this. I can't repro the
problem on my side, even when running as admin. I'm on Windows Server 2008 R2
64-bit (in case specific Windows version is causing a difference).
{quote}
I would prefer that we make Shell#getSetPermissionCommand accept File instead
of a string (we should do the same for other Shell methods that accept local
paths, but this is orthogonal)
{quote}
Please note that I am only merging an existing Shell method from trunk to
branch-1-win. If we make it accept File on branch-1-win, then we'll set
ourselves up for merge conflicts later as we try to manage patches across the 2
branches. If we change the trunk version, then it's going to break about 15
different call sites that have nothing to do with DiskChecker, so it's not
appropriate to change the trunk version within the scope of this jira. The
safest option is to defer this change to a different jira for refactoring all
of the Shell methods like you suggested. Is there a jira for that yet?
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch,
HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597704#comment-13597704 ] Arpit Agarwal commented on HADOOP-8973: --- I don't see any failures on Windows 2008 R2 64-bit either. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597799#comment-13597799
]
Ivan Mitic commented on HADOOP-8973:
OK, the issue we have is that winutils implicitly keeps all permissions for
admins/system accounts to provide the similar semantics as root on linux (this
is also needed from the scenario perspective as admins/system should be able to
access all files). This means that the test now fails when executed in the
admin context (since OS still allows admins to read/write/execute after chmod
000).
The reason why the tests succeed on your boxes is most likely because you have
User Account Control enabled on the machine (so you are not running as an
effective admin).
Would be good to check if the test passes on Unix if executed in the context of
the root?
If we go the POSIX route, we will have the expected results (chmod 000 admin
will result in 000 when we ls). I attached a prototype patch just for
demonstration. Let's think a bit more about what is the right longer term
solution. I am actually leaning towards what you have in the patch, as that is
the absolute truth about whether checkDir succeeded or failed. The downside is
that the test now fails if executed with admin context. Would be good to know
what is the unix behavior for root for symmetry.
{quote}
Please note that I am only merging an existing Shell method from trunk to
branch-1-win. If we make it accept File on branch-1-win, then we'll set
ourselves up for merge conflicts later as we try to manage patches across the 2
branches. If we change the trunk version, then it's going to break about 15
different call sites that have nothing to do with DiskChecker, so it's not
appropriate to change the trunk version within the scope of this jira. The
safest option is to defer this change to a different jira for refactoring all
of the Shell methods like you suggested. Is there a jira for that yet?
{quote}
Makes sense. Feel free to file a Jira and I can pick it up.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch,
HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597820#comment-13597820
]
Hadoop QA commented on HADOOP-8973:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572883/DiskChecker.proto.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2301//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2301//console
This message is automatically generated.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch,
HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596718#comment-13596718
]
Hadoop QA commented on HADOOP-8973:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572663/HADOOP-8973.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2295//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2295//console
This message is automatically generated.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973.3.patch,
HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594895#comment-13594895
]
Chris Nauroth commented on HADOOP-8973:
---
{quote}
So please make both versions of checkDirs consistent, ideally without code
duplication.
{quote}
Definitely agreed. Can we please keep the scope of this jira limited to
Windows compatibility for the current logic? Arpit has already filed
HADOOP-9367 for follow-up on the existing flaws in the code.
{quote}
It seems to me that what checkDir(FS, Path, Perm) is doing is sufficient for
our use cases. The code to check permissions via the implies functions are
used in a number of places in the code. What I am trying to understand is why
the check is sufficient/correct for all those cases and not here. If there is
an issue in that logic then we have issues in a number of critical places in
the code which should be fixed. Its more than a question of cross-platform
compatibility. If there is no issue that logic then its not clear why we need
to do something different for this particular case.
{quote}
There are 2 broad categories of usage for {{FsAction#implies}}:
# Inside the implementation of the HDFS permission model. This includes things
like {{FSDirectory}} and {{FSPermissionChecker}}. In this case, the logic of
{{FsAction#implies}} is sufficient, because we know that this is the only kind
of permission model supported by HDFS. It doesn't support ACLs (at least not
yet).
# In various places that need to interact with local disk. This includes
things like {{NameNode}} and {{DataNode}} initialization,
{{ClientDistributedCacheManager}} and {{FSDownload}}. In these cases, the
logic of {{FsAction#implies}} is incomplete, because it doesn't cover all
possible permission checks that might be enforced on the local disk by the host
OS, such as ACLs.
I believe this is the key distinction: HDFS (full permission model under our
control) vs. local disk (permission model partially invisible to our code).
I'm starting to think that instead of {{FsAction#implies}}, we need to write a
wrapper API that looks more like {{access}} from unistd.h, and route all local
disk interactions through that instead of {{FsAction#implies}}:
http://linux.die.net/man/2/access
This API just says tell me if I can read/write/execute this file. It
encapsulates the caller from the implementation details of the permission model.
Again though, I think this is far out of scope for the current issue.
{quote}
We need to be careful about the performance impact of how the check is done
{quote}
I believe the lists of directories to check are small in practice: things like
dfs.namenode.name.dir, dfs.datanode.data.dir, yarn.nodemanager.local-dirs.
Also, there is no performance regression on Linux, because this logic is
guarded in an if (WINDOWS) check. After migration to Java 7, even Windows
won't need to do this.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.2.patch,
HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594927#comment-13594927 ] Bikas Saha commented on HADOOP-8973: bq. Can we please keep the scope of this jira limited to Windows compatibility for the current logic? Not until we clearly understand what the flaw is that we are trying to fix. It this was branch-1 then I would agree. But on trunk we have the flexibility of not making stop-gap changes. As far as the permissions is concerned. I think Hadoop really depends on a Unix like rwx permission model on the local disk. This is in theory unrelated to the permission model HFDS itself imposes for its own filesystem and which also happens to be the rwx model. This is the main reason why we wrote the winutil layer that exposes the rwx model on top of Windows ACL's for local disk. Thats why I am trying to understand why we need different checks here because it may imply that our translation layer is not working. Your analysis might be correct and I would like an HDFS expert like [~sanjay.radia] to take a look. This jira was opened because TestDiskChecker was failing right? Would you be kind enough to change checkDir(File) to use the same logic as checkDir(FS, Path, perm) and check if the test passes? Hopefully the change will be small and wont take much of your time. It will help ascertain if the implies function will work or not. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594968#comment-13594968 ] Bikas Saha commented on HADOOP-8973: I synced offline with Arpit. I now see why the implies may be incorrect for scenarios you mention. In that case, I would really like this patch to fix the other broken checkDir also because that is incorrect too and it doesnt make sense to fix the same bug in 2 patches. This would also need to be backported to branch-1. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593713#comment-13593713 ] Arpit Agarwal commented on HADOOP-8973: --- +1 This patch works for functional parity with Linux/OSX. I don't see a reason to keep the second overload of checkDir. As long as the check is performed by the DataNode process, Chris's approach is simple and portable across all permission schemes. We should file a separate Jira to address the overload. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593760#comment-13593760 ] Bikas Saha commented on HADOOP-8973: I dont have a preference either way on how this we perform the check. What I dont want is to see 2 code paths do different things for what should essentially be the same operation. It is confusing, inconsistent and not good to maintain. So please make both versions of checkDirs consistent, ideally without code duplication. Now the question of how the check is done. It seems to me that what checkDir(FS, Path, Perm) is doing is sufficient for our use cases. The code to check permissions via the implies functions are used in a number of places in the code. What I am trying to understand is why the check is sufficient/correct for all those cases and not here. If there is an issue in that logic then we have issues in a number of critical places in the code which should be fixed. Its more than a question of cross-platform compatibility. If there is no issue that logic then its not clear why we need to do something different for this particular case. Does this make my questions clear? We need to be careful about the performance impact of how the check is done. I see a number of places that call checkDirs() method for a collection of directories. Do we want to write to a temp file for all those cases in order to check for writability? DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594045#comment-13594045 ] Arpit Agarwal commented on HADOOP-8973: --- Filed HADOOP-9367. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592734#comment-13592734
]
Chris Nauroth commented on HADOOP-8973:
---
Here was the last discussion on this issue:
{quote}
Can we try the following approach that has the advantage of being common across
platforms.
Get FileStatus of the file/dir using FileSystem API
getPermission() from FileStatus
There are a bunch of impliesRead(), impliesWrite() functions that tell whether
a given FsPermission object allows read/write etc. I am sorry I don't remember
where these functions are.
Using these functions one can get the equivalent of isReadable/isWritable
{quote}
Digging into this, unfortunately no, we cannot reconstruct the logic of
{{DiskChecker}} in a cross-platform way by using {{FsPermission}} and
{{FsAction}} (with delegation to winutils ls when running on Windows). The
reason that methods like {{File#canRead}} are used here is that it checks if
the current process can read the file, with full consideration of the specific
user that launched the process. Our {{FsPermission}} model considers
permissions, but it does not consider how those permissions map to specific
users (nor would it be trivial to expand it to include this information).
For example, using {{FsPermission}} + winutils ls, we could see that a
particular directory has permissions 770, so the owner and members of the
primary group have full access to this directory. If the user that launched
the process is either the owner or a member of the primary group, then
{{DiskChecker}} must report that the disk is usable. If not, then
{{DiskChecker}} must report that the disk is unusable. However,
{{DiskChecker}} doesn't have enough information to choose the correct result.
It would be error-prone to go down the path of trying to duplicate the exact
permission checking logic enforced by the OS on the local file system. You
might argue that we could solve the example above by trying to combine more
calls to get the primary group and check if the current user's groups include
that group. However, that wouldn't be sufficient to cover more complex rules
enforced by local file systems, such as POSIX ACLs. It would be challenging to
get this exactly right, and since the logic is different on different OSes, it
wouldn't satisfy the goal of trying to keep this layer of code
platform-agnostic anyway.
I propose that we accept the patch I wrote earlier, still attached to the jira,
so that {{DiskChecker}} actually attempts a few file system operations for
verification when running on Windows. These work-arounds could be removed
whenever Hadoop upgrades to JDK7. Then, we can rewrite {{DiskChecker}} to use
the new Java {{FileSystem}} API, which doesn't have these bugs. I've confirmed
that the existing patch still applies cleanly to branch-trunk-win and fixes
{{TestDiskChecker}} so that it passes on both Mac and Windows.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592795#comment-13592795 ] Bikas Saha commented on HADOOP-8973: During Datanode startup it checks if the data dir permissions are 755. Using the above logic, is there a hole in that check too because it does not imply that Datanode has permission to read and write to that directory? DiskChecker is used to perform that check too using another checkDir() method. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592830#comment-13592830
]
Chris Nauroth commented on HADOOP-8973:
---
That's interesting. Yes, I believe this is a bug in the existing code for the
other overload of {{DiskChecker#checkDir}}.
For example, suppose a dfs.datanode.data.dir on the local file system with
owner foo and perms set to 700. Now suppose we launch datanode as user
bar. {{DiskChecker#checkDir}} will just look for 700 and not consider the
running user, so it will think that the directory is usable. Then, it would
experience an I/O error later whenever the process first tries to use that
directory.
I'll file a separate jira for this.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592877#comment-13592877 ] Bikas Saha commented on HADOOP-8973: Well. In that case, it would be a standard pattern everywhere because everywhere code simply checks the value of the permissions and not whether the process checking that value actually has the right membership wrt that value. Isnt it so? Irrespective of OS. Also, one can make a case that checkDir(File dir) should end up calling checkDir(FileSystem, FilePath, rwx) instead of duplicating the logic. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592904#comment-13592904
]
Arpit Agarwal commented on HADOOP-8973:
---
+1 on the approach for its simplicity. It is sound to check for permissions in
this way and we should do the same on other platforms.
A few comments on the code.
Are there any bugs other than the one you link below? I wonder if dir.canRead
can return false when it should return true. We should skip this call on
Windows if it is known to be broken. Same for dir.canWrite and dir.canExecute.
{code}
// This method contains several workarounds to known JVM bugs that cause
// File.canRead, File.canWrite, and File.canExecute to return incorrect
// results on Windows with NTFS ACLs.
// http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6203387
{code}
Does this change the current directory of the calling process?
{code}String[] cdCmd = new String[] { cmd, /C, cd,
dir.getAbsolutePath() };
{code}
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592918#comment-13592918 ] Ivan Mitic commented on HADOOP-8973: Thanks Chris for the patch. I am fine with the current patch, but would still like to reiterate on Bikas' question on why we cannot use impliesRead/impliesWrite etc. At this point, we already assume the POSIX like semantic in Hadoop across the board (this is why we have winutils). Leveriging winutils in CheckDisk would provide a nice symmetry in the test. You bring up a good point about not completely covering the NTFS ACLs space. I think this is fine for now. Eventually, as we improve, we can think of good ways to abstract this out, but this goes way beyond today. Let me know what you think. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592931#comment-13592931
]
Chuan Liu commented on HADOOP-8973:
---
I want to chime in a little on this.
The reason that methods like File#canRead are used here is that it checks if
the current process can read the file, with full consideration of the
specific user that launched the process.
Chris, I think it should be relatively easy to provide some API like this
either through winutils or JNI. In Windows (and the new POSIX ACL mode), file
owner and file ACL are separate concepts (comparing with traditional chmod file
permission mode where the first 3 bit are tightly associated with the file
owner). In winutils ls, we retrieval effective rights for the owner through
the API call GetEffectiveRightsForSid() where Sid is the user sid that we want
to check access permissions. Calling this API with a given user SID, we should
be able to get its permission on the file.
I notice we already have the getfacl command line utility on Linux. What do
you think we provide a simple one in winutils? E.g.
{code}
winutils getfacl -u [username] file
user:[username]:r-x
{code}
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592988#comment-13592988 ] Bikas Saha commented on HADOOP-8973: Thats is a good idea to do if we need to find out if the current process user has certain permissions. But I guess the point we are debating on currently is whether we can make checkDisk(File) do what checkDisk(FS, Path, Perm) does. Perhaps by simply calling the second function. If checkDisk(FS, Path, Perm) meets our other needs then it should be enough. I think currently, the code checks for expected permissions, implicitly assuming the daemon processes to be running as the users who are supposed to have those permissions. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[
https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593055#comment-13593055
]
Chris Nauroth commented on HADOOP-8973:
---
Thanks for the comments, everyone. This is very helpful.
{quote}
Does this change the current directory of the calling process?
{quote}
No, this forks a whole new process, and runs the cd within that process. The
working directory of the calling process is unchanged. I think this is safe.
{quote}
Leveriging winutils in CheckDisk would provide a nice symmetry in the test.
{quote}
{quote}
Chris, I think it should be relatively easy to provide some API like this
either through winutils or JNI.
{quote}
On all of the other discussion points, I think the summary is that we have
discovered that there are deficiencies in the current logic of {{DiskChecker}},
and it's not a problem specific to Windows. It's a problem on Linux too.
Considering this, I'd still like to proceed with the basic approach in the
current patch. We can file a follow-up jira to fix the problem more
completely, with full consideration for other permission models that include
things like POSIX ACLs and NTFS ACLs. (My opinion is that we should just wait
for JDK7 instead of investing in JNI calls, but that's just my opinon.) The
scope of this follow-up jira would include both Linux and Windows.
Arpit had provided some feedback on the actual code, and I do want to provide a
new patch to address that feedback. I'm planning on uploading a new patch
tomorrow. If anyone disagrees with the approach though, please let me know so
that I don't waste time preparing a patch that is objectionable. :-)
{quote}
Well. In that case, it would be a standard pattern everywhere because
everywhere code simply checks the value of the permissions and not whether the
process checking that value actually has the right membership wrt that value.
Isnt it so? Irrespective of OS.
{quote}
I'm reluctant to change the code so that the permission checks are less
comprehensive on Linux for the sake of cross-platform consistency. Right now,
we have one overload of {{DiskChecker#checkDir}} that is correct AFAIK, and
another overload of {{DiskChecker#checkDir}} that is incomplete when
considering more sophisticated permission models on the local file system, like
POSIX ACLs. The approach in the current patch at least achieves consistent
behavior between Linux and Windows, so at least we have symmetry with regards
to that.
DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS
ACLs
-
Key: HADOOP-8973
URL: https://issues.apache.org/jira/browse/HADOOP-8973
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8973-branch-trunk-win.patch
DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to
check if a directory is inaccessible. These APIs are not reliable on Windows
with NTFS ACLs due to a known JVM bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581631#comment-13581631 ] Arpit Agarwal commented on HADOOP-8973: --- [~cnauroth], is this waiting for a code review? DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581638#comment-13581638 ] Chris Nauroth commented on HADOOP-8973: --- Hi, Arpit. There had been an earlier round of feedback on this patch that suggested trying a different approach. I haven't had a chance to investigate it and try it. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487796#comment-13487796 ] Bikas Saha commented on HADOOP-8973: Can we try the following approach that has the advantage of being common across platforms. Get FileStatus of the file/dir using FileSystem API getPermission() from FileStatus There are a bunch of impliesRead(), impliesWrite() functions that tell whether a given FsPermission object allows read/write etc. I am sorry I don't remember where these functions are. Using these functions one can get the equivalent of isReadable/isWritable DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487999#comment-13487999 ] Chris Nauroth commented on HADOOP-8973: --- I see FsAction.implies(FsAction), which for example can be used to check that rw- implies r-- but does not imply --x. Is this what you had in mind? Is this structure sufficient to cover NTFS ACLs? It seems like this would cover checks of UNIX-style permissions + file ownership, but then an NTFS ACL could still disallow. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488471#comment-13488471 ] Bikas Saha commented on HADOOP-8973: On branch-1-win internally winutils would be used to infer the Windows ACL's into rwx permissions. If those changes have been ported over then it should work on Windows too. Other places use similar logic which can be checked by looking for references to the FsAction.implies() method. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483589#comment-13483589 ] Chris Nauroth commented on HADOOP-8973: --- Here is the JVM bug report: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6203387 This was closed as Won't Fix with suggestions to code work-arounds that involve actually attempting file system operations or using JNI to call a Windows API. The bug report also says that the new file APIs in JDK7 will address this. The problem is visible when running TestDiskChecker, which fails on Windows. I suspect no one noticed the problem on branch-1-win, because that version of the code does not have a test suite for DiskChecker. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483599#comment-13483599 ] Chris Nauroth commented on HADOOP-8973: --- One other note on that patch: I also noticed a bug in TestDiskChecker.java. The _checkDirs helper method was not using the isDir flag. This meant that the test case testCheckDir_notDir_local was not really testing what it intended. I updated _checkDirs to make use of the flag. If isDir is true, then the test runs against a directory. If isDir is false, then the test runs against a file. All of these tests pass on Mac and Windows with this patch. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
