[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993871#comment-12993871 ] Hudson commented on HADOOP-6978: Integrated in Hadoop-Common-22-branch #24 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/24/]) HADOOP-6978. svn merge -c 1070021 from trunk > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: fstat.patch, hadoop-6978.txt, hadoop-6978.txt, > hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966491#action_12966491 ] Hudson commented on HADOOP-6978: Integrated in Hadoop-Common-trunk #534 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/534/]) > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: fstat.patch, hadoop-6978.txt, hadoop-6978.txt, > hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965615#action_12965615 ] Hudson commented on HADOOP-6978: Integrated in Hadoop-Common-trunk-Commit #450 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/450/]) HADOOP-6978. Adds support for NativeIO using JNI. Contributed by Todd Lipcon, Devaraj Das & Owen O'Malley. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: fstat.patch, hadoop-6978.txt, hadoop-6978.txt, > hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935539#action_12935539 ] Hadoop QA commented on HADOOP-6978: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12460406/hadoop-6978.txt against trunk revision 1038493. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//console This message is automatically generated. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: fstat.patch, hadoop-6978.txt, hadoop-6978.txt, > hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935518#action_12935518 ] Hadoop QA commented on HADOOP-6978: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12460400/fstat.patch against trunk revision 1038493. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/121//console This message is automatically generated. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: fstat.patch, hadoop-6978.txt, hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932216#action_12932216 ] Rajiv Chittajallu commented on HADOOP-6978: --- bq. That seems fairly reasonable (adding a cache here). Systems generally have nscd or an equivalent to handle nss cache. I think we should leave that uid/gid caching to underlying system. Most of the getpw calls are for uid to username lookups since hadoop deals only with username. Wouldn't it be simple to pass the uid along with the username when the JT hands off the task to the tt? > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt, hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931751#action_12931751 ] Devaraj Das commented on HADOOP-6978: - Yeah, our belief is that the shuffle process ends up making a lot of the getpw* calls and we have already seen a couple of ldap servers outages. We can do a follow up patch though. If the cluster has a configuration similar to what i mentioned earlier, then yeah, it'd be really good to have this cache before deployment... > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt, hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931750#action_12931750 ] Todd Lipcon commented on HADOOP-6978: - Hey Devaraj, That seems fairly reasonable (adding a cache here). Since this is a blocker security bug for 0.22, though, maybe we should get this in and then add the cache as a follow-on? If you think this is un-deployable without the cache we may as well do it now. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt, hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[
https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931690#action_12931690
]
Devaraj Das commented on HADOOP-6978:
-
We have noticed that sometimes the C calls like getpwuid_r ends up making
direct calls to the ldap server. It probably is configuration/environment
specific, but in Yahoo! the password entries are maintained by the ldap server.
In order to prevent ldap servers from getting overloaded with password
look-ups, we have a daemon called nscd run on all the compute nodes, that
caches the results of such look-ups. The calls such as getpwuid_r should
terminate at the local nscd daemon, but if, for whatever reason, the nscd
daemon is down on the node, the calls end up talking to the ldap server
directly. Apparently, nscd is not that stable...
We have seen the above happening at Yahoo! and in a couple of occasions brought
down the ldap servers. So I was wondering whether we should reduce the number
of calls to the getpwuid_r and such by caching the resolutions
{uid,gid}->{username,groupname} in Hadoop.. Thoughts?
> Add JNI support for secure IO operations
>
>
> Key: HADOOP-6978
> URL: https://issues.apache.org/jira/browse/HADOOP-6978
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io, native, security
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.22.0
>
> Attachments: hadoop-6978.txt, hadoop-6978.txt
>
>
> In support of MAPREDUCE-2096, we need to add some JNI functionality. In
> particular, we need the ability to use fstat() on an open file stream, and to
> use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929906#action_12929906 ] Hadoop QA commented on HADOOP-6978: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459130/hadoop-6978.txt against trunk revision 1032730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//console This message is automatically generated. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt, hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929894#action_12929894 ] Hadoop QA commented on HADOOP-6978: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459125/hadoop-6978.txt against trunk revision 1032730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//console This message is automatically generated. > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
[ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929881#action_12929881 ] Todd Lipcon commented on HADOOP-6978: - Patch up on review board at: https://reviews.apache.org/r/52/ > Add JNI support for secure IO operations > > > Key: HADOOP-6978 > URL: https://issues.apache.org/jira/browse/HADOOP-6978 > Project: Hadoop Common > Issue Type: New Feature > Components: io, native, security >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.22.0 > > Attachments: hadoop-6978.txt > > > In support of MAPREDUCE-2096, we need to add some JNI functionality. In > particular, we need the ability to use fstat() on an open file stream, and to > use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
