[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692918#comment-13692918 ] Hudson commented on HADOOP-9439: Integrated in Hadoop-Yarn-trunk #251 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/251/]) HADOOP-9439. JniBasedUnixGroupsMapping: fix some crash bugs (Colin Patrick McCabe) (Revision 1496112) Result = FAILURE cmccabe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1496112 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocket.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMappingWin.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/getGroup.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.h JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13693008#comment-13693008 ] Hudson commented on HADOOP-9439: Integrated in Hadoop-Hdfs-trunk #1441 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1441/]) HADOOP-9439. JniBasedUnixGroupsMapping: fix some crash bugs (Colin Patrick McCabe) (Revision 1496112) Result = FAILURE cmccabe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1496112 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocket.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMappingWin.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/getGroup.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.h JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13693039#comment-13693039 ] Hudson commented on HADOOP-9439: Integrated in Hadoop-Mapreduce-trunk #1468 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1468/]) HADOOP-9439. JniBasedUnixGroupsMapping: fix some crash bugs (Colin Patrick McCabe) (Revision 1496112) Result = FAILURE cmccabe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1496112 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocket.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMappingWin.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/getGroup.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.h JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692095#comment-13692095 ] Todd Lipcon commented on HADOOP-9439: - +1, latest rev of the patch lgtm. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692122#comment-13692122 ] Hudson commented on HADOOP-9439: Integrated in Hadoop-trunk-Commit #4004 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4004/]) HADOOP-9439. JniBasedUnixGroupsMapping: fix some crash bugs (Colin Patrick McCabe) (Revision 1496112) Result = SUCCESS cmccabe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1496112 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/exception.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/unix/DomainSocket.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMapping.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsMappingWin.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/getGroup.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_group_info.h * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.h JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688318#comment-13688318
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588656/HADOOP-9439.007.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestSaslRPC
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2676//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2676//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688357#comment-13688357
]
Chris Nauroth commented on HADOOP-9439:
---
{quote}
Do we not allow C99 style declaration in the middle of a function in our JNI
code? I've always liked that better than the original C style of declaring all
at the top.
{quote}
One more thing about this: I think Visual Studio still does not support C99.
In the Windows native code, we're declaring all variables at the top of the
function, and it's a compilation error to put declarations in the middle. With
conditional compilation, we could potentially do C99 in the Linux path and C89
in the Windows path, but this might cause confusion.
This isn't an issue for this patch, but I thought I'd mention it.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch,
HADOOP-9439.008.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688385#comment-13688385
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588669/HADOOP-9439.008.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestSaslRPC
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2677//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2677//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch,
HADOOP-9439.008.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688421#comment-13688421
]
Chris Nauroth commented on HADOOP-9439:
---
The {{TestSaslRPC}} failure appears to be unrelated. [~atm], do you think it's
related to this? http://svn.apache.org/viewvc?view=revisionrevision=1494702
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch,
HADOOP-9439.008.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688486#comment-13688486 ] Colin Patrick McCabe commented on HADOOP-9439: -- Thanks, Chris. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688488#comment-13688488 ] Aaron T. Myers commented on HADOOP-9439: Yea, it's unrelated. I'm fixing it. Thanks. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, HADOOP-9439.008.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687295#comment-13687295
]
Todd Lipcon commented on HADOOP-9439:
-
- seems like {{pw_lock_object}} is declared in two .c files... one should
probably be marked extern
- The following code is smelly to me:
{code}
int i, ret, nvalid, pw_lock_locked = 0;
{code}
because only the last variable is actually initialized. I don't _think_ you end
up using any of the uninitialized variables, so it's not a true bug, but I
think it's worth separating them out into their own lines. Do we not allow C99
style declaration in the middle of a function in our JNI code? I've always
liked that better than the original C style of declaring all at the top.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687314#comment-13687314 ] Colin Patrick McCabe commented on HADOOP-9439: -- I don't like C99 style declarations because they tend to introduce bugs. When you have some cleanup label at the end of the function, if your goto passes over the variable declaration, the variable will be treated as undefined. So, for example, if you have: int i = 4; JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687317#comment-13687317
]
Colin Patrick McCabe commented on HADOOP-9439:
--
ok, these keyboard shortcuts defeated me again. I wish I could turn that off.
Anyway, if you have:
{code}
goto cleanup:
struct foo *foo= calloc(1, sizeof(*foo));
...
cleanup:
free(foo);
{code}
then you're freeing an uninitialized pointer. Not good.
I also don't like declarations in the middle because of a philosophical reason:
it tends to lead to excessively long functions, rather than encouraging
modularity.
As to the extern thing, yeah, I agree. It's rather odd that the linker does
that kind of symbol merging, but apparently it does.
I don't see the point of putting all variable declarations on separate lines.
Java allows declaration of variables on the same line as well. C++0x allows
things like vector to be initialized with a reasonable syntax, so you are
finally allowed to admit that this is a good thing, you C++ advocate, you. :)
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687398#comment-13687398
]
Todd Lipcon commented on HADOOP-9439:
-
bq. I don't see the point of putting all variable declarations on separate
lines. Java allows declaration of variables on the same line as well
The point is that when a lot of people see {{int a, b, c = 0;}} they think that
all three variables are getting set to 0, but in fact it's only {{c}} that is
set. Putting on separate lines makes it more obvious -- probably not important
if you expect mostly veteran C programmers to read the code, but given that a
lot of folks in the Hadoop community rarely look at C, I figured it would be
more obvious this way.
In Java, you always get a compile time error for accessing an uninitialized
variable, whereas in C it's just a warning, which I'm certain we'll miss in the
compile spew of a Hadoop build. So, probably better to explicitly initialize
all the variables, whether on one line or multiple.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687524#comment-13687524
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588505/HADOOP-9439.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2671//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678312#comment-13678312 ] Colin Patrick McCabe commented on HADOOP-9439: -- bq. Invalid groups are a sore point for ShellBasedUnixGroupsMapping. If any invalid groups are associated with a user, the groups program will fail with a non-zero return code, and no information is returned Actually, I just tested the behavior of ShellBasedUnixGroupsMapping when a group can't be resolved. Apparently it just creates an empty array entry for that group, but returns all the others (at least on Linux.) I think the reason is because although the groups command has the behavior I described, we are now using the id -Gn command, which ignores illegal groups. I also verified that this patch works in the case where a group can't be resolved. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, HADOOP-9439.005.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678397#comment-13678397
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586769/HADOOP-9439.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2621//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2621//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HADOOP-9439.005.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677450#comment-13677450
]
Todd Lipcon commented on HADOOP-9439:
-
It looks like you've inverted the meaning of the boolean in some places here:
{code}
+ * @param reentrantTrue if we should use the reentrant versions of
+ * getgrent, getpwent, etc. They are faster, but
+ * buggy in some implementations.
+ *
+ * @return The set of groups associated with a user.
+ */
+ native static String[] getGroupsForUser(String username, boolean reentrant);
{code}
suggests that the second parameter being 'false' disables locking. But then:
{code}
public ListString getGroups(String user) throws IOException {
String[] groups = new String[0];
try {
- groups = getGroupForUser(user);
+ groups = getGroupsForUser(user, removeConcurrency);
{code}
passes 'true' if you want to disable locking.
The native implementation seems to agree with the latter (i.e that passing true
will introduce the locking)
{code}
+ static private void logError(int groupIdx, String error) {
+LOG.error(error looking up the name of group + groupIdx + : + error);
{code}
This parameter should be 'groupId' not 'groupIdx'
{code}
+ g_log_error_method = (*env)-GetStaticMethodID(env, clazz, logError,
+(ILjava/lang/String;)V);
+ if (!g_log_error_method) {
+jthrowable jthr = newRuntimeException(env,
+JniBasedUnixGroupsMapping#anchorNative: failed to look
+up JniBasedUnixGroupsMapping#logError method\n);
+(*env)-Throw(env, jthr);
{code}
No need to throw an exception here - GetStaticMethodID already throws
NoSuchMethodError if it fails. Same with the {{FindClass}} call below, and I
assume NewGlobalRef as well (at least I've never seen this pattern of checking
the result of NewGlobalRef).
{code}
+ error_msg = (*env)-NewStringUTF(env, terror(ret));
+ if (!error_msg) {
+(*env)-ExceptionClear(env);
+return;
+ }
{code}
Why are you ignoring exceptions in this method? Add a comment explaining this.
- In general, you don't need to {{DeleteLocalRef}} inside short-lived JNI
methods. It just adds clutter to the code -- they're automatically deleted at
the end of the method. It's only important if you plan on doing a lot of
allocation/freeing inside the method and need to let GC collect the stuff
before the method returns.
- Can you explain how you tested the various error code paths here in the JNI?
eg did you create a user which has some invalid groups? I'm nervous about
missing some bug until we hit it in production.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677534#comment-13677534
]
Colin Patrick McCabe commented on HADOOP-9439:
--
Thanks, very thorough review.
I fixed the naming of the parameters to getGroupsForUser and logError.
Although it didn't affect correctness, it certainly was very confusing.
You're right that {{GetStaticMethodID}} and {{FindClass}} throw exceptions on
failure. There is no need to throw another one (and it is probably actually
harmful). Thanks for finding that. However, {{NewGlobalRef}} does not throw
an exception, but merely returns {{NULL}} when you're out of memory.
I will add a comment clarifying why we ignore exceptions in logError.
I agree that we should probably just use the existing monitor lock from
NativeIO.java. That way, I don't have to modify that code. I wasn't aware
that there were other folks poking the user/group functions in Hadoop. Right
now, it looks like {{NativeIO#getUserName}} is only called from tests calling
{{NativeIO#POSIX#getFstat}}, but of course that may change in the future.
Invalid groups are a sore point for {{ShellBasedUnixGroupsMapping}}. If any
invalid groups are associated with a user, the {{groups}} program will fail
with a non-zero return code, and no information is returned. For
{{JniBasedUnixGroupsMapping}}, I would prefer to return the groups that were
valid, rather than nothing at all. I suppose this is debatable, though. I can
test creating such invalid groups.
I understand that sometimes it's unnecessary, but I'd rather have
{{DeleteLocalRef}} used for all allocations. For one thing, in libhdfs, it
really *is* necessary everywhere (the JNI invocation API never automatically
disposes of Java references that are made in the invoking C code). It confuses
people when they copy a piece of code from one part of the source tree to
another and it suddenly becomes incorrect. For another thing, the spec only
says that the JVM has to provide at least 16 local references at once, which is
not very many at all. It's only one line of overhead per Java reference, and
{{DeleteLocalRef}} already ignores NULLs, so I'd rather just have a consistent
style everywhere, than try to be clever.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658686#comment-13658686
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583352/HADOOP-9439.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2544//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2544//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657348#comment-13657348
]
Todd Lipcon commented on HADOOP-9439:
-
{code}
+ static String empty[] = new String[0];
{code}
Should probably be static final, right? And rename to {{NO_GROUPS}} or
{{EMPTY_STIRNG_ARRAY}} or something a little more descriptive
{code}
+ native static String[] getGroupsForUser(String username, String empty[],
+boolean reentrant);
{code}
I don't get why {{empty}} is a parameter here... aside from it being a
premature optimization (is allocating an empty array that bad?), it seems like
the JNI code could just grab this field itself and return a reference, rather
than taking it as a parameter.
{code}
+ groups = getGroupsForUser(user, empty, useReentrant );
{code}
Style nit: extra space here before ')'
{code}
+ snprintf(buf, sizeof(buf), getgrouplist error %d, ret);
+ THROW(env, java/lang/RuntimeException, buf);
{code}
Can we use strerror here on the return, assuming it's probably a standard
errnum?
{code}
+ret = hadoop_group_info_fetch(ginfo, uinfo-gids[i]);
+if (!ret) {
{code}
Here if the group info lookup fails for a particular gid, you end up swallowing
the error without any warnings, etc. Maybe instead we should just return the
numeric gid as a group? This is what the 'groups' shell command does:
{code}
todd@todd-w510:~$ groups todd
todd : groups: cannot find name for group ID 1050
1050 adm dialout cdrom audio video plugdev lpadmin admin sambashare mrtest
wireshark
{code}
- I noticed you use different locks for getgrnam and getpwnam. But when I was
working on HADOOP-7156, I found that the buggy implementations were racey in
other parts of their code -- ie a concurrent getgrnam and a concurrent getpwnam
might race with eachother and cause either to fail. I know that you added this
function to deal with a crash we saw on a production cluster - did you verify
that with the same buggy underlying implementation that the workaround prevents
the issue?
We may have to actually share this lock and configuration with the lock used in
NativeIO.c.
Additionally, when I did that patch, I found it simpler to continue to use the
reentrant functions wrapped with a lock -- less repeated code, etc.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615542#comment-13615542 ] Colin Patrick McCabe commented on HADOOP-9439: -- moved this to common at ATM's suggestion. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615592#comment-13615592
]
Chris Nauroth commented on HADOOP-9439:
---
Hi, Colin. I had filed HADOOP-9312 for a potential memory leak around the lazy
initialization of {{emptyGroups}}. From my quick scan of the patch here, it
looks like this will fix it (at least for the non-Windows codebase). Do you
agree? If so, I will relate the 2 jiras, or perhaps file a new jira for
porting this patch to the Windows version. Thanks!
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615601#comment-13615601
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575606/HDFS-4640.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2371//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2371//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13615948#comment-13615948 ] Colin Patrick McCabe commented on HADOOP-9439: -- yeah, this will definitely fix the memory leak identified in HADOOP-9312. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
