[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13430336#comment-13430336
]
Hudson commented on HDFS-3579:
--
Integrated in Hadoop-Hdfs-trunk #1128 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/])
Add two new files missed by last commit of HDFS-3579. (Revision 1370017)
HDFS-3579. libhdfs: fix exception handling. Contributed by Colin Patrick
McCabe. (Revision 1370015)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370017
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.h
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370015
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.2.0-alpha
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429264#comment-13429264
]
Aaron T. Myers commented on HDFS-3579:
--
I've taken a look at the patch and it looks good to me. I agree that this is
some good cleanup to do. Thanks a lot, Andy, for your very thorough review.
One question before I commit this patch, though: can you please describe what
sort of testing you did to verify this change? Are the warnings about pending
exceptions now gone from the logs? Were you able to ensure that memory is no
longer leaked when exceptions are thrown?
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429408#comment-13429408
]
Colin Patrick McCabe commented on HDFS-3579:
bq. Are the warnings about pending exceptions now gone from the logs?
Yes. Running a before and after with LIBHDFS_OPTS=-Xcheck:jni
-Xcheck:nabounds confirms that the messages about JNI call made with
exception pending are gone after the patch. The test I ran was
test_libhdfs_threaded.
I also ran test_fuse_dfs and verified that it passed successfully. That test
also exercises libhdfs, albeit in a slightly different way.
We need a longer running test that exercises more failure conditions to fully
establish that all memory leaks are fixed. I think writing such a test is a
little bit of out scope for this JIRA, but it's definitely something we should
do in the future.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429427#comment-13429427
]
Colin Patrick McCabe commented on HDFS-3579:
Thanks, atm. I have tried running valgrind on fuse_dfs in the past. It
doesn't really work-- I get tons of false positives. I think there's a general
problem running valgrind with JNI code. I did try adding more and more stuff
to the exclude lists, but it didn't seem to work. Maybe someone more
knowledgeable on this topic can come up with a workaround.
I'm also confused about whether valgrind can identify memory leaks of memory
managed by the JVM. I suspect that the answer is no, which would mean that
the local reference leaks fixed by the patch would have been invisible to
valgrind anyway. As far as I know, valgrind only deals with memory allocated
via {{malloc}}-- although I'm happy to be corrected on this topic if someone
has more info ( ? )
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429430#comment-13429430
]
Aaron T. Myers commented on HDFS-3579:
--
bq. I have tried running valgrind on fuse_dfs in the past. It doesn't really
work-- snip
Got it, thanks for the explanation.
I've just committed this to trunk and branch-2. Thanks a lot for the
contribution, Colin. Fixes like this are yeoman's work. Thanks for doing it.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.2.0-alpha
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429443#comment-13429443
]
Hudson commented on HDFS-3579:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2621 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2621/])
Add two new files missed by last commit of HDFS-3579. (Revision 1370017)
HDFS-3579. libhdfs: fix exception handling. Contributed by Colin Patrick
McCabe. (Revision 1370015)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370017
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.h
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370015
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.2.0-alpha
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429445#comment-13429445
]
Hudson commented on HDFS-3579:
--
Integrated in Hadoop-Common-trunk-Commit #2556 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2556/])
Add two new files missed by last commit of HDFS-3579. (Revision 1370017)
HDFS-3579. libhdfs: fix exception handling. Contributed by Colin Patrick
McCabe. (Revision 1370015)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370017
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.h
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370015
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.2.0-alpha
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429481#comment-13429481
]
Hudson commented on HDFS-3579:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2575 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2575/])
Add two new files missed by last commit of HDFS-3579. (Revision 1370017)
HDFS-3579. libhdfs: fix exception handling. Contributed by Colin Patrick
McCabe. (Revision 1370015)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370017
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.h
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1370015
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.2.0-alpha
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426805#comment-13426805
]
Andy Isaacson commented on HDFS-3579:
-
bq. There's no way to actually get an hdfsFile which has file-type
UNINITIALIZED.
That's not what I was asking for. I want us to deal with {{file == NULL}}, it
is a simple user error, no need to SEGV on it.
If {{file-file == NULL}} or {{file-type == UNINITIALIZED}} the data
structures are corrupt and we can SEGV all we want.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426932#comment-13426932
]
Colin Patrick McCabe commented on HDFS-3579:
bq. I want us to deal with file == NULL
The latest patch has this in hdfsClose:
{code}
//Sanity check
if (!file || file-type == UNINITIALIZED) {
errno = EBADF;
return -1;
}
{code}
So it should set errno = EBADF and return -1, just as before.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426166#comment-13426166
]
Andy Isaacson commented on HDFS-3579:
-
{code}
...
+jvalue jVal;
+
+jthrowable jthr = classNameOfObject(exc, env, className);
+if (jthr) {
{code}
the blank line should be after the declaration of jthr.
{code}
+fprintf(stderr, PrintExceptionAndFree: error determining class name
+of exception!);
{code}
add a \n to this printf. If it were me I would replace all the surprised !s
with . but I am not going to insist on unexcitifying the error messages.
{code}
+fprintf(stderr, error: (no exception));
{code}
another missing \n.
{code}
-constructNewObjectOfClass(env, NULL, org/apache/hadoop/fs/Path,
+jthr = constructNewObjectOfClass(env, jPath, org/apache/hadoop/fs/Path,
(Ljava/lang/String;)V, jPathString);
{code}
indent the continuation line to match the (.
{code}
+jthr = newCStr(env, jRet, val);
+if (jthr)
goto done;
done:
{code}
having a goto done on the line before done: is a bit confusing. I'd just
drop it.
{code}
--- hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
+++ hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.h
...
+void destroyLocalReference(JNIEnv *env, jobject jObject);
{code}
Since this is external linkage C API, I think we need to use a reserved
namespace prefix to avoid breaking outside code. So, make this
hdfsDestroyLocalReference perhaps? This is a large change, and this is newly
exposed code, so that change can be done in a separate jira.
An alternate fix, if we only support a .so and don't ship a .a, is to use a
linker script to control symbol visibility.
But one way or another we need to make sure we have sanitary symbol table usage.
{code}
+if (jthr) {
+ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
+hdfsDisconnect: org.apache.hadoop.fs.FileSystem::close);
{code}
I don't think :: is right for Java? I think it should be
org.apache.hadoop.fs.FileSystem#close.
Please check all the printExceptionAndFree calls and verify that they are
consistent, I see some with and some without the org.class.path. I don't care
what standard, but I think cFuncName: org.class.path#method is the right
string unless there's a better idea.
{code}
//bufferSize
if (!bufferSize) {
{code}
Please delete this useless comment. (Not your code, but you're editing right
here, let's just fix it.)
{code}
} else if ((flags O_WRONLY) (flags O_APPEND)) {
+// WRITE/APPEND?
+ jthr = invokeMethod(env, jVal, INSTANCE, jFS, HADOOP_FS,
{code}
There's a bunch of funky whitespace here, let's fix it. } else has an extra
space, and I think the // WRITE is indented one space too far.
{code}
+file = calloc(1, sizeof(struct hdfsFile_internal));
{code}
I find {{file = calloc(1, sizeof *file);}} easier to convince myself of, but I
don't know if we have a local style guideline on this point?
{code}
+if ((flags O_WRONLY) == 0) {
{code}
This is wrong per HDFS-3710. (Many occurrences of this pattern.) No need to
fix the existing ones outside of the code you're touching, but please don't add
more.
more to come in next comment...
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426214#comment-13426214
]
Colin Patrick McCabe commented on HDFS-3579:
Thanks for the review, Andy. I'm looking forward to the next comment.
bq. [O_WRONLY discussion]
I held off on the O_WRONLY fixes, since we have HDFS-3710 open for that. What
I've done here is just move the code, not add any new (mis)uses.
bq. [linker script discussion]
Yeah, HDFS-3742 is open for this.
I tried to fix as much whitespace as I could; I'm sure there's still funky
stuff lingering somewhere. We can always circle back on that later, though--
as long as this patch moves things in the right direction.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426253#comment-13426253
]
Hadoop QA commented on HDFS-3579:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538632/HDFS-3579.006.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2932//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2932//console
This message is automatically generated.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426262#comment-13426262
]
Andy Isaacson commented on HDFS-3579:
-
{code}
@@ -870,15 +812,12 @@ int hdfsCloseFile(hdfsFS fs, hdfsFile file)
-//Parameters
-jobject jStream = (jobject)(file ? file-file : NULL);
...
+(*env)-DeleteGlobalRef(env, file-file);
free(file);
-(*env)-DeleteGlobalRef(env, jStream);
{code}
Let's preserve the interface that {{hdfsFile file}} can be NULL without causing
SEGV. Just toss in a {{if (file == NULL) return -1;}} near the top.
{code}
+jthr = invokeMethod(env, jVal, INSTANCE, jInputStream, HADOOP_ISTRM,
read, ([B)I, jbRarray);
...
+if (jVal.i 0) {
+// EOF
+destroyLocalReference(env, jbRarray);
+return 0;
+} else if (jVal.i == 0) {
+destroyLocalReference(env, jbRarray);
+errno = EINTR;
+return -1;
{code}
Is this correct? FSDataInputStream#read returns -1 on EOF and 0 on EINTR?
That's special. I see docs for the -1 case, but I don't see anywhere that the
0 could come from?
{code}
tSize hdfsPread(hdfsFS fs, hdfsFile f, tOffset position,
void* buffer, tSize length)
{
-// JAVA EQUIVALENT:
-// byte [] bR = new byte[length];
-// fis.read(pos, bR, 0, length);
{code}
I find these JAVA EQUIVALENT comments to be very helpful, could we keep them
around? so long as they're accurate, I mean. If they're misleading then
deleting is correct.
{code}
+if (jthr) {
+errno = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
+hdfsTell: org.apache.hadoop.fs.%s::getPos,
+((f-type == INPUT) ? FSDataInputStream :
+ FSDataOutputStream));
{code}
Please use {{interface}} here rather than recapitulating its ternary.
more review to come ... two-thirds done now ...
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3579) libhdfs: fix exception handling
[
https://issues.apache.org/jira/browse/HDFS-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426291#comment-13426291
]
Colin Patrick McCabe commented on HDFS-3579:
bq. I find these JAVA EQUIVALENT comments to be very helpful, could we keep
them around? so long as they're accurate, I mean. If they're misleading then
deleting is correct.
ok.
bq. Please use interface here rather than recapitulating its ternary.
It was done to avoid printing out org/apache/hadoop/fs/FSDataInputStream etc.
since- as you commeted above-- it's nicer to print something shorter. I don't
have a strong feeling about it either way, but I suspect it's easier just to
redo the ternary here.
bq. Is this correct? FSDataInputStream#read returns -1 on EOF and 0 on EINTR?
That's special. I see docs for the -1 case, but I don't see anywhere that the 0
could come from?
The standard Java convention is that -1 means EOF, and 0 is just a short read.
hdfsRead, on the other hand, follows the UNIX convention. See HADOOP-1582 for
more details.
bq. Let's preserve the interface that hdfsFile file can be NULL without causing
SEGV. Just toss in a if (file == NULL) return -1; near the top.
The whole thing is kind of messy. Passing a NULL pointer to hdfsClose is a
user error, yet we check for it for some reason.
There's no way to actually *get* an hdfsFile which has file-type
UNINITIALIZED. Every hdfsOpen path either leads to returning null, or
returning a file of type INPUT or OUTPUT. There's no way to close a file that
hasn't been opened either. Similarly, there's no way to get a file where
file-file is NULL.
The original code didn't check for file-file being NULL either (look at it
carefully, you'll see what I mean).
tl;dr: I didn't change the behavior here. But someone should eventually.
libhdfs: fix exception handling
---
Key: HDFS-3579
URL: https://issues.apache.org/jira/browse/HDFS-3579
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs
Affects Versions: 2.0.1-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3579.004.patch, HDFS-3579.005.patch,
HDFS-3579.006.patch
libhdfs does not consistently handle exceptions. Sometimes we don't free the
memory associated with them (memory leak). Sometimes we invoke JNI functions
that are not supposed to be invoked when an exception is active.
Running a libhdfs test program with -Xcheck:jni shows the latter problem
clearly:
{code}
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
WARNING in native method: JNI call made with exception pending
Exception in thread main java.io.IOException: ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
