[jira] [Commented] (HDFS-3134) Harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273280#comment-13273280 ] Hudson commented on HDFS-3134: -- Integrated in Hadoop-Mapreduce-trunk #1076 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1076/]) HDFS-3134. harden edit log loader against malformed or malicious input. Contributed by Colin Patrick McCabe (Revision 1336943) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1336943 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 2.0.0 Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) Harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273230#comment-13273230 ] Hudson commented on HDFS-3134: -- Integrated in Hadoop-Hdfs-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1040/]) HDFS-3134. harden edit log loader against malformed or malicious input. Contributed by Colin Patrick McCabe (Revision 1336943) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1336943 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 2.0.0 Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272879#comment-13272879 ] Eli Collins commented on HDFS-3134: --- +1 looks good harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) Harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272895#comment-13272895 ] Hudson commented on HDFS-3134: -- Integrated in Hadoop-Hdfs-trunk-Commit #2299 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2299/]) HDFS-3134. harden edit log loader against malformed or malicious input. Contributed by Colin Patrick McCabe (Revision 1336943) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1336943 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 2.0.0 Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) Harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272924#comment-13272924 ] Hudson commented on HDFS-3134: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2242 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2242/]) HDFS-3134. harden edit log loader against malformed or malicious input. Contributed by Colin Patrick McCabe (Revision 1336943) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1336943 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java Harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 2.0.0 Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271971#comment-13271971 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526231/HDFS-3134.009.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2397//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2397//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266728#comment-13266728 ] Colin Patrick McCabe commented on HDFS-3134: I looked at https://builds.apache.org/job/PreCommit-HDFS-Build/2359//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html and it contains this: bq. org.apache.hadoop.hdfs.server.namenode.INodeFileUnderConstruction doesn't override INodeFile.equals(Object) Since this patch doesn't change INodeFileUnderConstruction at all, I can only assume that this is a known problem where someone introduced a findbugs bug without introducing a suppression. So this patch should be ready to submit... harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266189#comment-13266189 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525230/HDFS-3134.006.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2357//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266308#comment-13266308 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525248/HDFS-3134.007.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2359//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2359//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2359//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, HDFS-3134.006.patch, HDFS-3134.007.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252834#comment-13252834 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522464/HDFS-3134.004.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2267//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch, HDFS-3134.004.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252144#comment-13252144 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522365/HDFS-3134.003.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2255//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, HDFS-3134.003.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248624#comment-13248624 ] Colin Patrick McCabe commented on HDFS-3134: Strictly speaking, readPositiveVInt should be readNonNegativeVInt. See if you want to change it. Ha! You know, I was actually thinking that. 0 is not positive, but it is non-negative. I suppose I should just bite the bullet and call it readNonNegativeVInt, despite the fact that it's a longer name. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248722#comment-13248722 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521726/HDFS-3134.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2218//console This message is automatically generated. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240565#comment-13240565 ] Colin Patrick McCabe commented on HDFS-3134: Hi Suresh, I'm sorry if my description was unclear. I am not talking about blindly translating unchecked exceptions into something else. I'm talking about fixing the code so it doesn't generate those unchecked exceptions in the first place. Hope this helps. Colin harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240181#comment-13240181 ] Suresh Srinivas commented on HDFS-3134: --- bq. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. From which methods? Unchecked exceptions indicate programming errors. Blindly turning them into checked exceptions is not a good idea (as you say so in some of your comments). I am not sure which part of the code you are talking about. harden edit log loader against malformed or malicious input --- Key: HDFS-3134 URL: https://issues.apache.org/jira/browse/HDFS-3134 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Currently, the edit log loader does not handle bad or malicious input sensibly. We can often cause OutOfMemory exceptions, null pointer exceptions, or other unchecked exceptions to be thrown by feeding the edit log loader bad input. In some environments, an out of memory error can cause the JVM process to be terminated. It's clear that we want these exceptions to be thrown as IOException instead of as unchecked exceptions. We also want to avoid out of memory situations. The main task here is to put a sensible upper limit on the lengths of arrays and strings we allocate on command. The other task is to try to avoid creating unchecked exceptions (by dereferencing potentially-NULL pointers, for example). Instead, we should verify ahead of time and give a more sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
