[jira] [Updated] (HDFS-2573) TestFiDataXceiverServer is failing, not testing OOME
[ https://issues.apache.org/jira/browse/HDFS-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2573: - Affects Version/s: 0.24.0 Also committed to the trunk TestFiDataXceiverServer is failing, not testing OOME Key: HDFS-2573 URL: https://issues.apache.org/jira/browse/HDFS-2573 Project: Hadoop HDFS Issue Type: Bug Components: data-node, test Affects Versions: 0.22.0, 0.24.0 Reporter: Konstantin Shvachko Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: HDFS-2573.patch, HDFS-2573.patch TestFiDataXceiverServer is failing occasionally. It turns out also that the test is not testing the desired condition, because OOME is not caught in DataXceiverServer.run() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2573) TestFiDataXceiverServer is failing, not testing OOME
[ https://issues.apache.org/jira/browse/HDFS-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2573: - Attachment: HDFS-2573.patch Good riddance. TestFiDataXceiverServer is failing, not testing OOME Key: HDFS-2573 URL: https://issues.apache.org/jira/browse/HDFS-2573 Project: Hadoop HDFS Issue Type: Bug Components: data-node, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Attachments: HDFS-2573.patch TestFiDataXceiverServer is failing occasionally. It turns out also that the test is not testing the desired condition, because OOME is not caught in DataXceiverServer.run() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2513) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/HDFS-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2513: - Attachment: HDFS-2513.patch Corrected patch addressing hdfsproxy dependencies (which is sorta useless because hdfsproxy seems to be abandoned any way) Bump jetty to 6.1.26 Key: HDFS-2513 URL: https://issues.apache.org/jira/browse/HDFS-2513 Project: Hadoop HDFS Issue Type: Task Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: HADOOP-2513.patch, HDFS-2513.patch HDFS part of Hadoop-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2513) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/HDFS-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2513: - Attachment: HADOOP-2513.patch I have ran commit tests locally with HADOOP-7450 and this one. {{org.apache.hadoop.hdfs.TestHDFSTrash}} timeouted. And {{org.apache.hadoop.hdfs.TestFileAppend3#testAppendToPartialChunk}} failed. The problems aren't related to the patch apparently: it is likely my Mac issues. Bump jetty to 6.1.26 Key: HDFS-2513 URL: https://issues.apache.org/jira/browse/HDFS-2513 Project: Hadoop HDFS Issue Type: Task Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: HADOOP-2513.patch HDFS part of Hadoop-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Attachment: HDFS-2452.patch Here's a patch for trunk. Something wrong with my ant installation or something and maven complains that it can't run the build. Thus, I can't verify it right now - will do later. The patch also contains new test and instrumentation but it won't be verified until AOP is fixed in trunk. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, HDFS-2452-22branch_with-around_.patch, HDFS-2452.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Attachment: HDFS-2452-22branch_with-around_.patch OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, HDFS-2452-22branch_with-around_.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2451) TestNodeCount.testNodeCount failes with NPE
[ https://issues.apache.org/jira/browse/HDFS-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2451: - Attachment: HDFS-2451.patch Because I've missed it ;( Thanks for catching. TestNodeCount.testNodeCount failes with NPE --- Key: HDFS-2451 URL: https://issues.apache.org/jira/browse/HDFS-2451 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Priority: Blocker Attachments: HDFS-2451.patch, HDFS-2451.patch in the [commit build #97|http://is.gd/UPuXg2] TestNodeCount.testNodeCount failed with NPE {noformat} java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.BlockManager.countNodes(BlockManager.java:1436) at org.apache.hadoop.hdfs.server.namenode.TestNodeCount.__CLR2_4_39bdgm6whp(TestNodeCount.java:119) at org.apache.hadoop.hdfs.server.namenode.TestNodeCount.testNodeCount(TestNodeCount.java:40) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Status: Open (was: Patch Available) We aren't running auto-validation of .22 patches anyway. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2451) TestNodeCount.testNodeCount failes with NPE
[ https://issues.apache.org/jira/browse/HDFS-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2451: - Attachment: HDFS-2451.patch Seems like the call to {{blockManager.countNodes()}} is happening under namesystem's write locking elsewhere. Otherwise a race condition is likely. TestNodeCount.testNodeCount failes with NPE --- Key: HDFS-2451 URL: https://issues.apache.org/jira/browse/HDFS-2451 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Priority: Blocker Attachments: HDFS-2451.patch in the [commit build #97|http://is.gd/UPuXg2] TestNodeCount.testNodeCount failed with NPE {noformat} java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.BlockManager.countNodes(BlockManager.java:1436) at org.apache.hadoop.hdfs.server.namenode.TestNodeCount.__CLR2_4_39bdgm6whp(TestNodeCount.java:119) at org.apache.hadoop.hdfs.server.namenode.TestNodeCount.testNodeCount(TestNodeCount.java:40) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Attachment: HDFS-2452-22branch.patch With respect to the case I think getting rid of Mockito.doThrow and spying on {{DataXceiverServer}} should do the trick. So, something along the lines {code} ... DataNode dn = Mockito.mock(DataNode.class); DataXceiverServer server = new DataXceiverServer(sock, new Configuration(), dn); DataXceiverServer spyServer = spy(server); doThrow(new OutOfMemoryError(Faulting...)).when(spyServer).run(); dn.shouldRun = true; ... {code} would work, I believe. Fixed patch version is attached. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: HDFS-1762.hdfs.patch Addressing issue for hostnames with '-' Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: HDFS-1762.hdfs.patch Fixing comparators for 4 cases. Now everything passes with hdfs-0.22 cluster. Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: HDFS-1762.patch Added missing license. Also, I'd propose to address any regexp issues separately. The reason for two files was to avoid too wide (e.g. generic) expressions in the first place. Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch, HDFS-1762.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: (was: HDFS-1762.common.patch) Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: (was: HDFS-1762.mapreduce.patch) Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.hdfs.patch, HDFS-1762.hdfs.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Attachment: HDFS-1762.mapreduce.patch HDFS-1762.hdfs.patch HDFS-1762.common.patch These are patches for all three subprojects for 0.22 branch. I will create separate tickets in a bit. They should be mostly good for 0.23 except the ant build part which needs to be re-worked for maven. Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.common.patch, HDFS-1762.hdfs.patch, HDFS-1762.mapreduce.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Component/s: test build Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Components: build, test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch, HDFS-1762.common.patch, HDFS-1762.hdfs.patch, HDFS-1762.mapreduce.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1762) Allow TestHDFSCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/HDFS-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1762: - Affects Version/s: 0.22.0 Allow TestHDFSCLI to be run against a cluster - Key: HDFS-1762 URL: https://issues.apache.org/jira/browse/HDFS-1762 Project: Hadoop HDFS Issue Type: Test Affects Versions: 0.22.0 Reporter: Tom White Assignee: Konstantin Boudnik Attachments: HDFS-1762-20.patch Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-1943: - Affects Version/s: 0.22.0 Fix Version/s: 0.22.0 fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.22.0, 0.23.0 Reporter: Wei Yongjun Assignee: Wei Yongjun Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira