[ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133209#comment-13133209 ]
Hadoop QA commented on HDFS-2476: --------------------------------- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500263/hashStructures.patch-3 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestFileCreationDelete org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.fs.TestFcHdfsCreateMkdir org.apache.hadoop.security.TestPermission org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.fs.viewfs.TestViewFsHdfs org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.TestDFSPermission org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.fs.TestFcHdfsSymlink org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.tools.TestJMXGet org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1412//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1412//console This message is automatically generated. > More CPU efficient data structure for > under-replicated/over-replicated/invalidate blocks > ---------------------------------------------------------------------------------------- > > Key: HDFS-2476 > URL: https://issues.apache.org/jira/browse/HDFS-2476 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node > Reporter: Tomasz Nykiel > Assignee: Tomasz Nykiel > Attachments: hashStructures.patch, hashStructures.patch-2, > hashStructures.patch-3 > > > This patch introduces two hash data structures for storing under-replicated, > over-replicated and invalidated blocks. > 1. LightWeightHashSet > 2. LightWeightLinkedSet > Currently in all these cases we are using java.util.TreeSet which adds > unnecessary overhead. > The main bottlenecks addressed by this patch are: > -cluster instability times, when these queues (especially under-replicated) > tend to grow quite drastically, > -initial cluster startup, when the queues are initialized, after leaving > safemode, > -block reports, > -explicit acks for block addition and deletion > 1. The introduced structures are CPU-optimized. > 2. They shrink and expand according to current capacity. > 3. Add/contains/delete ops are performed in O(1) time (unlike current log n > for TreeSet). > 4. The sets are equipped with fast access methods for polling a number of > elements (get+remove), which are used for handling the queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira