Riza Suminto created HDFS-11684: ----------------------------------- Summary: Potential scalability bug on datanode recommission Key: HDFS-11684 URL: https://issues.apache.org/jira/browse/HDFS-11684 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.3 Reporter: Riza Suminto
We are academic researcher working on scalability bug detection. Upon analyzing Hadoop code, our static analysis program detect potential scalability problem along this function calls: DatanodeManager.refreshNodes() -> DatanodeManager.refreshDatanodes() -> DecommissionManager.stopDecommission() -> BlockManager.processExtraRedundancyBlocksOnInService() DatanodeManager.refreshNodes() call namesystem.writeLock(), which will lock FSNamesystem. While locking FSNamesystem, it does nested loop over datanodes in DatanodeManager.refreshDatanodes() and over datanode blocks in BlockManager.processExtraRedundancyBlocksOnInService(). Looking only at number of executed lines, refreshDatanodes will execute N*(136*B+137) lines of codes in total, where N is number of datanodes and B is number of blocks per node. This is a potential scalability bug that may render FSNamesystem unresponsive for long time if refreshNodes trigger recommissioning of many fat datanodes, having large number of over replicated datablocks. This bug also discussed in HDFS-10477, but still left unresolved. In older hadoop version, our program also detect similar scalability problem in decommissioning datanodes, which has been fixed on HADOOP-4061. However, recommissioning still have scalabillity problem in current version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org