Srinivasu Majeti created HDFS-14859: ---------------------------------------
Summary: Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero Key: HDFS-14859 URL: https://issues.apache.org/jira/browse/HDFS-14859 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.1.0, 3.3.0, 3.1.4 Reporter: Srinivasu Majeti There have been improvements like HDFS-14171 and HDFS-14632 to the performance issue caused from getNumLiveDataNodes calls per block. The improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes paramter being set to 0 or not. private boolean areThresholdsMet() { assert namesystem.hasWriteLock(); - int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes(); + // Calculating the number of live datanodes is time-consuming + // in large clusters. Skip it when datanodeThreshold is zero. + int datanodeNum = 0; + if (datanodeThreshold > 0) { + datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes(); + } synchronized (this) { return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold; } I feel above logic would create similar situation of un-necessary evaluations of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is set > 0 even though "blockSafe >= blockThreshold" is false for most of the time in NN startup safe mode. We could do something like below to avoid this private boolean areThresholdsMet() { assert namesystem.hasWriteLock(); synchronized (this) { return blockSafe >= blockThreshold && (datanodeThreshold > 0)? blockManager.getDatanodeManager().getNumLiveDataNodes() >= datanodeThreshold : true; } } -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org