[ https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939287#comment-16939287 ]
Srinivasu Majeti edited comment on HDFS-14859 at 10/4/19 9:04 AM: ------------------------------------------------------------------ Hi [~dineshchitlangia], any comments before [~arp] commits the patch ? Could we mark current Jira "supercedes" HDFS-14171 now or after commit ? was (Author: smajeti): Hi [~dineshchitlangia], any comments before [~arp] commits the patch ? > Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when > dfs.namenode.safemode.min.datanodes is not zero > --------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-14859 > URL: https://issues.apache.org/jira/browse/HDFS-14859 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Affects Versions: 3.1.0, 3.3.0, 3.1.4 > Reporter: Srinivasu Majeti > Assignee: Srinivasu Majeti > Priority: Major > Labels: block > Attachments: HDFS-14859.001.patch, HDFS-14859.002.patch, > HDFS-14859.003.patch, HDFS-14859.004.patch, HDFS-14859.005.patch, > HDFS-14859.006.patch, HDFS-14859.007.patch > > > There have been improvements like HDFS-14171 and HDFS-14632 to the > performance issue caused from getNumLiveDataNodes calls per block. The > improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes > paramter being set to 0 or not. > {code} > private boolean areThresholdsMet() { > assert namesystem.hasWriteLock(); > - int datanodeNum = > blockManager.getDatanodeManager().getNumLiveDataNodes(); > + // Calculating the number of live datanodes is time-consuming > + // in large clusters. Skip it when datanodeThreshold is zero. > + int datanodeNum = 0; > + if (datanodeThreshold > 0) { > + datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes(); > + } > synchronized (this) { > return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold; > } > {code} > I feel above logic would create similar situation of un-necessary evaluations > of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is > set > 0 even though "blockSafe >= blockThreshold" is false for most of the > time in NN startup safe mode. We could do something like below to avoid this > {code} > private boolean areThresholdsMet() { > assert namesystem.hasWriteLock(); > synchronized (this) { > return blockSafe >= blockThreshold && (datanodeThreshold > 0)? > blockManager.getDatanodeManager().getNumLiveDataNodes() >= > datanodeThreshold : true; > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org