Srinivasu Majeti created HDFS-14859:
---------------------------------------
Summary: Prevent Un-necessary evaluation of costly operation
getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero
Key: HDFS-14859
URL: https://issues.apache.org/jira/browse/HDFS-14859
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.1.0, 3.3.0, 3.1.4
Reporter: Srinivasu Majeti
There have been improvements like HDFS-14171 and HDFS-14632 to the performance
issue caused from getNumLiveDataNodes calls per block. The improvement has been
only done w.r.t dfs.namenode.safemode.min.datanodes paramter being set to 0 or
not.
private boolean areThresholdsMet() {
assert namesystem.hasWriteLock();
- int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
+ // Calculating the number of live datanodes is time-consuming
+ // in large clusters. Skip it when datanodeThreshold is zero.
+ int datanodeNum = 0;
+ if (datanodeThreshold > 0) {
+ datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
+ }
synchronized (this) {
return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
}
I feel above logic would create similar situation of un-necessary evaluations
of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is set
> 0 even though "blockSafe >= blockThreshold" is false for most of the time in
NN startup safe mode. We could do something like below to avoid this
private boolean areThresholdsMet() {
assert namesystem.hasWriteLock();
synchronized (this) {
return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
blockManager.getDatanodeManager().getNumLiveDataNodes() >=
datanodeThreshold : true;
}
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
