sodonnel commented on code in PR #4496:
URL: https://github.com/apache/ozone/pull/4496#discussion_r1153297201
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java:
##########
@@ -1160,16 +1183,31 @@ public void setMaintenanceReplicaMinimum(int
replicaCount) {
defaultValue = "20",
tags = { SCM, DATANODE },
description = "A limit to restrict the total number of replication " +
- "commands queued on a datanode. Note this is intended to be a " +
- " temporary config until we have a more dynamic way of limiting " +
- "load."
+ "and reconstruction commands queued on a datanode. Note this is " +
+ "intended to be a temporary config until we have a more dynamic " +
+ "way of limiting load."
)
private int datanodeReplicationLimit = 20;
public int getDatanodeReplicationLimit() {
return datanodeReplicationLimit;
}
+ @Config(key = "datanode.reconstruction.weight",
+ type = ConfigType.INT,
+ defaultValue = "3",
+ tags = { SCM, DATANODE },
+ description = "When counting the number of replication commands on a "
+
+ "datanode, the number of reconstruction commands is multiplied " +
+ "by this weight to ensure reconstruction commands use more of " +
+ "the capacity, as they are more expensive to process."
+ )
+ private int reconstructionCommandWeight = 3;
Review Comment:
Yea this is a good point, and warrants more thought. The question is what
value should be used. Eg we could use the parity number, or some functions of
the nodes involved, but we may still need a config setting to "tune" it. I will
create another Jira under our RM improvements epic so we can consider how best
to do this.
Its difficult to pick sensible defaults for these sort of settings without
testing on real hardware and data, and even then, the best setting probably
depends on the hardware capabilities, network and how the data looks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
