[ https://issues.apache.org/jira/browse/HDFS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730348#comment-17730348 ]
ASF GitHub Bot commented on HDFS-17037: --------------------------------------- Hexiaoqiao commented on code in PR #5715: URL: https://github.com/apache/hadoop/pull/5715#discussion_r1222377990 ########## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java: ########## @@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) { for(StorageReport s : r.getStorageReports()) { final StorageType t = s.getStorage().getStorageType(); totalCapacities.add(t, s.getCapacity()); - totalUsedSpaces.add(t, s.getDfsUsed()); + totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining())); Review Comment: It was not necessary to involve extra outer brackets here. `totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining()));` -> `totalUsedSpaces.add(t, s.getCapacity() - s.getRemaining());` ########## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java: ########## @@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) { for(StorageReport s : r.getStorageReports()) { final StorageType t = s.getStorage().getStorageType(); totalCapacities.add(t, s.getCapacity()); - totalUsedSpaces.add(t, s.getDfsUsed()); + totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining())); } } @Override Double getUtilization(DatanodeStorageReport r, final StorageType t) { long capacity = 0L; - long dfsUsed = 0L; + long totalUsed = 0L; for(StorageReport s : r.getStorageReports()) { if (s.getStorage().getStorageType() == t) { capacity += s.getCapacity(); - dfsUsed += s.getDfsUsed(); + totalUsed += (s.getCapacity() - s.getRemaining()); Review Comment: As mentioned above. ########## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java: ########## @@ -138,7 +138,7 @@ String getName() { void accumulateSpaces(DatanodeStorageReport r) { for(StorageReport s : r.getStorageReports()) { final StorageType t = s.getStorage().getStorageType(); - totalCapacities.add(t, s.getCapacity()); + totalCapacities.add(t, s.getRemaining() + s.getBlockPoolUsed()); Review Comment: IMO, it's difficult to understand the meaning of this improvement, just suggest add some annotations for Pool policy and describe what and why. > Consider nonDfsUsed when running balancer > ----------------------------------------- > > Key: HDFS-17037 > URL: https://issues.apache.org/jira/browse/HDFS-17037 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Shuyan Zhang > Assignee: Shuyan Zhang > Priority: Major > Labels: pull-request-available > > When we run balancer with `BalancingPolicy.Node` policy, our goal is to make > each datanode storage balanced. But in the current implementation, the > balancer doesn't account for storage used by non-dfs on the datanodes, which > can make the situation worse for datanodes that are already strained on > storage. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org