anuragp010 commented on code in PR #12062:
URL: https://github.com/apache/ozone/pull/12062#discussion_r4624707957
##########
hadoop-hdds/docs/content/feature/ContainerBalancer.md:
##########
@@ -84,9 +84,29 @@ does not count as spare SSD space. Byte and datanode limits
remain shared across
Assessment warns when storage-type reports are incomplete, and estimation
rejects those
reports. When no storage-type breakdown is available, both commands use
overall usage.
-These commands report and estimate balancing work. The running balancer still
chooses
-moves using overall datanode usage; selecting moves separately for each
storage type is
-separate work.
+### Balancing by storage type
+
+When complete storage-type usage reports are available, the running balancer
selects
+sources and targets separately for each type. For example, a datanode at 80%
SSD usage
+can send an SSD replica to another datanode at 20% SSD usage, even when both
datanodes
+have the same overall utilization. Free DISK space is not a destination for
that SSD copy.
+The selected type is included in the copy command so the destination uses a
matching volume.
+
+Storage types take turns within each iteration, and the first type rotates
between
+iterations. The iteration byte limit, each datanode's incoming and outgoing
byte limits,
+and the distinct-datanode limit are shared across all types. A datanode that
sends SSD
+replicas and receives DISK replicas counts once toward the datanode limit.
+
+For an existing replica without a recorded storage type, the balancer uses its
reported
+volume type. It skips replicas whose type is unknown or whose recorded type
conflicts
Review Comment:
Nit: `whose type is unknown` is vague. Could this be rewritten as `It skips
replicas whose recorded type and reported volume type are both unknown or whose
recorded type conflicts with their reported volume type` ?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerSelectionCriteria.java:
##########
@@ -426,6 +458,32 @@ Set<ContainerID> getExcludeNotFoundContainers() {
return excludeContainersNotFound;
}
+ /**
+ * Whether the container's replica on {@code node} sits on {@code
storageType}.
+ * Unknown types and conflicting recorded/volume types cannot participate in
a typed move.
+ */
+ private static boolean isReplicaOnStorageType(Set<ContainerReplica> replicas,
+ DatanodeDetails node, StorageType storageType) {
+ for (ContainerReplica replica : replicas) {
+ if (replica.getDatanodeDetails().equals(node)) {
+ if (replica.getStorageType() != null && replica.getVolumeStorageType()
!= null
+ && replica.getStorageType() != replica.getVolumeStorageType()) {
Review Comment:
The check for unknown and conflicting types is in multiple places. Can we
extract this into a single method in `ContainerReplica` and use that ? Perhaps
something like:
```java
/** True when the recorded type and the reported volume type are both known
and differ. */
public boolean hasConflictingStorageTypes() {
return storageType != null && volumeStorageType != null && storageType !=
volumeStorageType;
}
```
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplica.java:
##########
@@ -150,6 +150,14 @@ public StorageType getVolumeStorageType() {
return volumeStorageType;
}
+ /**
+ * Returns the recorded replica type, or its reported volume type when the
recorded type is absent.
+ * A null result means neither type is known. Callers must handle
conflicting known types separately.
+ */
+ public StorageType getTargetStorageTypeForCopy() {
Review Comment:
Nit: `getTargetStorageTypeForCopy()` returns the type of this replica: the
recorded type, or the volume type if no type is recorded. This actually
represents the effective storage type of the existing replica. Would
`getEffectiveStorageType()` be a better method name here?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerTask.java:
##########
@@ -557,56 +560,45 @@ private boolean initializeIteration() {
this.totalNodesInCluster = datanodeUsageInfos.size();
- double clusterAvgUtilisation = calculateAvgUtilization(datanodeUsageInfos);
- if (LOG.isDebugEnabled()) {
- LOG.debug("Average utilization of the cluster is {}",
clusterAvgUtilisation);
+ // Node include/exclude filters have already been applied above.
+ ContainerBalancerClusterSnapshot analysis =
ContainerBalancerClusterAnalyzer.analyze(datanodeUsageInfos.stream()
+ .map(node ->
node.toProto(ClientVersion.DEFAULT_VERSION)).collect(Collectors.toList()),
+ config.getThresholdAsRatio(), Collections.emptySet(),
Collections.emptySet());
+ if (analysis.hasIncompleteStorageTypeUsage()) {
+ lastInitializationFailureDetail = "Incomplete storage-type usage
reports; cannot safely select typed moves";
+ LOG.warn(lastInitializationFailureDetail);
+ return false;
}
-
- double threshold = config.getThresholdAsRatio();
- // over utilized nodes have utilization(that is, used / capacity) greater
than upper limit
- this.upperLimit = ContainerBalancerClusterAnalyzer.computeUpperLimit(
- clusterAvgUtilisation, threshold);
- // under utilized nodes have utilization(that is, used / capacity) less
than lower limit
- this.lowerLimit = ContainerBalancerClusterAnalyzer.computeLowerLimit(
- clusterAvgUtilisation, threshold);
-
- if (LOG.isDebugEnabled()) {
- LOG.debug("Lower limit for utilization is {} and Upper limit for
utilization is {}", lowerLimit, upperLimit);
+ if (analysis.getStorageTypeSnapshots().isEmpty()) {
+ storageTypeStates.add(new StorageTypeBalancingState(null,
analysis.getLowerLimit(), analysis.getUpperLimit(),
+ datanodeUsageInfos, analysis.getSourceDatanodes(),
analysis.getTargetDatanodes()));
+ } else {
+ analysis.getStorageTypeSnapshots().forEach((type, snapshot) ->
+ storageTypeStates.add(new StorageTypeBalancingState(type,
snapshot.getLowerLimit(), snapshot.getUpperLimit(),
+ datanodeUsageInfos, snapshot.getSourceDatanodes(),
snapshot.getTargetDatanodes())));
}
-
- long totalOverUtilizedBytes = 0L, totalUnderUtilizedBytes = 0L;
- // find over and under utilized nodes
- for (DatanodeUsageInfo datanodeUsageInfo : datanodeUsageInfos) {
+ Set<DatanodeDetails> sources = new HashSet<>();
+ Set<DatanodeDetails> targets = new HashSet<>();
Review Comment:
I want to check my understanding here. `sources` and `targets` are built
from every `StorageTypeBalancingState`. That includes a type whose snapshot has
sources but no targets, or targets but no sources. For the combined
`analysis.getSourceDatanodes()` and `getTargetDatanodes()`,
`combineStorageTypes` only adds a type when `snapshot.getBytesToMove() > 0`.
So `numDatanodesUnbalanced` and `overUtilizedNodes` can include a node that
no typed move can use. Is that what is intended?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java:
##########
@@ -34,6 +34,9 @@ public final class ContainerBalancerMetrics {
private final MetricsSystem ms;
+ @Metric("Number of times scheduling reached the shared iteration byte limit")
+ private MutableCounterLong numIterationLimitsSkipped;
Review Comment:
Since this tracks the number of times an iteration of the container balancer
stops because the `maxSizeToMovePerIteration` limit is reached, should we
rename this to `numIterationsReachedMaxSizeToMoveLimit` or
`numIterationsReachedMaxSizeToMove` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]