lokeshj1703 commented on a change in pull request #3049:
URL: https://github.com/apache/ozone/pull/3049#discussion_r804389987
##########
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java
##########
@@ -37,27 +36,16 @@
private final MetricsSystem ms;
- @Metric(about = "The total amount of used space in GigaBytes that needs to "
+
- "be balanced.")
- private MutableGaugeLong dataSizeToBalanceGB;
+ @Metric(about = "The amount of Gigabytes that Container Balancer moved" +
+ " in the last iteration.")
+ private MutableCounterLong dataSizeMovedGB;
- @Metric(about = "The amount of Giga Bytes that have been moved to achieve " +
- "balance.")
- private MutableGaugeLong dataSizeMovedGB;
+ @Metric(about = "Number of containers that Container Balancer moved" +
+ " in the last iteration.")
+ private MutableCounterLong movedContainersNum;
- @Metric(about = "Number of containers that Container Balancer has moved" +
- " until now.")
- private MutableGaugeLong movedContainersNum;
-
- @Metric(about = "The total number of datanodes that need to be balanced.")
- private MutableGaugeLong datanodesNumToBalance;
Review comment:
Instead of removing these metrics should we replace them with a more
suitable metric?
##########
File path:
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancer.java
##########
@@ -500,29 +501,22 @@ public void
balancerShouldObeyMaxSizeEnteringTargetLimit() {
@Test
public void testMetrics() {
+ conf.set("hdds.datanode.du.refresh.period", "1ms");
+ balancerConfiguration.setBalancingInterval(Duration.ofMillis(2));
balancerConfiguration.setThreshold(10);
- balancerConfiguration.setIterations(1);
- balancerConfiguration.setMaxSizeEnteringTarget(10 * OzoneConsts.GB);
- balancerConfiguration.setMaxSizeToMovePerIteration(100 * OzoneConsts.GB);
+ balancerConfiguration.setIterations(2);
+ balancerConfiguration.setMaxSizeEnteringTarget(6 * OzoneConsts.GB);
+ // deliberately set max size per iteration to a low value, 6GB
+ balancerConfiguration.setMaxSizeToMovePerIteration(6 * OzoneConsts.GB);
balancerConfiguration.setMaxDatanodesPercentageToInvolvePerIteration(100);
containerBalancer.start(balancerConfiguration);
+ sleepWhileBalancing(500);
- // waiting for balance completed.
- // TODO: this is a temporary implementation for now
- // modify this after balancer is fully completed
- try {
- Thread.sleep(500);
- } catch (InterruptedException e) {}
-
- containerBalancer.stop();
ContainerBalancerMetrics metrics = containerBalancer.getMetrics();
- Assert.assertEquals(determineExpectedUnBalancedNodes(
- balancerConfiguration.getThreshold()).size(),
- metrics.getDatanodesNumToBalance());
- Assert.assertEquals(ContainerBalancer.ratioToPercent(
- nodeUtilizations.get(nodeUtilizations.size() - 1)),
- metrics.getMaxDatanodeUtilizedPercentage());
+ Assert.assertTrue(metrics.getDataSizeMovedGB() <= 6);
+ Assert.assertEquals(2, metrics.getCountIterations());
+ containerBalancer.stop();
Review comment:
Also it would be a good idea to test these metric values in one of the
tests which balance the data via mock apis.
##########
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java
##########
@@ -37,27 +36,16 @@
private final MetricsSystem ms;
- @Metric(about = "The total amount of used space in GigaBytes that needs to "
+
- "be balanced.")
- private MutableGaugeLong dataSizeToBalanceGB;
+ @Metric(about = "The amount of Gigabytes that Container Balancer moved" +
+ " in the last iteration.")
+ private MutableCounterLong dataSizeMovedGB;
- @Metric(about = "The amount of Giga Bytes that have been moved to achieve " +
- "balance.")
- private MutableGaugeLong dataSizeMovedGB;
+ @Metric(about = "Number of containers that Container Balancer moved" +
+ " in the last iteration.")
+ private MutableCounterLong movedContainersNum;
Review comment:
Since this metric is valid only per iteration, maybe we can add
perIteration in the metric name.
Also it might be good to support aggregate metrics across iterations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
