Gargi-jais11 commented on code in PR #9858:
URL: https://github.com/apache/ozone/pull/9858#discussion_r2922517124
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/node/TestDiskBalancerPolicyPerformance.java:
##########
@@ -181,90 +177,165 @@ private void testPolicyPerformance(String policyName,
ContainerChoosingPolicy po
for (int i = 0; i < NUM_THREADS; i++) {
executor.submit(() -> {
try {
- long threadStart = System.nanoTime();
- int containerChosen = 0;
- int containerNotChosen = 0;
+ int containerCandidateChosen = 0;
+ int containerCandidateNotChosen = 0;
int failures = 0;
- // Choose a random volume
- HddsVolume srcVolume = volumes.get(rand.nextInt(NUM_VOLUMES));
- HddsVolume destVolume;
-
- do {
- destVolume = volumes.get(rand.nextInt(NUM_VOLUMES));
- } while (srcVolume.equals(destVolume));
for (int j = 0; j < NUM_ITERATIONS; j++) {
+ // Block all volumes except the highest-util (source) - no
destination has space
+ if (rand.nextDouble() < 0.05 && volumes.size() >= 2) {
+ List<HddsVolume> sorted = new ArrayList<>(volumes);
+ sorted.sort(Comparator.comparingDouble(v ->
+ v.getCurrentUsage().getUsedSpace() / (double) Math.max(1,
v.getCurrentUsage().getCapacity())));
+ for (int k = 0; k < sorted.size() - 1; k++) {
+ HddsVolume v = sorted.get(k);
+ long avail = v.getCurrentUsage().getAvailable();
+ if (avail > 0) {
+ v.incCommittedBytes(avail);
Review Comment:
> And please find out why there are far few nulls than expected, if there is
any hidden bug.
Actually the background activity which I added before blocked-all sorted by
raw `usedSpace/capacity`, while the policy sorts by effective utilization
(including deltaMap and committedBytes). After blocking, the blocked volumes’
effective utilization became ~100%, so they became the source volumes. The
volume that left unblocked stayed at lower effective utilization and became a
valid destination with usable space. The policy therefore kept finding a valid
destination and returned a chosen container instead of null.
So due to that check the policy was moreover titled to one side of the
scenario.
Now in the updated performance test I have made sure that during volume
creation they are created with varying utilisation in a way that all the
scenarios are covered and similarly for containers created many zero bytes or
open or closed and created batches of container with different used space so
that when volumes are available then container won't be sometime due to less
available space on destination disk.
the test shows:
(1) valid pair and contianer chosen while imbalanced,
(2) valid pair but no container in some cases, and
(3) no valid destination when volumes are balanced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
