Sergey Soldatov created HDDS-17847:
--------------------------------------

             Summary: Intermittent failure in balancer acceptance test: 
recommissioned datanode keeps 2 replicas instead of 3
                 Key: HDDS-17847
                 URL: https://issues.apache.org/jira/browse/HDDS-17847
             Project: Apache Ozone
          Issue Type: Bug
          Components: Balancer
    Affects Versions: 2.3.0
            Reporter: Sergey Soldatov
            Assignee: Sergey Soldatov


Verify Container Balancer for RATIS/EC containers in 
ozone-balancer/test-ratis.sh fails sometimes:
{noformat}
'209719296 > 104857600 * 3' should be true.
{noformat}

The test empties datanode1, writes 3 keys of 100 MB (9 replicas on 4 
datanodes), puts datanode1 back in service and runs:
{noformat}
ozone admin containerbalancer start -t 0.1 -d 100 -i 3
{noformat}
It then expects datanode1 to hold exactly 3 replicas (LOWER_LIMIT:3, 
UPPER_LIMIT:3.5).

The threshold is a percentage, so -t 0.1 allows only about 2 MB of difference 
on a 2 GB datanode. With 9 replicas on 4 datanodes the cluster can never be 
balanced: a datanode with 3 replicas is always over-utilized. The balancer 
keeps moving containers until it runs out of iterations or valid moves.

In the failed run, iteration 1 moved all 3 containers to datanode1. In 
iteration 2 datanode1 counted as over-utilized and the balancer moved one 
container back. All moves succeeded, so the balancer did what it was asked. 
Where the third replica ends up depends on how fresh the datanode usage in SCM 
is (node report every 20 s, iteration interval 25 s).

Fix: in test-ratis.sh use LOWER_LIMIT:2 (or 1.9). The test then checks that 
datanode1 received data and ended with 2 or 3 replicas. The earlier step in the 
test already checks that data moved.

Seen in:
* 2026-11-16, attempt 1 (passed on re-run): 
https://github.com/ss77892/ozone/actions/runs/43892876037/job/133394786063



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to