Sergey Soldatov created HDDS-17847:
--------------------------------------
Summary: Intermittent failure in balancer acceptance test:
recommissioned datanode keeps 2 replicas instead of 3
Key: HDDS-17847
URL: https://issues.apache.org/jira/browse/HDDS-17847
Project: Apache Ozone
Issue Type: Bug
Components: Balancer
Affects Versions: 2.3.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
Verify Container Balancer for RATIS/EC containers in
ozone-balancer/test-ratis.sh fails sometimes:
{noformat}
'209719296 > 104857600 * 3' should be true.
{noformat}
The test empties datanode1, writes 3 keys of 100 MB (9 replicas on 4
datanodes), puts datanode1 back in service and runs:
{noformat}
ozone admin containerbalancer start -t 0.1 -d 100 -i 3
{noformat}
It then expects datanode1 to hold exactly 3 replicas (LOWER_LIMIT:3,
UPPER_LIMIT:3.5).
The threshold is a percentage, so -t 0.1 allows only about 2 MB of difference
on a 2 GB datanode. With 9 replicas on 4 datanodes the cluster can never be
balanced: a datanode with 3 replicas is always over-utilized. The balancer
keeps moving containers until it runs out of iterations or valid moves.
In the failed run, iteration 1 moved all 3 containers to datanode1. In
iteration 2 datanode1 counted as over-utilized and the balancer moved one
container back. All moves succeeded, so the balancer did what it was asked.
Where the third replica ends up depends on how fresh the datanode usage in SCM
is (node report every 20 s, iteration interval 25 s).
Fix: in test-ratis.sh use LOWER_LIMIT:2 (or 1.9). The test then checks that
datanode1 received data and ended with 2 or 3 replicas. The earlier step in the
test already checks that data moved.
Seen in:
* 2026-11-16, attempt 1 (passed on re-run):
https://github.com/ss77892/ozone/actions/runs/43892876037/job/133394786063
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]