[
https://issues.apache.org/jira/browse/HDDS-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HDDS-17847:
-----------------------------------
Parent: HDDS-5626
Issue Type: Sub-task (was: Bug)
> Intermittent failure in balancer acceptance test: recommissioned datanode
> keeps 2 replicas instead of 3
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17847
> URL: https://issues.apache.org/jira/browse/HDDS-17847
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Balancer
> Affects Versions: 2.3.0
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
>
> Verify Container Balancer for RATIS/EC containers in
> ozone-balancer/test-ratis.sh fails sometimes:
> {noformat}
> '209719296 > 104857600 * 3' should be true.
> {noformat}
> The test empties datanode1, writes 3 keys of 100 MB (9 replicas on 4
> datanodes), puts datanode1 back in service and runs:
> {noformat}
> ozone admin containerbalancer start -t 0.1 -d 100 -i 3
> {noformat}
> It then expects datanode1 to hold exactly 3 replicas (LOWER_LIMIT:3,
> UPPER_LIMIT:3.5).
> The threshold is a percentage, so -t 0.1 allows only about 2 MB of difference
> on a 2 GB datanode. With 9 replicas on 4 datanodes the cluster can never be
> balanced: a datanode with 3 replicas is always over-utilized. The balancer
> keeps moving containers until it runs out of iterations or valid moves.
> In the failed run, iteration 1 moved all 3 containers to datanode1. In
> iteration 2 datanode1 counted as over-utilized and the balancer moved one
> container back. All moves succeeded, so the balancer did what it was asked.
> Where the third replica ends up depends on how fresh the datanode usage in
> SCM is (node report every 20 s, iteration interval 25 s).
> Fix: in test-ratis.sh use LOWER_LIMIT:2 (or 1.9). The test then checks that
> datanode1 received data and ended with 2 or 3 replicas. The earlier step in
> the test already checks that data moved.
> Seen in:
> * 2026-11-16, attempt 1 (passed on re-run):
> https://github.com/ss77892/ozone/actions/runs/43892876037/job/133394786063
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]