Rakesh Radhakrishnan created HDDS-16606:
-------------------------------------------
Summary: [Test Script] Automated container balancer orchestration
script: assessment, drift-based policy enforcement & observability metrics
Key: HDDS-16606
URL: https://issues.apache.org/jira/browse/HDDS-16606
Project: Apache Ozone
Issue Type: Sub-task
Components: Balancer, Tools
Reporter: Rakesh Radhakrishnan
Assignee: Sreeja Chintalapati
The idea here is to expose an external orchestration test script that automates
the full container balancer lifecycle — periodic assessment, drift-based speed
selection(or can propose a better approach), execution, progress monitoring,
and post-run statistics collection — as a standalone script.
As we know, the existing balancer workflow is operator-driven:
{noformat}
assess → dry-run → recommend → start → monitor → stop
{noformat}
This script is explicitly *not* an SCM-embedded service. The orchestrator runs
interacts with Ozone exclusively through:
* {{ozone admin containerbalancer}} CLI commands
* Prometheus metric scraping
* JMX queries etc
h2. User Behaviors
h3. Persona 1: Performance Engineer (Benchmarking)
* Deploys a test cluster, ingests a known dataset to create deliberate
imbalance
* Runs the script with SLOW, MEDIUM, FAST profiles against identical starting
conditions
* Watch for the report
h3. Persona 2: Developer (Regression Testing) in a System Test Env
* After modifying ContainerBalancer, runs the script on a CI cluster
* Generate a benchmark report against a known baseline (bytes/hour, success
rate, system impact)
* Flags regressions: e.g., throughput dropped 30%, or DN I/O wait increased 2×
* They can integrates the pre-built script into a CI pipeline as a gating check
h3. Persona 3: Operator (Pre-Production Dry Run Validation)
* Before scheduling a production balancer run during a maintenance window,
runs the harness script in dry-run mode on a staging cluster.
* Reviews the SLA estimate and system impact predictions
* Uses the generated {{report}} and can tune respective configs before the
production run
h2. Design
The script operates in three stages:
h3. Stage 1: Preparation
*Step 1.1 — Periodic Assessment*
* Invokes {{ozone admin containerbalancer assess}} on a configurable interval
(default: 5 hours)
* Appends each {{BalanceAssessment}} result to a JSONL time-series log (drift,
severity, bytesToMove, source/target counts)
*Step 1.2 — Pre-Built Profiles: Drift-Based Speed Selection*
* Evaluates the latest drift against static rules to select a speed profile:
||Drift Range||Speed Profile||Rationale||
|drift ≤ 2%|No action|Within tolerance|
|2% < drift ≤ 5%|SLOW|Background correction, minimal impact|
|5% < drift ≤ 10%|MEDIUM|Standard correction pace|
|drift > 10%|FAST|Aggressive correction|
* Speed profiles map to existing advisor parameters
({{{}balancingBandwidthFraction{}}}, {{{}datanodes.involved.max.percentage{}}},
{{{}balancing.iteration.interval{}}})
* Logs the selected profile, drift value, and timestamp
h3. Stage 2: Execution
*Step 2.1 — Trigger*
* Runs {{ozone admin containerbalancer recommend --profile <speed>}} to
generate config
* Runs {{ozone admin containerbalancer start --auto-config --profile <speed>}}
* Records start timestamp, initial drift, selected profile, and full
recommended config
*Step 2.2 — Monitor to Completion*
* Polls {{ozone admin containerbalancer status --verbose}} every 2 minutes
* Each poll captures: iteration number, bytes moved (cumulative + delta),
bytes remaining, containers moved/scheduled/timed-out/failed, datanodes
involved etc
* Detects three terminal states:
** *Completed* — balancer reports cluster balanced
** *Stalled* — zero progress for N consecutive polls (configurable, default 3)
** *Timed out* — wall-clock exceeds maximum (configurable, default 48h)
* On stall or timeout: stops the balancer and records the reason
h3. Stage 3: Statistics & Observability
*Step 3.1 — Benchmark Report*
Produces a structured report covering:
* Preparation: assessment count, drift at trigger, bytes to move,
source/target node counts
* Execution: duration, iterations, bytes moved, containers moved,
success/timeout/failure rates
* Result: drift before vs. after, SLA pass/fail
* System impact: peak resource utilization during the run vs. baseline
*Step 3.2 — System Impact Metrics*
Captures before/during/after snapshots via Prometheus queries. Below are a few
examples, please discover more concrete metrics.
||Level||Metrics||
|DataNode|Java heap related metrics(usage, GC pause time etc), CPU utilization,
I/O wait, disk throughput, disk IOPS system resources etc. Also, include DN
metrics to understand the impact of balancer execution on it|
|SCM|RPC latency, heartbeat processing time, FCR load, Java heap related
metrics(usage, GC pause time etc). Also, include SCM metrics to understand the
impact of balancer execution on it|
|Replication Manager|Inflight replication/deletion etc.|
*Step 3.3 — Prometheus Alert Thresholds*
The script ships with recommended thresholds for monitoring during and after
balancer runs. Below is an example. Need to discover insights, notifications
and alerts thresholds.
||Metric||Warning||Critical||
|{{ozone_cluster_utilization_drift_percent}}|> 10% for 10hrs|> 20% for 5hrs|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
