[
https://issues.apache.org/jira/browse/HDDS-15014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075218#comment-18075218
]
Wei-Chiu Chuang commented on HDDS-15014:
----------------------------------------
This implementation plan outlines the transition from single-source replication
to multi-source reconstruction for EC container decommission, as described
in HDDS-15014. The plan focuses on SCM-side dynamic switching and
Datanode-side disk-level fairness.
Phase 1: SCM Configuration and Global Capacity
We will introduce new configuration properties in
ReplicationManagerConfiguration to control the behavior and protect cluster
resources.
1. New Configuration Keys:
* hdds.scm.replication.decommission.ec.reconstruction.enabled (Boolean,
default: false): Feature flag to enable/disable the switch to reconstruction
during decommission.
* hdds.scm.replication.decommission.ec.reconstruction.load.factor
(Double, default: 0.9): The threshold of a node's replication limit at which SCM
switches to reconstruction.
* hdds.scm.replication.reconstruction.global.limit (Int, default: 50):
Cluster-wide cap on concurrent reconstruction tasks to prevent aggregate
network saturation.
2. Global Throttling Implementation:
* Implement an atomic counter in ReplicationManager to track active
ReconstructECContainersCommand tasks.
* The SCM will skip scheduling new reconstruction tasks if this global
limit is reached, falling back to replication or re-queuing the container.
Phase 2: SCM Logic - The Dynamic Switch
The SCM will monitor the load on decommissioning Datanodes and dynamically
shift to reconstruction to offload the source node.
1. Load Factor Calculation:
* SCM will calculate a node's load using: $\frac{\text{queued
replication} + (\text{queued reconstruction} \times
\text{weight})}{\text{effective
replication limit}}$.
* The "effective replication limit" correctly accounts for the
hdds.datanode.replication.outofservice.limit.factor for
decommissioning/maintenance
nodes.
2. ECUnderReplicationHandler Enhancement:
* In processDecommissioningIndexes, if a source Datanode's load factor
exceeds the configured threshold (default 90%):
* Action: Generate a ReconstructECContainersCommand instead of
individual ReplicateContainerCommands.
* Optimization: When selecting source Datanodes for this
reconstruction, SCM will exclude the decommissioning node if $k$ other replicas
are
available. This transforms a node-level bottleneck into a
parallelized cluster-wide task.
Phase 3: Datanode Logic - Disk-Level Fairness
To protect physical disks from head contention (thrashing) when many
replication tasks are permitted on a single node, we will implement a
volume-aware
dispatcher in the ReplicationSupervisor.
1. Volume Tracking:
* Add an activeOutboundReplications atomic counter to HddsVolume.
* Introduce hdds.datanode.replication.volume.outbound.limit (Default: 2).
2. Volume-Aware Dispatching:
* Replace the standard PriorityBlockingQueue dispatcher with a logic
that "looks ahead" in the queue.
* When a worker thread is ready:
1. It iterates through the priority queue for the next task.
2. It resolves the task's source volume(s) using the
ContainerController.
3. If a task's source volume is at the outbound limit, the thread
skips over it (leaving it at the head of the queue) and checks the next task.
4. This allows tasks for idle volumes to "leapfrog" bottlenecked
volumes, maximizing node throughput without disk thrashing.
3. Starvation Prevention:
* By keeping skipped tasks at the head of the queue, they are guaranteed
to be evaluated first as soon as a volume slot opens or a thread becomes
available.
* A Condition wake-up will be triggered whenever a task completes and
releases a volume slot.
Phase 4: Observability and Robustness
1. SCM Metrics:
* ec_reconstruction_decommission_triggered_total: Counter for switches
triggered by the load factor.
* ec_reconstruction_global_limit_reached_total: Counter for global
reconstruction throttling.
2. Datanode Metrics:
* volume_outbound_concurrency_wait_total: Count of times a task was
skipped due to volume load.
3. Fault Tolerance:
* If reconstruction fails due to source/target issues, SCM will
automatically retry in the next cycle, re-evaluating the best strategy
(replication
vs. reconstruction) based on the latest node load.
Phase 5: Verification Strategy
1. Simulation: Decommission a dense node and verify SCM switching behavior
at the 90% load mark.
2. Disk Fairness: Stress test a single Datanode volume with 10+ replication
commands and verify that only 2 (by default) are active, while others are
correctly bypassed for tasks targeting other volumes.
3. Global Cap: Verify that the cluster-wide reconstruction limit effectively
throttles background traffic during simultaneous decommission of multiple
large nodes.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
> Speed up EC container decommission
> ----------------------------------
>
> Key: HDDS-15014
> URL: https://issues.apache.org/jira/browse/HDDS-15014
> Project: Apache Ozone
> Issue Type: Epic
> Components: SCM
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Labels: pull-request-available
>
> When Ozone decommission a datanode, it must wait until all containers are
> fully replicated to other datanodes in the cluster.
>
> In the case of EC containers, instead of reconstructing the block, it
> replicates to a destination. That replication becomes a bottleneck because it
> doesn't matter how big the cluster is, the container block is always
> replicated from the same source (the decommissioning datanode)
>
> Solution: keep doing replication, but when there are too many containers
> pending, do reconstruction instead. Reconstruction leverages other datanodes
> in the cluster so it increases parallelism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]