Andrey Yarovoy created HDDS-17611:
-------------------------------------
Summary: size the per-datanode RATIS/THREE pipeline limit from the
number of data volumes
Key: HDDS-17611
URL: https://issues.apache.org/jira/browse/HDDS-17611
Project: Apache Ozone
Issue Type: Improvement
Components: SCM
Reporter: Andrey Yarovoy
## Description
**Today** (`SCMNodeManager.pipelineLimit`):
- If `ozone.scm.datanode.pipeline.limit` > 0 (default **2**), that's the limit
for every datanode.
- Only if it is set to 0, the limit is `ozone.scm.pipeline.per.metadata.disk`
(default 2) × the number of
**metadata** volumes the datanode reports, that is, its Raft log directories.
Most datanodes have exactly one.
So the default is 2 pipelines per datanode whatever its size, and the
alternative gives 2 as well on the usual layout
of one Raft log SSD. A datanode with 16 HDDs gets as many pipelines as one with
a single disk, unless an operator
sets the limit by hand. Our cluster, for example, had
`ozone.scm.datanode.pipeline.limit = 20` set by its
deployment tooling, not derived from anything.
The open-container count per pipeline already scales with the data volumes
(HDDS-3700:
`1 + (ozone.scm.pipeline.owner.container.count × healthy volumes − 1) /
pipelineLimit`). The pipeline count is the
one piece that doesn't.
**Proposal:**
- **A new setting,** `ozone.scm.pipeline.per.data.volume` (a decimal, default 0
= off). When it is > 0 and
`ozone.scm.datanode.pipeline.limit` is 0, the limit is `max(1,
round(per.data.volume × healthy data volumes))`.
- **Keep the metadata-volume rule** as it is when `per.data.volume` is 0, for
compatibility.
- **Document how to pick the value** (HDDS-15246): the Raft log disk's capacity
for many concurrent groups, datanode
memory per group, and the recovery time after a datanode failure all grow with
the number of pipelines.
- **Possibly, as a follow-up,** make the volume-based limit the default
(HDDS-9516).
Because healthy volumes are re-reported, a datanode that loses disks would get
a lower limit. Existing pipelines above
the new limit stay until they close; no forced teardown.
**Evidence (12 datanodes, 16 HDDs + 2 SSDs each, Ozone 2.2.0, TestDFSIO 3-way
writes):**
| per-datanode limit | pipelines (per datanode) | fitted 3-way write rate |
largest write (1000 × 1 GB) |
|---|---|---|---|
| 20 (1.25 per data volume) | 71 (5–20) | 6.9 GB/s | 5.9 GB/s |
| 40 (2.5 per data volume) | 154 (28–40) | 9.6 GB/s | 7.8 GB/s |
Part of that gain is the layout: at 20, two datanodes were stranded at 5 and 8
pipelines, see HDDS-6225. After
separating it out, each datanode still moved about 12% more data at 40. The
data HDDs went from about 66% to 80% busy
on the largest write, so more pipelines per datanode put the disks to better
use. We didn't test a value above 40.
**Tests:**
- `SCMNodeManager.pipelineLimit` with the new setting: 0, 1, many volumes;
volume failure lowers the limit.
- The fallback order: `datanode.pipeline.limit` > 0 wins, then
`per.data.volume`, then `per.metadata.disk`.
- `openContainerLimit` stays consistent with the new limit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
