Wei-Chiu Chuang created HDDS-15030:
--------------------------------------
Summary: [Website v2][Troubleshooting] Datanode running out of
space
Key: HDDS-15030
URL: https://issues.apache.org/jira/browse/HDDS-15030
Project: Apache Ozone
Issue Type: Sub-task
Components: documentation
Reporter: Wei-Chiu Chuang
a cluster will not function well if it reaches storage limit.
SCM does not assign new writes to Datanodes near storage limit. If all
Datanodes are out of capacity, the cluster will appear slow, because it is
unable to quickly assign a Datanode to store the data.
{noformat}
<property>
<name>hdds.datanode.storage.utilization.warning.threshold</name>
<value>0.75</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, a warning will be logged while processing the nodeReport in SCM.
</description>
</property>
<property>
<name>hdds.datanode.storage.utilization.critical.threshold</name>
<value>0.95</value>
<tag>OZONE, SCM, MANAGEMENT</tag>
<description>
If a datanode overall storage utilization exceeds more than this
value, the datanode will be marked out of space.
</description>
</property>
{noformat}
If a Datanode runs out of capacity, it can add more disks (data volumes) to the
node. However, because new writes will mostly go to the new disk, it creates
imbalance as more traffic will be served on the new disk. Administrators should
run disk balancer to alleviate this problem.
{noformat}
<property>
<name>hdds.datanode.volume.choosing.policy</name>
<value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
<tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
<description>
The class name of the policy for choosing volumes in the list of
directories.
Defaults to
org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
This volume choosing policy randomly chooses two volumes with remaining
space and then picks the one with lower utilization.
</description>
</property>
{noformat}
Similarly, the Datanode ratis will fail to write more Ratis transaction logs if
there's no sufficient space available. Reserve at least 1GB free space for
Ratis.
{noformat}
<property>
<name>ozone.scm.datanode.ratis.volume.free-space.min</name>
<value>1GB</value>
<tag>OZONE, DATANODE</tag>
<description>Minimum amount of storage space required for each ratis
volume on a datanode to hold a new pipeline.
Datanodes with all its ratis volumes with space under this value
will not be allocated a pipeline or container replica.
</description>
</property>
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
