ZhenyuLi created HDFS-18026:
-------------------------------
Summary: Stop excluding slow disks after
dfs.datanode.max.slowdisks.to.exclude is reconfigured to 0
Key: HDFS-18026
URL: https://issues.apache.org/jira/browse/HDFS-18026
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.4.0
Reporter: ZhenyuLi
h3. Problem
{\{dfs.datanode.max.slowdisks.to.exclude}} (added in HDFS-16371) is
reconfigurable at runtime, and hdfs-default.xml documents that the value 0
"disables excluding slow disk when choosing volume". However, after
reconfiguring it from a positive value to 0, the DataNode keeps excluding the
slow disks that were published before the change, until the DataNode is
restarted.
h3. Root cause
{\{DataNodeDiskMetrics}} keeps the limit (\{{maxSlowDisksToExclude}}) and the
published exclusion list (\{{slowDisksToExclude}}) as two separate fields:
* \{{setMaxSlowDisksToExclude()}}, called from
\{{DataNode#reconfSlowDiskParameters}}, only updates the limit; it does not
touch the list.
* The detector thread only assigns \{{slowDisksToExclude}} inside \{{if
(maxSlowDisksToExclude > 0)}}, so while the limit is 0 the old list is never
cleared.
* \{{FsVolumeList#chooseVolume}} filters candidate volumes by
\{{getSlowDisksToExclude()}} without checking the limit.
h3. Impact
* Setting the value to 0 has no effect at runtime; previously excluded disks
keep being skipped for new blocks.
* If the stale list covers every volume of the requested storage type (e.g. the
only SSD volume of a DataNode, written with the ALL_SSD policy),
\{{RoundRobinVolumeChoosingPolicy}} throws \{{DiskOutOfSpaceException("No more
available volumes")}}, so that DataNode rejects writes for that storage type
even though exclusion is supposed to be disabled.
h3. Steps to reproduce
# Enable disk metrics (\{{dfs.datanode.fileio.profiling.sampling.percentage}} >
0) and set \{{dfs.datanode.max.slowdisks.to.exclude}} to 1.
# Wait until a slow disk is detected and
\{{DataNodeDiskMetrics#getSlowDisksToExclude()}} is non-empty.
# Change the value to 0 and run \{{hdfs dfsadmin -reconfig datanode
<host:ipc_port> start}}.
# \{{getSlowDisksToExclude()}} still returns the old disk, and new blocks are
never placed on it.
h3. Proposed fix
Make \{{getSlowDisksToExclude()}} return an empty list whenever the limit is
not positive, so disabling takes effect immediately and does not race with the
detector thread. Also clear the list in the detector thread in that case, and
make the field \{{volatile}} since it is written and read by different threads.
A unit test in \{{TestFsVolumeList}} covers the reconfiguration.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]