[
https://issues.apache.org/jira/browse/HDFS-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDFS-18026.
--------------------------------------
Fix Version/s: 3.6.0
Resolution: Fixed
> Stop excluding slow disks after dfs.datanode.max.slowdisks.to.exclude is
> reconfigured to 0
> ------------------------------------------------------------------------------------------
>
> Key: HDFS-18026
> URL: https://issues.apache.org/jira/browse/HDFS-18026
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: ZhenyuLi
> Assignee: ZhenyuLi
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
>
> h3. Problem
> {\{dfs.datanode.max.slowdisks.to.exclude}} (added in HDFS-16371) is
> reconfigurable at runtime, and hdfs-default.xml documents that the value 0
> "disables excluding slow disk when choosing volume". However, after
> reconfiguring it from a positive value to 0, the DataNode keeps excluding the
> slow disks that were published before the change, until the DataNode is
> restarted.
> h3. Root cause
> {\{DataNodeDiskMetrics}} keeps the limit (\{{maxSlowDisksToExclude}}) and the
> published exclusion list (\{{slowDisksToExclude}}) as two separate fields:
> * \{{setMaxSlowDisksToExclude()}}, called from
> \{{DataNode#reconfSlowDiskParameters}}, only updates the limit; it does not
> touch the list.
> * The detector thread only assigns \{{slowDisksToExclude}} inside \{{if
> (maxSlowDisksToExclude > 0)}}, so while the limit is 0 the old list is never
> cleared.
> * \{{FsVolumeList#chooseVolume}} filters candidate volumes by
> \{{getSlowDisksToExclude()}} without checking the limit.
> h3. Impact
> * Setting the value to 0 has no effect at runtime; previously excluded disks
> keep being skipped for new blocks.
> * If the stale list covers every volume of the requested storage type (e.g.
> the only SSD volume of a DataNode, written with the ALL_SSD policy),
> \{{RoundRobinVolumeChoosingPolicy}} throws \{{DiskOutOfSpaceException("No
> more available volumes")}}, so that DataNode rejects writes for that storage
> type even though exclusion is supposed to be disabled.
> h3. Steps to reproduce
> # Enable disk metrics (\{{dfs.datanode.fileio.profiling.sampling.percentage}}
> > 0) and set \{{dfs.datanode.max.slowdisks.to.exclude}} to 1.
> # Wait until a slow disk is detected and
> \{{DataNodeDiskMetrics#getSlowDisksToExclude()}} is non-empty.
> # Change the value to 0 and run \{{hdfs dfsadmin -reconfig datanode
> <host:ipc_port> start}}.
> # \{{getSlowDisksToExclude()}} still returns the old disk, and new blocks are
> never placed on it.
> h3. Proposed fix
> Make \{{getSlowDisksToExclude()}} return an empty list whenever the limit is
> not positive, so disabling takes effect immediately and does not race with
> the detector thread. Also clear the list in the detector thread in that case,
> and make the field \{{volatile}} since it is written and read by different
> threads. A unit test in \{{TestFsVolumeList}} covers the reconfiguration.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]