[ 
https://issues.apache.org/jira/browse/HDFS-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen O'Donnell resolved HDFS-18026.
--------------------------------------
    Fix Version/s: 3.6.0
       Resolution: Fixed

> Stop excluding slow disks after dfs.datanode.max.slowdisks.to.exclude is 
> reconfigured to 0
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-18026
>                 URL: https://issues.apache.org/jira/browse/HDFS-18026
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.4.0
>            Reporter: ZhenyuLi
>            Assignee: ZhenyuLi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.6.0
>
>
> h3. Problem
> {\{dfs.datanode.max.slowdisks.to.exclude}} (added in HDFS-16371) is 
> reconfigurable at runtime, and hdfs-default.xml documents that the value 0 
> "disables excluding slow disk when choosing volume". However, after 
> reconfiguring it from a positive value to 0, the DataNode keeps excluding the 
> slow disks that were published before the change, until the DataNode is 
> restarted.
> h3. Root cause
> {\{DataNodeDiskMetrics}} keeps the limit (\{{maxSlowDisksToExclude}}) and the 
> published exclusion list (\{{slowDisksToExclude}}) as two separate fields:
> * \{{setMaxSlowDisksToExclude()}}, called from 
> \{{DataNode#reconfSlowDiskParameters}}, only updates the limit; it does not 
> touch the list.
> * The detector thread only assigns \{{slowDisksToExclude}} inside \{{if 
> (maxSlowDisksToExclude > 0)}}, so while the limit is 0 the old list is never 
> cleared.
> * \{{FsVolumeList#chooseVolume}} filters candidate volumes by 
> \{{getSlowDisksToExclude()}} without checking the limit.
> h3. Impact
> * Setting the value to 0 has no effect at runtime; previously excluded disks 
> keep being skipped for new blocks.
> * If the stale list covers every volume of the requested storage type (e.g. 
> the only SSD volume of a DataNode, written with the ALL_SSD policy), 
> \{{RoundRobinVolumeChoosingPolicy}} throws \{{DiskOutOfSpaceException("No 
> more available volumes")}}, so that DataNode rejects writes for that storage 
> type even though exclusion is supposed to be disabled.
> h3. Steps to reproduce
> # Enable disk metrics (\{{dfs.datanode.fileio.profiling.sampling.percentage}} 
> > 0) and set \{{dfs.datanode.max.slowdisks.to.exclude}} to 1.
> # Wait until a slow disk is detected and 
> \{{DataNodeDiskMetrics#getSlowDisksToExclude()}} is non-empty.
> # Change the value to 0 and run \{{hdfs dfsadmin -reconfig datanode 
> <host:ipc_port> start}}.
> # \{{getSlowDisksToExclude()}} still returns the old disk, and new blocks are 
> never placed on it.
> h3. Proposed fix
> Make \{{getSlowDisksToExclude()}} return an empty list whenever the limit is 
> not positive, so disabling takes effect immediately and does not race with 
> the detector thread. Also clear the list in the detector thread in that case, 
> and make the field \{{volatile}} since it is written and read by different 
> threads. A unit test in \{{TestFsVolumeList}} covers the reconfiguration.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to