[
https://issues.apache.org/jira/browse/HDFS-17505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZanderXu resolved HDFS-17505.
-----------------------------
Resolution: Fixed
> [FGL] Solutions for StoragePolicy
> ---------------------------------
>
> Key: HDFS-17505
> URL: https://issues.apache.org/jira/browse/HDFS-17505
> Project: Hadoop HDFS
> Issue Type: Task
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
>
> StoragePolicy may be set on an ancestor path and used by some children files.
>
> This storage policy are used by many operations, but there are two special
> operations:
> * ChooseExcessRedundancies
> * BlockReconstructionWork
>
> ChooseExcessRedundancies is used to process some excess redundancy replicas.
> BR and IBR may need to process some excess redundancy replicas.
> It means that :
> * BR and IBR needs to hold not only the global BM write lock but also the
> global FS write lock in phase 2.
> * BR and IBR needs to hold not only the write lock of last iNode file but
> also the full directory tree locks in phase 3.
>
> It will results in a low performance.
>
> In order to fix it, there are some solutions.
> Solution 1:
> * Using a separate thread to process these excess redundancy blocks
> periodically, such as the Redundancy monitor.
> * This thread can hold the full directory tree locks to process these blocks
> in phase 3.
>
> Solution 2:(preferred)
> * Try to set the storage policy to each iNodeFile, so that BM related
> operations can only involve the iNodeFile to which the processing block
> belongs.
> * It's solution will not cost more memory.
> * The storage policy can be set synchronously to all children iNodeFiles.
> setStoragePolicy, unsetStoragePolicy and rename require this operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
