[
https://issues.apache.org/jira/browse/HDFS-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Yun reassigned HDFS-14627:
-------------------------------
Assignee: Yang Yun
> Improvements to make slow archive storage works on HDFS
> -------------------------------------------------------
>
> Key: HDFS-14627
> URL: https://issues.apache.org/jira/browse/HDFS-14627
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Yang Yun
> Assignee: Yang Yun
> Priority: Minor
> Attachments: HDFS-14627.patch,
> data_flow_between_datanode_and_aws_s3.jpg
>
>
> In our setup, we mount archival storage from remote. the write speed is about
> 20M/Sec, the read speed is about 40M/Sec, and the normal file operations, for
> example 'ls', are time consuming.
> we add some improvements to make this kind of archive storage works in
> currrent hdfs system.
> 1. Add multiply to read/write timeout if block saved on archive storage.
> 2. Save replica cache file of archive storage to other fast disk for quick
> restart datanode, shutdownHook may does not execute if the saving takes too
> long time.
> 3. Check mount file system before using mounted archive storage.
> 4. Reduce or avoid call DF during generating heartbeat report for archive
> storage.
> 5. Add option to skip archive block during decommission.
> 6. Use multi-threads to scan archive storage.
> 7. Check archive storage error with retry times.
> 8. Add option to disable scan block on archive storage.
> 9. Sleep a heartBeat time if there are too many difference when call
> checkAndUpdate in DirectoryScanner
> 10. An auto-service to scan fsimage and set the storage policy of files
> according to policy.
> 11. An auto-service to call mover to move the blocks to right storage.
> 12. Dedup files on remote storage if the storage is reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
