[
https://issues.apache.org/jira/browse/HDFS-15714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342396#comment-17342396
]
Feilong He commented on HDFS-15714:
-----------------------------------
[~bpatel] Thanks for your comments!
In _ReadMountManager#prepareMount_, _FSMountAttrOp.getRemotePaths_ is used to
get remote metadata for both remote path being mounted and its children.
_FSTreeWalk_ (see its constructor) will instantiate a _FileSystem_ according to
remote mount path url. For S3 url, it's _S3AFileSystem_. And by digging more in
S3AFileSystem, you can find S3 client is employed to get file status in
mounting phase. And with metadata wrapped in remote file status, like
_modification time, access time, permission, etc_, HDFS will create its INode
file accordingly and set Provided Storage type.
Any comment is welcome!
> HDFS Provided Storage Read/Write Mount Support On-the-fly
> ---------------------------------------------------------
>
> Key: HDFS-15714
> URL: https://issues.apache.org/jira/browse/HDFS-15714
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: 3.4.0
> Reporter: Feilong He
> Assignee: Feilong He
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-15714-01.patch,
> HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems.
> In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through
> configuring external storage with PROVIDED tag for DataNode, user can enable
> application to access data stored externally from HDFS side. However, there
> are two issues need to be addressed. Firstly, mounting external storage
> on-the-fly, namely dynamic mount, is lacking. It is necessary to get it
> supported to flexibly combine HDFS with an external storage at runtime.
> Secondly, PS write is not supported by current HDFS. But in real
> applications, it is common to transfer data bi-directionally for read/write
> between HDFS and external storage.
> Through this JIRA, we are presenting our work for PS write support and
> dynamic mount support for both read & write. Please note in the community
> several JIRAs have been filed for these topics. Our work is based on these
> previous community work, with new design & implementation to support called
> writeBack mount and enable admin to add any mount on-the-fly. We appreciate
> those folks in the community for their great contribution! See their pending
> JIRAs: HDFS-14805 & HDFS-12090.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
