[jira] [Comment Edited] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[
https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214628#comment-17214628
]
Junfan Zhang edited comment on HDFS-15289 at 10/15/20, 11:50 AM:
-
Hi [~umamaheswararao] thanks for posting this. We have also implemented a
similar file system and have already applied it online to solve the problems of
hybrid-cloud architecture and cluster data migration . What I want to know is:
Does {{ViewFSOverloadScheme}} support the mounting of different cluster paths?
was (Author: zuston):
Hi [~umamaheswararao] Thanks your post. We have also implemented a similar file
system and have already applied it online to solve the problems of hybrid-cloud
architecture and cluster data migration . What I want to know is: Does
{{ViewFSOverloadScheme}} support the mounting of different cluster paths?
> Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
> -
>
> Key: HDFS-15289
> URL: https://issues.apache.org/jira/browse/HDFS-15289
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png
>
>
> ViewFS provides flexibility to mount different filesystem types with mount
> points configuration table. This approach is solving the scalability
> problems, but users need to reconfigure the filesystem to ViewFS and to its
> scheme. This will be problematic in the case of paths persisted in meta
> stores, ex: Hive. In systems like Hive, it will store uris in meta store. So,
> changing the file system scheme will create a burden to upgrade/recreate meta
> stores. In our experience many users are not ready to change that.
> Router based federation is another implementation to provide coordinated
> mount points for HDFS federation clusters. Even though this provides
> flexibility to handle mount points easily, this will not allow
> other(non-HDFS) file systems to mount. So, this does not solve the purpose
> when users want to mount external(non-HDFS) filesystems.
> So, the problem here is: Even though many users want to adapt to the scalable
> fs options available, technical challenges of changing schemes (ex: in meta
> stores) in deployments are obstructing them.
> So, we propose to allow hdfs scheme in ViewFS like client side mount system
> and provision user to create mount links without changing URI paths.
> I will upload detailed design doc shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[
https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214628#comment-17214628
]
Junfan Zhang edited comment on HDFS-15289 at 10/15/20, 11:47 AM:
-
Hi [~umamaheswararao] Thanks your post. We have also implemented a similar file
system and have already applied it online to solve the problems of hybrid-cloud
architecture and cluster data migration . What I want to know is: Does
{{ViewFSOverloadScheme}} support the mounting of different cluster paths?
was (Author: zuston):
Hi [~umamaheswararao] Thanks your post. We have also implemented a similar file
system and have already applied it online to solve the problems of multi-cloud
architecture and cluster migration . What I want to know is: Does
{{ViewFSOverloadScheme}} support the mounting of different cluster paths?
> Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
> -
>
> Key: HDFS-15289
> URL: https://issues.apache.org/jira/browse/HDFS-15289
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.2.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png
>
>
> ViewFS provides flexibility to mount different filesystem types with mount
> points configuration table. This approach is solving the scalability
> problems, but users need to reconfigure the filesystem to ViewFS and to its
> scheme. This will be problematic in the case of paths persisted in meta
> stores, ex: Hive. In systems like Hive, it will store uris in meta store. So,
> changing the file system scheme will create a burden to upgrade/recreate meta
> stores. In our experience many users are not ready to change that.
> Router based federation is another implementation to provide coordinated
> mount points for HDFS federation clusters. Even though this provides
> flexibility to handle mount points easily, this will not allow
> other(non-HDFS) file systems to mount. So, this does not solve the purpose
> when users want to mount external(non-HDFS) filesystems.
> So, the problem here is: Even though many users want to adapt to the scalable
> fs options available, technical challenges of changing schemes (ex: in meta
> stores) in deployments are obstructing them.
> So, we propose to allow hdfs scheme in ViewFS like client side mount system
> and provision user to create mount links without changing URI paths.
> I will upload detailed design doc shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
