This is an automated email from the ASF dual-hosted git repository.
umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new e7cead1 HDFS-15330. Document the ViewFSOverloadScheme details in
ViewFS guide. Contributed by Uma Maheswara Rao G.
e7cead1 is described below
commit e7cead114314412d461ed1c4b28ba33726c65b9b
Author: Uma Maheswara Rao G <umamah...@apache.org>
AuthorDate: Fri Jun 5 10:58:21 2020 -0700
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide.
Contributed by Uma Maheswara Rao G.
(cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
(cherry picked from commit 418580446b65be3a0674762e76fc2cb9a1e5629a)
---
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 40 ++++-
.../hadoop-hdfs/src/site/markdown/ViewFs.md | 6 +
.../src/site/markdown/ViewFsOverloadScheme.md | 163 +++++++++++++++++++++
.../site/resources/images/ViewFSOverloadScheme.png | Bin 0 -> 190004 bytes
hadoop-project/src/site/site.xml | 1 +
5 files changed, 209 insertions(+), 1 deletion(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index 306884c..32e1a7b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -672,4 +672,42 @@ Usage: `hdfs debug recoverLease -path <path> [-retries
<num-retries>]`
| [`-path` *path*] | HDFS path for which to recover the lease. |
| [`-retries` *num-retries*] | Number of times the client will retry calling
recoverLease. The default number of retries is 1. |
-Recover the lease on the specified path. The path must reside on an HDFS
filesystem. The default number of retries is 1.
+Recover the lease on the specified path. The path must reside on an HDFS file
system. The default number of retries is 1.
+
+dfsadmin with ViewFsOverloadScheme
+----------------------------------
+
+Usage: `hdfs dfsadmin -fs <child fs mount link URI> <dfsadmin command options>`
+
+| COMMAND\_OPTION | Description |
+|:---- |:---- |
+| `-fs` *child fs mount link URI* | Its a logical mount link path to child
file system in ViewFS world. This uri typically formed as src mount link
prefixed with fs.defaultFS. Please note, this is not an actual child file
system uri, instead its a logical mount link uri pointing to actual child file
system|
+
+Example command usage:
+ `hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
+
+In ViewFsOverloadScheme, we may have multiple child file systems as mount
point mappings as shown in [ViewFsOverloadScheme
Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic
parameter supported by dfsadmin. When users want to execute commands on one of
the child file system, they need to pass that file system mount mapping link
uri to -fs option. Let's take an example mount link configuration and dfsadmin
command below.
+
+Mount link:
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://MyCluster1</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.MyCluster1./user</name>
+ <value>hdfs://MyCluster2/user</value>
+ <!-- mount table name : MyCluster1
+ mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user
+ mount link path: /user
+ mount link uri: hdfs://MyCluster1/user
+ mount target uri for /user: hdfs://MyCluster2/user -->
+</property>
+```
+
+If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option
(`-fs hdfs://MyCluster1/user`)
+Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve
the passed (`-fs hdfs://MyCluster1/user`) to target fs
(`hdfs://MyCluster2/user`).
+This way users can get the access to all hdfs child file systems in
ViewFsOverloadScheme.
+If there is no `-fs` option provided, then it will try to connect to the
configured fs.defaultFS cluster if a cluster running with the fs.defaultFS uri.
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
index f851ef6..52ad49c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
@@ -361,6 +361,12 @@ resume its work, it's a good idea to provision some sort
of cron job to purge su
Delegation tokens for the cluster to which you are submitting the job
(including all mounted volumes for that cluster’s mount table), and for input
and output paths to your map-reduce job (including all volumes mounted via
mount tables for the specified input and output paths) are all handled
automatically. In addition, there is a way to add additional delegation tokens
to the base cluster configuration for special circumstances.
+Don't want to change scheme or difficult to copy mount-table configurations to
all clients?
+-------------------------------------------------------------------------------------------
+
+Please refer to the [View File System Overload Scheme
Guide](./ViewFsOverloadScheme.html)
+
+
Appendix: A Mount Table Configuration Example
---------------------------------------------
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
new file mode 100644
index 0000000..5fd8633
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
@@ -0,0 +1,163 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+View File System Overload Scheme Guide
+======================================
+
+<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
+
+Introduction
+------------
+
+The View File System Overload Scheme introduced to solve two key challenges
with the View File System(ViewFS). The first problem is, to use ViewFS, users
need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second
problem is that users need to copy the mount-table configurations to all the
client nodes.
+The ViewFileSystemOverloadScheme is addressing these challenges.
+
+View File System Overload Scheme
+--------------------------------
+
+### Details
+
+The View File System Overload Scheme is an extension to the View File System.
This will allow users to continue to use their existing fs.defaultFS configured
scheme or any new scheme name instead of using scheme `viewfs`. Mount link
configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html).
If a user wants to continue use the same fs.defaultFS and wants to have more
mount points, then mount link configurations should have the current
fs.defaultFS authority name as mo [...]
+
+Another important improvement with the ViewFileSystemOverloadScheme is,
administrators need not copy the `mount-table.xml` configuration file to 1000s
of client nodes. Instead they can keep the mount-table configuration file in a
Hadoop compatible file system. So, keeping the configuration file in a central
place makes administrators life easier as they can update mount-table in single
place.
+
+### Enabling View File System Overload Scheme
+
+To use this class, the following configurations needed to be added in
core-site.xml file.
+
+```xml
+<property>
+ <name>fs.<scheme>.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS.
For example if fs.defaultFS was configured with `hdfs://mycluster`, then the
above configuration would be like below:
+
+```xml
+<property>
+ <name>fs.hdfs.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+
+### Example Configurations
+
+**Example 1:**
+
+If users want some of their existing cluster (`hdfs://mycluster`) data to
mount with hdfs(`hdfs://mycluster`) and other object store
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.Cluster./user</name>
+ <value>hdfs://mycluster/user</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./data</name>
+ <value>o3fs://bucket1.volume1/data</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./backup</name>
+ <value>s3a://bucket1/backup/</value>
+</property>
+```
+
+Let's consider the following operations to understand where these operations
will be delegated based on mount links.
+
+ *Op1:* Create a file with the the path `hdfs://mycluster/user/fileA`, then
physically this file will be created at `hdfs://mycluster/user/fileA`. This
delegation happened based on the first configuration parameter in above
configurations. Here `/user` mapped to `hdfs://mycluster/user/`.
+
+ *Op2:* Create a file the the path `hdfs://mycluster/data/datafile`, then
this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`.
This delegation happened based on second configurations parameter in above
configurations. Here `/data` was mapped with
`o3fs://bucket1.volume1.omhost/data/`.
+
+ *Op3:* Create a file with the the path `hdfs://Cluster/backup/data.zip`,
then physically this file will be created at `s3a://bucket1/backup/data.zip`.
This delegation happened based on the third configuration parameter in above
configurations. Here `/backup` was mapped to `s3a://bucket1/backup/`.
+
+
+**Example 2:**
+
+If users want some of their existing cluster (`s3a://bucketA/`) data to mount
with other hdfs cluster(`hdfs://Cluster`) and object store
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucketA/`), the following
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.bucketA./user</name>
+ <value>hdfs://Cluster/user</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.bucketA./data</name>
+ <value>o3fs://bucket1.volume1.omhost/data</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.bucketA./salesDB</name>
+ <value>s3a://bucketA/salesDB/</value>
+</property>
+```
+Let's consider the following operations to understand to where these
operations will be delegated based on mount links.
+
+ *Op1:* Create a file with the the path `s3a://bucketA/user/fileA`, then this
file will be created physically at `hdfs://Cluster/user/fileA`. This delegation
happened based on the first configuration parameter in above configurations.
Here `/user` mapped to `hdfs://Cluster/user`.
+
+ *Op2:* Create a file the the path `s3a://bucketA/data/datafile`, then this
file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This
delegation happened based on second configurations parameter in above
configurations. Here `/data` was mapped with
`o3fs://bucket1.volume1.omhost/data/`.
+
+ *Op3:* Create a file with the the path `s3a://bucketA/salesDB/dbfile`, then
physically this file will be created at `s3a://bucketA/salesDB/dbfile`. This
delegation happened based on the third configuration parameter in above
configurations. Here `/salesDB` was mapped to `s3a://bucket1/salesDB`.
+
+Note: In above examples we used create operation only, but the same mechanism
applies to any other file system APIs here.
+
+The following picture shows how the different schemes can be used in
ViewFileSystemOverloadScheme compared to the ViewFileSystem.
+
+<img src="./images/ViewFSOverloadScheme.png" width="1050" height="550"/>
+
+### Central Mount Table Configurations
+
+To enable central mount table configuration, we need to configure
`fs.viewfs.mounttable.path` in `core-site.xml` with the value as the Hadoop
compatible file system directory/file path, where the
`mount-table-<versionNumber>.xml` file copied. Here versionNumber is an integer
number and need to increase the version number and upload new file in same
directory.
+
+The ViewFileSystemOverloadScheme always loads the highest version number
`mount-table-<versionNumber>.xml`. Please don't replace the file with same
name. Always increment the version number to take new file picked by newly
initializing clients. Why we don't recommend to replace the files is that, some
client might have already opened the connections to old mount-table files
already and in middle of loading configuration files, and replacing files can
make them fail.
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.path</name>
+ <value>hdfs://Cluster/config/mount-table-dir</value>
+</property>
+```
+ If you are sure, you will never do updates to mount-table file, you can also
configure file path directly like below. If you configure file path, it will
not check any highest version number loading. Whatever file configured it will
be loaded. However file name format should be same.
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.path</name>
+
<value>hdfs://Cluster/config/mount-table-dir/mount-table-<versionNumber>.xml</value>
+</property>
+```
+Note: we recommend not to configure mount-links in `core-site.xml` if you
configure above valid path. Otherwise both mount links will be mixed and can
lead to a confused behavior.
+
+If you copy the `mount-table-<versionNumber>.xml`, you may consider having big
replication factor depending on your cluster size. So, that file will be
available locally to majority of clients as applications(MR/YARN/HBASE..etc)
use locality on HDFS when reading `mount-table-<versionNumber>.xml`.
+
+DFSAdmin commands with View File System Overload Scheme
+-------------------------------------------------------
+
+Please refer to the [HDFSCommands
Guide](./HDFSCommands.html#dfsadmin_with_ViewFsOverloadScheme)
+
+Appendix: A Mount Table Configuration with XInclude
+---------------------------------------------------
+
+If users have a HTTP server in trusted network and don't need authentication
mechanism to it, you can also place your mount-table.xml file in that server
and configure
+ XInclude xml tag with `mount-table.xml` file.
+
+```xml
+<configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
+ <xi:include href="http://myserver/mountTable/mountTable.xml"; />
+</configuration>
+```
+
+The Apache Hadoop configuration has the capability to read the http urls from
XInclude and load into configurations. If you choose this option, please don't
configure mount-table configuration items in `core-site.xml` or at
`fs.viewfs.mounttable.path`. Please note, Hadoop configuration XInclude does
not use SPNego authentication when opening url. So, this will not work if http
server where you placed `mount-table.xml` needs authentication.
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
b/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
new file mode 100644
index 0000000..b5029c1
Binary files /dev/null and
b/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
differ
diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index 36f2e0d..ba08ef9 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -83,6 +83,7 @@
<item name="Observer NameNode"
href="hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html"/>
<item name="Federation"
href="hadoop-project-dist/hadoop-hdfs/Federation.html"/>
<item name="ViewFs" href="hadoop-project-dist/hadoop-hdfs/ViewFs.html"/>
+ <item name="ViewFsOverloadScheme"
href="hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html"/>
<item name="Snapshots"
href="hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html"/>
<item name="Edits Viewer"
href="hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html"/>
<item name="Image Viewer"
href="hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html"/>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
