Repository: hadoop Updated Branches: refs/heads/branch-3.1 b6bc0f409 -> 77dd45646 refs/heads/trunk 987d8191a -> 96892c469
HDFS-13237. [Documentation] RBF: Mount points across multiple subclusters. Contributed Ãñigo Goiri Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/96892c46 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/96892c46 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/96892c46 Branch: refs/heads/trunk Commit: 96892c469b16c5aaff1b7c42f66f820344256bc2 Parents: 987d819 Author: Brahma Reddy Battula <bra...@apache.org> Authored: Tue Sep 11 10:12:34 2018 +0530 Committer: Brahma Reddy Battula <bra...@apache.org> Committed: Tue Sep 11 10:12:34 2018 +0530 ---------------------------------------------------------------------- .../src/site/markdown/HDFSRouterFederation.md | 26 ++++++++++++++++++++ 1 file changed, 26 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/96892c46/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md index 2f49587..edc9918 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md @@ -214,6 +214,7 @@ Mount table permission can be set by following command: The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755. +#### Quotas Router-based federation supports global quota at mount table level. Mount table entries may spread multiple subclusters and the global quota will be accounted across these subclusters. @@ -229,6 +230,31 @@ Ls command will show below information for each mount table entry: Source Destinations Owner Group Mode Quota/Usage /path ns0->/path root supergroup rwxr-xr-x [NsQuota: 50/0, SsQuota: 100 B/0 B] +#### Multiple subclusters +A mount point also supports mapping multiple subclusters. +For example, to create a mount point that stores files in subclusters `ns1` and `ns2`. + + [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -add /data ns1,ns2 /data -order SPACE + +When listing `/data`, it will show all the folders and files in both subclusters. +For deciding where to create a new file/folder it uses the order parameter, it currently supports the following methods: + +* HASH: Follow consistent hashing in the first level. Deeper levels will be in the one of the parent. +* LOCAL: Try to write data in the local subcluster. +* RANDOM: Random subcluster. This is usually useful for balancing the load across. Folders are created in all subclusters. +* HASH_ALL: Follow consistent hashing at all the levels. This approach tries to balance the reads and writes evenly across subclusters. Folders are created in all subclusters. +* SPACE: Try to write data in the subcluster with the most available space. Folders are created in all subclusters. + +For the hash-based approaches, the difference is that HASH would make all the files/folders within a folder belong to the same subcluster while HASH_ALL will spread all files under a mount point. +For example, assuming we have a HASH mount point for `/data/hash`, files and folders under `/data/hash/folder0` will all be in the same subcluster. +On the other hand, a HASH_ALL mount point for `/data/hash_all`, will spread files under `/data/hash_all/folder0` across all the subclusters for that mount point (subfolders will be created to all subclusters). + +RANDOM can be used for reading and writing data from/into different subclusters. +The common use for this approach is to have the same data in multiple subclusters and balance the reads across subclusters. +For example, if thousands of containers need to read the same data (e.g., a library), one can use RANDOM to read the data from any of the subclusters. + +Note that consistency of the data across subclusters is not guaranteed by the Router. + ### Disabling nameservices To prevent accessing a nameservice (sublcuster), it can be disabled from the federation. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org