date:20200627

[hadoop] branch branch-3.1 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new c4e0098  HDFS-15436. Default mount table name used by ViewFileSystem 
should be configurable (#2100)
c4e0098 is described below

commit c4e0098bd6054421397e33bd64239bdd13bd307b
Author: Virajith Jalaparti 
AuthorDate: Fri Jun 26 13:19:16 2020 -0700

HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable (#2100)

* HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
---
 .../org/apache/hadoop/fs/viewfs/ConfigUtil.java| 33 +++
 .../org/apache/hadoop/fs/viewfs/Constants.java | 10 +++-
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |  2 +-
 .../fs/viewfs/TestViewFsWithAuthorityLocalFs.java  |  5 +-
 .../apache/hadoop/fs/viewfs/ViewFsBaseTest.java| 10 +++-
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  2 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  | 33 +++
 .../hadoop/fs/viewfs/TestViewFileSystemHdfs.java   |  6 +-
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 68 --
 9 files changed, 141 insertions(+), 28 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
index 6dd1f65..7d29b8f 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
@@ -66,8 +66,7 @@ public class ConfigUtil {
*/
   public static void addLink(final Configuration conf, final String src,
   final URI target) {
-addLink( conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, 
-src, target);   
+addLink(conf, getDefaultMountTableName(conf), src, target);
   }
 
   /**
@@ -88,8 +87,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkMergeSlash(Configuration conf, final URI target) {
-addLinkMergeSlash(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkMergeSlash(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -110,8 +108,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkFallback(Configuration conf, final URI target) {
-addLinkFallback(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkFallback(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -132,7 +129,7 @@ public class ConfigUtil {
* @param targets
*/
   public static void addLinkMerge(Configuration conf, final URI[] targets) {
-addLinkMerge(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, targets);
+addLinkMerge(conf, getDefaultMountTableName(conf), targets);
   }
 
   /**
@@ -166,8 +163,7 @@ public class ConfigUtil {
 
   public static void addLinkNfly(final Configuration conf, final String src,
   final URI ... targets) {
-addLinkNfly(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, src, null,
-targets);
+addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
   }
 
   /**
@@ -177,8 +173,7 @@ public class ConfigUtil {
*/
   public static void setHomeDirConf(final Configuration conf,
   final String homedir) {
-setHomeDirConf(  conf,
-Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,   homedir);
+setHomeDirConf(conf, getDefaultMountTableName(conf), homedir);
   }
   
   /**
@@ -202,7 +197,7 @@ public class ConfigUtil {
* @return home dir value, null if variable is not in conf
*/
   public static String getHomeDirValue(final Configuration conf) {
-return getHomeDirValue(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE);
+return getHomeDirValue(conf, getDefaultMountTableName(conf));
   }
   
   /**
@@ -216,4 +211,18 @@ public class ConfigUtil {
 return conf.get(getConfigViewFsPrefix(mountTableName) + "." +
 Constants.CONFIG_VIEWFS_HOMEDIR);
   }
+
+  /**
+   * Get the name of the default mount table to use. If
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE_NAME_KEY} is specified,
+   * it's value is returned. Otherwise,
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE} is returned.
+   *
+   * @param conf Configuration to use.
+   * @return the name of the default mount table to use.
+   */
+  public static String

[hadoop] branch branch-3.2 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 704b273  HDFS-15436. Default mount table name used by ViewFileSystem 
should be configurable (#2100)
704b273 is described below

commit 704b273ec86d9387bfbc7b316a4880a7561198a7
Author: Virajith Jalaparti 
AuthorDate: Fri Jun 26 13:19:16 2020 -0700

HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable (#2100)

* HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
---
 .../org/apache/hadoop/fs/viewfs/ConfigUtil.java| 33 +++
 .../org/apache/hadoop/fs/viewfs/Constants.java | 10 +++-
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |  2 +-
 .../fs/viewfs/TestViewFsWithAuthorityLocalFs.java  |  5 +-
 .../apache/hadoop/fs/viewfs/ViewFsBaseTest.java| 10 +++-
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  2 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  | 33 +++
 .../hadoop/fs/viewfs/TestViewFileSystemHdfs.java   |  6 +-
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 68 --
 9 files changed, 141 insertions(+), 28 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
index 6dd1f65..7d29b8f 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
@@ -66,8 +66,7 @@ public class ConfigUtil {
*/
   public static void addLink(final Configuration conf, final String src,
   final URI target) {
-addLink( conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, 
-src, target);   
+addLink(conf, getDefaultMountTableName(conf), src, target);
   }
 
   /**
@@ -88,8 +87,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkMergeSlash(Configuration conf, final URI target) {
-addLinkMergeSlash(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkMergeSlash(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -110,8 +108,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkFallback(Configuration conf, final URI target) {
-addLinkFallback(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkFallback(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -132,7 +129,7 @@ public class ConfigUtil {
* @param targets
*/
   public static void addLinkMerge(Configuration conf, final URI[] targets) {
-addLinkMerge(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, targets);
+addLinkMerge(conf, getDefaultMountTableName(conf), targets);
   }
 
   /**
@@ -166,8 +163,7 @@ public class ConfigUtil {
 
   public static void addLinkNfly(final Configuration conf, final String src,
   final URI ... targets) {
-addLinkNfly(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, src, null,
-targets);
+addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
   }
 
   /**
@@ -177,8 +173,7 @@ public class ConfigUtil {
*/
   public static void setHomeDirConf(final Configuration conf,
   final String homedir) {
-setHomeDirConf(  conf,
-Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,   homedir);
+setHomeDirConf(conf, getDefaultMountTableName(conf), homedir);
   }
   
   /**
@@ -202,7 +197,7 @@ public class ConfigUtil {
* @return home dir value, null if variable is not in conf
*/
   public static String getHomeDirValue(final Configuration conf) {
-return getHomeDirValue(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE);
+return getHomeDirValue(conf, getDefaultMountTableName(conf));
   }
   
   /**
@@ -216,4 +211,18 @@ public class ConfigUtil {
 return conf.get(getConfigViewFsPrefix(mountTableName) + "." +
 Constants.CONFIG_VIEWFS_HOMEDIR);
   }
+
+  /**
+   * Get the name of the default mount table to use. If
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE_NAME_KEY} is specified,
+   * it's value is returned. Otherwise,
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE} is returned.
+   *
+   * @param conf Configuration to use.
+   * @return the name of the default mount table to use.
+   */
+  public static String

[hadoop] branch branch-3.3 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new ea97fe2  HDFS-15436. Default mount table name used by ViewFileSystem 
should be configurable (#2100)
ea97fe2 is described below

commit ea97fe250c7df1bfe7590b823b2b76cae8454e6b
Author: Virajith Jalaparti 
AuthorDate: Fri Jun 26 13:19:16 2020 -0700

HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable (#2100)

* HDFS-15436. Default mount table name used by ViewFileSystem should be 
configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
---
 .../org/apache/hadoop/fs/viewfs/ConfigUtil.java| 33 +++
 .../org/apache/hadoop/fs/viewfs/Constants.java | 10 +++-
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |  2 +-
 .../fs/viewfs/TestViewFsWithAuthorityLocalFs.java  |  5 +-
 .../apache/hadoop/fs/viewfs/ViewFsBaseTest.java| 10 +++-
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  2 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  | 33 +++
 .../hadoop/fs/viewfs/TestViewFileSystemHdfs.java   |  6 +-
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 68 --
 9 files changed, 141 insertions(+), 28 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
index 6dd1f65..7d29b8f 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
@@ -66,8 +66,7 @@ public class ConfigUtil {
*/
   public static void addLink(final Configuration conf, final String src,
   final URI target) {
-addLink( conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, 
-src, target);   
+addLink(conf, getDefaultMountTableName(conf), src, target);
   }
 
   /**
@@ -88,8 +87,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkMergeSlash(Configuration conf, final URI target) {
-addLinkMergeSlash(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkMergeSlash(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -110,8 +108,7 @@ public class ConfigUtil {
* @param target
*/
   public static void addLinkFallback(Configuration conf, final URI target) {
-addLinkFallback(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,
-target);
+addLinkFallback(conf, getDefaultMountTableName(conf), target);
   }
 
   /**
@@ -132,7 +129,7 @@ public class ConfigUtil {
* @param targets
*/
   public static void addLinkMerge(Configuration conf, final URI[] targets) {
-addLinkMerge(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, targets);
+addLinkMerge(conf, getDefaultMountTableName(conf), targets);
   }
 
   /**
@@ -166,8 +163,7 @@ public class ConfigUtil {
 
   public static void addLinkNfly(final Configuration conf, final String src,
   final URI ... targets) {
-addLinkNfly(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE, src, null,
-targets);
+addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
   }
 
   /**
@@ -177,8 +173,7 @@ public class ConfigUtil {
*/
   public static void setHomeDirConf(final Configuration conf,
   final String homedir) {
-setHomeDirConf(  conf,
-Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE,   homedir);
+setHomeDirConf(conf, getDefaultMountTableName(conf), homedir);
   }
   
   /**
@@ -202,7 +197,7 @@ public class ConfigUtil {
* @return home dir value, null if variable is not in conf
*/
   public static String getHomeDirValue(final Configuration conf) {
-return getHomeDirValue(conf, Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE);
+return getHomeDirValue(conf, getDefaultMountTableName(conf));
   }
   
   /**
@@ -216,4 +211,18 @@ public class ConfigUtil {
 return conf.get(getConfigViewFsPrefix(mountTableName) + "." +
 Constants.CONFIG_VIEWFS_HOMEDIR);
   }
+
+  /**
+   * Get the name of the default mount table to use. If
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE_NAME_KEY} is specified,
+   * it's value is returned. Otherwise,
+   * {@link Constants#CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE} is returned.
+   *
+   * @param conf Configuration to use.
+   * @return the name of the default mount table to use.
+   */
+  public static String

[hadoop] branch branch-3.1 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 30d1d29  HDFS-15429. mkdirs should work when parent dir is an 
internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
30d1d29 is described below

commit 30d1d2907643871ccaa3b0906e7af1d3f4f1e6fb
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 26 01:29:38 2020 -0700

HDFS-15429. mkdirs should work when parent dir is an internalDir and 
fallback configured. Contributed by Uma Maheswara Rao G.

(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  25 ++
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  28 +-
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 229 +---
 .../hadoop/fs/viewfs/TestViewFsLinkFallback.java   | 297 +
 4 files changed, 542 insertions(+), 37 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 16a5e08..c960a21 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1300,6 +1300,31 @@ public class ViewFileSystem extends FileSystem {
   dir.toString().substring(1))) {
 return true; // this is the stupid semantics of FileSystem
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+
+try {
+  return linkedFallbackFs.mkdirs(dirToCreate, permission);
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg =
+new StringBuilder("Failed to create ").append(dirToCreate)
+.append(" at fallback : ")
+.append(linkedFallbackFs.getUri());
+LOG.debug(msg.toString(), e);
+  }
+  return false;
+}
+  }
+
   throw readOnlyMountTable("mkdirs",  dir);
 }
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index b10c897..770f43b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -1127,11 +1127,35 @@ public class ViewFs extends AbstractFileSystem {
 
 @Override
 public void mkdir(final Path dir, final FsPermission permission,
-final boolean createParent) throws AccessControlException,
-FileAlreadyExistsException {
+final boolean createParent) throws IOException {
   if (theInternalDir.isRoot() && dir == null) {
 throw new FileAlreadyExistsException("/ already exits");
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+AbstractFileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+try {
+  // We are here because, the parent dir already exist in the mount
+  // table internal tree. So, let's create parent always in fallback.
+  linkedFallbackFs.mkdir(dirToCreate, permission, true);
+  return;
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg = new StringBuilder("Failed to create {}")
+.append(" at fallback fs : {}");
+LOG.debug(msg.toString(), dirToCreate, linkedFallbackFs.getUri());
+  }
+  throw e;
+}
+  }
+
   throw readOnlyMountTable("mkdir", dir);
 }
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
index

[hadoop] branch branch-3.1 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 2087013  HADOOP-17060. Clarify listStatus and getFileStatus behaviors 
inconsistent in the case of ViewFs implementation for isDirectory. Contributed 
by Uma Maheswara Rao G.
2087013 is described below

commit 208701329c47019d1f9db8222121f83436b0affd
Author: Uma Maheswara Rao G 
AuthorDate: Wed Jun 10 15:00:02 2020 -0700

HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent 
in the case of ViewFs implementation for isDirectory. Contributed by Uma 
Maheswara Rao G.

(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java| 36 --
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   | 24 +++
 .../src/main/java/org/apache/hadoop/fs/Hdfs.java   | 22 +
 .../apache/hadoop/hdfs/DistributedFileSystem.java  | 25 ---
 4 files changed, 94 insertions(+), 13 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 7552c06..e2d8eac 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -486,6 +486,14 @@ public class ViewFileSystem extends FileSystem {
 : new ViewFsFileStatus(orig, qualified);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a symlink(mount link), the path will be resolved to a
+   * target path and it will get the resolved path's FileStatus object. It will
+   * not be represented as a symlink and isDirectory API returns true if the
+   * resolved path is a directory, false otherwise.
+   */
   @Override
   public FileStatus getFileStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -503,6 +511,25 @@ public class ViewFileSystem extends FileSystem {
 res.targetFileSystem.access(res.remainingPath, mode);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * Note: listStatus on root("/") considers listing from fallbackLink if
+   * available. If the same directory name is present in configured mount path
+   * as well as in fallback link, then only the configured mount path will be
+   * listed in the returned result.
+   *
+   * If any of the the immediate children of the given path f is a 
symlink(mount
+   * link), the returned FileStatus object of that children would be 
represented
+   * as a symlink. It will not be resolved to the target path and will not get
+   * the target path FileStatus object. The target path will be available via
+   * getSymlink on that children's FileStatus object. Since it represents as
+   * symlink, isDirectory on that children's FileStatus will return false.
+   *
+   * If you want to get the FileStatus of target path for that children, you 
may
+   * want to use GetFileStatus API with that children's symlink path. Please 
see
+   * {@link ViewFileSystem#getFileStatus(Path f)}
+   */
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -1135,20 +1162,11 @@ public class ViewFileSystem extends FileSystem {
   checkPathIsSlash(f);
   return new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-
   new Path(theInternalDir.fullPath).makeQualified(
   myUri, ROOT_PATH));
 }
 
 
-/**
- * {@inheritDoc}
- *
- * Note: listStatus on root("/") considers listing from fallbackLink if
- * available. If the same directory name is present in configured mount
- * path as well as in fallback link, then only the configured mount path
- * will be listed in the returned result.
- */
 @Override
 public FileStatus[] listStatus(Path f) throws AccessControlException,
 FileNotFoundException, IOException {
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index 8cebc76..5d06b30 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -351,6 +351,14 @@ public class ViewFs extends AbstractFileSystem {
 return res.targetFileSystem.getFileChecksum(res.remainingPath);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a

[hadoop] branch branch-3.1 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new e17770d  HDFS-15418. ViewFileSystemOverloadScheme should represent 
mount links as non symlinks. Contributed by Uma Maheswara Rao G.
e17770d is described below

commit e17770dec60a560c26d873ae3cfcb3b2f943930e
Author: Uma Maheswara Rao G 
AuthorDate: Sat Jun 20 00:32:02 2020 -0700

HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as 
non symlinks. Contributed by Uma Maheswara Rao G.

(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
---
 .../org/apache/hadoop/fs/viewfs/Constants.java |   8 ++
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  71 +++
 .../fs/viewfs/ViewFileSystemOverloadScheme.java|  20 +++-
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  80 -
 .../viewfs/TestViewFsOverloadSchemeListStatus.java | 132 +
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java |   4 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  |  42 +++
 ...SystemOverloadSchemeHdfsFileSystemContract.java |   5 +
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java |   9 ++
 9 files changed, 295 insertions(+), 76 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
index 0a5d4b4..f454f63 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
@@ -90,4 +90,12 @@ public interface Constants {
   String CONFIG_VIEWFS_ENABLE_INNER_CACHE = "fs.viewfs.enable.inner.cache";
 
   boolean CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT = true;
+
+  /**
+   * Enable ViewFileSystem to show mountlinks as symlinks.
+   */
+  String CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS =
+  "fs.viewfs.mount.links.as.symlinks";
+
+  boolean CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT = true;
 }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index e2d8eac..a1fd14b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -19,6 +19,8 @@ package org.apache.hadoop.fs.viewfs;
 
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT;
 import static org.apache.hadoop.fs.viewfs.Constants.PERMISSION_555;
 
 import java.io.FileNotFoundException;
@@ -525,10 +527,18 @@ public class ViewFileSystem extends FileSystem {
* the target path FileStatus object. The target path will be available via
* getSymlink on that children's FileStatus object. Since it represents as
* symlink, isDirectory on that children's FileStatus will return false.
+   * This behavior can be changed by setting an advanced configuration
+   * fs.viewfs.mount.links.as.symlinks to false. In this case, mount points 
will
+   * be represented as non-symlinks and all the file/directory attributes like
+   * permissions, isDirectory etc will be assigned from it's resolved target
+   * directory/file.
*
* If you want to get the FileStatus of target path for that children, you 
may
* want to use GetFileStatus API with that children's symlink path. Please 
see
* {@link ViewFileSystem#getFileStatus(Path f)}
+   *
+   * Note: In ViewFileSystem, by default the mount links are represented as
+   * symlinks.
*/
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
@@ -1075,6 +1085,7 @@ public class ViewFileSystem extends FileSystem {
 final long creationTime; // of the the mount table
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
+private final boolean showMountLinksAsSymlinks;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
@@ -1088,6 +1099,9 @@ public class ViewFileSystem extends FileSystem {
   theInternalDir = dir;
   creationTime = cTime;
   this.ugi = ugi;
+  showMountLinksAsSymlinks = config
+  .getBoolean(CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS,

[hadoop] branch branch-3.1 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 81e6261  HDFS-15427. Merged ListStatus with Fallback target filesystem 
and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
81e6261 is described below

commit 81e62613fbf7d6185a69f3ff26dd0121b9318d68
Author: Uma Maheswara Rao G 
AuthorDate: Tue Jun 23 01:42:25 2020 -0700

HDFS-15427. Merged ListStatus with Fallback target filesystem and 
InternalDirViewFS. Contributed by Uma Maheswara Rao G.

(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
---
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |   4 +-
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  89 
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  94 +---
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 251 -
 4 files changed, 360 insertions(+), 78 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
index 50c839b..d1e5d3a 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
@@ -374,7 +374,7 @@ abstract class InodeTree {
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(INodeDir dir)
-  throws URISyntaxException;
+  throws URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(String settings, URI[] mergeFsURIs)
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
@@ -393,7 +393,7 @@ abstract class InodeTree {
 return rootFallbackLink != null;
   }
 
-  private INodeLink getRootFallbackLink() {
+  protected INodeLink getRootFallbackLink() {
 Preconditions.checkState(root.isInternalDir());
 return rootFallbackLink;
   }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a1fd14b..16a5e08 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -288,8 +288,9 @@ public class ViewFileSystem extends FileSystem {
 
 @Override
 protected FileSystem getTargetFileSystem(final INodeDir 
dir)
-  throws URISyntaxException {
-  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, 
config);
+throws URISyntaxException {
+  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, config,
+  this);
 }
 
 @Override
@@ -516,10 +517,10 @@ public class ViewFileSystem extends FileSystem {
   /**
* {@inheritDoc}
*
-   * Note: listStatus on root("/") considers listing from fallbackLink if
-   * available. If the same directory name is present in configured mount path
-   * as well as in fallback link, then only the configured mount path will be
-   * listed in the returned result.
+   * Note: listStatus considers listing from fallbackLink if available. If the
+   * same directory path is present in configured mount path as well as in
+   * fallback fs, then only the fallback path will be listed in the returned
+   * result except for link.
*
* If any of the the immediate children of the given path f is a 
symlink(mount
* link), the returned FileStatus object of that children would be 
represented
@@ -1086,11 +1087,13 @@ public class ViewFileSystem extends FileSystem {
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
 private final boolean showMountLinksAsSymlinks;
+private InodeTree fsState;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
-Configuration config) throws URISyntaxException {
+Configuration config, InodeTree fsState) throws URISyntaxException {
   myUri = uri;
+  this.fsState = fsState;
   try {
 initialize(myUri, config);
   } catch (IOException e) {
@@ -1186,7 +1189,8 @@ public class ViewFileSystem extends FileSystem {
 FileNotFoundException, IOException {
   checkPathIsSlash(f);
   FileStatus[] fallbackStatuses = listStatusForFallbackLink();
-  FileStatus[] result = new 
FileStatus[theInternalDir.getChildren().size()];
+  Set linkStatuses = new HashSet<>();
+  Set internalDirStatuses = new HashSet<>();

[hadoop] branch branch-3.1 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 0e9b6b0  HADOOP-17029. Return correct permission and owner for listing 
on internal directories in ViewFs. Contributed by Abhishek Das.
0e9b6b0 is described below

commit 0e9b6b0124cf0c2b314e3ede301bb14ccf539cc1
Author: Abhishek Das 
AuthorDate: Fri Jun 5 14:56:51 2020 -0700

HADOOP-17029. Return correct permission and owner for listing on internal 
directories in ViewFs. Contributed by Abhishek Das.

(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  27 +++--
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  41 +--
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java | 118 -
 3 files changed, 146 insertions(+), 40 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index bdf429e..a19366e 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1161,13 +1161,26 @@ public class ViewFileSystem extends FileSystem {
 INode inode = iEntry.getValue();
 if (inode.isLink()) {
   INodeLink link = (INodeLink) inode;
-
-  result[i++] = new FileStatus(0, false, 0, 0,
-creationTime, creationTime, PERMISSION_555,
-ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-link.getTargetLink(),
-new Path(inode.fullPath).makeQualified(
-myUri, null));
+  try {
+String linkedPath = link.getTargetFileSystem().getUri().getPath();
+FileStatus status =
+((ChRootedFileSystem)link.getTargetFileSystem())
+.getMyFs().getFileStatus(new Path(linkedPath));
+result[i++] = new FileStatus(status.getLen(), false,
+  status.getReplication(), status.getBlockSize(),
+  status.getModificationTime(), status.getAccessTime(),
+  status.getPermission(), status.getOwner(), status.getGroup(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  } catch (FileNotFoundException ex) {
+result[i++] = new FileStatus(0, false, 0, 0,
+  creationTime, creationTime, PERMISSION_555,
+  ugi.getShortUserName(), ugi.getPrimaryGroupName(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  }
 } else {
   result[i++] = new FileStatus(0, true, 0, 0,
 creationTime, creationTime, PERMISSION_555,
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index dde6649..8cebc76 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -910,11 +910,25 @@ public class ViewFs extends AbstractFileSystem {
   if (inode.isLink()) {
 INodeLink inodelink = 
   (INodeLink) inode;
-result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
+try {
+  String linkedPath = inodelink.getTargetFileSystem()
+  .getUri().getPath();
+  FileStatus status = ((ChRootedFs)inodelink.getTargetFileSystem())
+  .getMyFs().getFileStatus(new Path(linkedPath));
+  result = new FileStatus(status.getLen(), false,
+status.getReplication(), status.getBlockSize(),
+status.getModificationTime(), status.getAccessTime(),
+status.getPermission(), status.getOwner(), status.getGroup(),
+inodelink.getTargetLink(),
+new Path(inode.fullPath).makeQualified(
+myUri, null));
+} catch (FileNotFoundException ex) {
+  result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
 PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
 inodelink.getTargetLink(),
 new Path(inode.fullPath).makeQualified(
 myUri, null));
+}
   } else {
 result = new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
@@ -969,12 +983,25 @@ public class ViewFs extends

[hadoop] branch branch-3.1 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new b720d77  HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.
b720d77 is described below

commit b720d7770d93e67afe05beb3943913da241696d2
Author: Ayush Saxena 
AuthorDate: Mon Jun 8 01:59:10 2020 +0530

HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.

(cherry picked from commit a8610c15c498531bf3c011f1b0ace8ef61f2)
---
 .../src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java  | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a19366e..7552c06 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1163,6 +1163,9 @@ public class ViewFileSystem extends FileSystem {
   INodeLink link = (INodeLink) inode;
   try {
 String linkedPath = link.getTargetFileSystem().getUri().getPath();
+if("".equals(linkedPath)) {
+  linkedPath = "/";
+}
 FileStatus status =
 ((ChRootedFileSystem)link.getTargetFileSystem())
 .getMyFs().getFileStatus(new Path(linkedPath));


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.1 updated: HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 100c139  HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme 
in processPath. Contributed by Uma Maheswara Rao G.
100c139 is described below

commit 100c13967ea713b473e0742e3e40c17e8de62147
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 12 14:32:19 2020 -0700

HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit 785b1def959fab6b8b766410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
---
 .../java/org/apache/hadoop/fs/shell/FsUsage.java   |   3 +-
 .../hadoop/fs/viewfs/ViewFileSystemUtil.java   |  14 +-
 ...ViewFileSystemOverloadSchemeWithFSCommands.java | 173 +
 3 files changed, 188 insertions(+), 2 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
index 6596527..64aade3 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
@@ -128,7 +128,8 @@ class FsUsage extends FsCommand {
 
 @Override
 protected void processPath(PathData item) throws IOException {
-  if (ViewFileSystemUtil.isViewFileSystem(item.fs)) {
+  if (ViewFileSystemUtil.isViewFileSystem(item.fs)
+  || ViewFileSystemUtil.isViewFileSystemOverloadScheme(item.fs)) {
 ViewFileSystem viewFileSystem = (ViewFileSystem) item.fs;
 Map  fsStatusMap =
 ViewFileSystemUtil.getStatus(viewFileSystem, item.path);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
index c8a1d78..f486a10 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
@@ -52,6 +52,17 @@ public final class ViewFileSystemUtil {
   }
 
   /**
+   * Check if the FileSystem is a ViewFileSystemOverloadScheme.
+   *
+   * @param fileSystem
+   * @return true if the fileSystem is ViewFileSystemOverloadScheme
+   */
+  public static boolean isViewFileSystemOverloadScheme(
+  final FileSystem fileSystem) {
+return fileSystem instanceof ViewFileSystemOverloadScheme;
+  }
+
+  /**
* Get FsStatus for all ViewFsMountPoints matching path for the given
* ViewFileSystem.
*
@@ -93,7 +104,8 @@ public final class ViewFileSystemUtil {
*/
   public static Map getStatus(
   FileSystem fileSystem, Path path) throws IOException {
-if (!isViewFileSystem(fileSystem)) {
+if (!(isViewFileSystem(fileSystem)
+|| isViewFileSystemOverloadScheme(fileSystem))) {
   throw new UnsupportedFileSystemException("FileSystem '"
   + fileSystem.getUri() + "'is not a ViewFileSystem.");
 }
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
new file mode 100644
index 000..a974377
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
@@ -0,0 +1,173 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.tools;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;

[hadoop] branch branch-3.1 updated: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 64dbb39  HDFS-15330. Document the ViewFSOverloadScheme details in 
ViewFS guide. Contributed by Uma Maheswara Rao G.
64dbb39 is described below

commit 64dbb39e7134cc59750502949d16043bcc50288d
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 5 10:58:21 2020 -0700

HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
(cherry picked from commit 418580446b65be3a0674762e76fc2cb9a1e5629a)
---
 .../hadoop-hdfs/src/site/markdown/HDFSCommands.md  |  40 -
 .../hadoop-hdfs/src/site/markdown/ViewFs.md|   6 +
 .../src/site/markdown/ViewFsOverloadScheme.md  | 163 +
 .../site/resources/images/ViewFSOverloadScheme.png | Bin 0 -> 190004 bytes
 hadoop-project/src/site/site.xml   |   1 +
 5 files changed, 209 insertions(+), 1 deletion(-)

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index c8a9184..b545e9b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -670,4 +670,42 @@ Usage: `hdfs debug recoverLease -path  [-retries 
]`
 | [`-path` *path*] | HDFS path for which to recover the lease. |
 | [`-retries` *num-retries*] | Number of times the client will retry calling 
recoverLease. The default number of retries is 1. |
 
-Recover the lease on the specified path. The path must reside on an HDFS 
filesystem. The default number of retries is 1.
+Recover the lease on the specified path. The path must reside on an HDFS file 
system. The default number of retries is 1.
+
+dfsadmin with ViewFsOverloadScheme
+--
+
+Usage: `hdfs dfsadmin -fs  `
+
+| COMMAND\_OPTION | Description |
+|: |: |
+| `-fs` *child fs mount link URI* | Its a logical mount link path to child 
file system in ViewFS world. This uri typically formed as src mount link 
prefixed with fs.defaultFS. Please note, this is not an actual child file 
system uri, instead its a logical mount link uri pointing to actual child file 
system|
+
+Example command usage:
+   `hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
+
+In ViewFsOverloadScheme, we may have multiple child file systems as mount 
point mappings as shown in [ViewFsOverloadScheme 
Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic 
parameter supported by dfsadmin. When users want to execute commands on one of 
the child file system, they need to pass that file system mount mapping link 
uri to -fs option. Let's take an example mount link configuration and dfsadmin 
command below.
+
+Mount link:
+
+```xml
+
+  fs.defaultFS
+  hdfs://MyCluster1
+
+
+
+  fs.viewfs.mounttable.MyCluster1./user
+  hdfs://MyCluster2/user
+   hdfs://MyCluster2/user
+   mount link path: /user
+   mount link uri: hdfs://MyCluster1/user
+   mount target uri for /user: hdfs://MyCluster2/user -->
+
+```
+
+If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option 
(`-fs hdfs://MyCluster1/user`)
+Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve 
the passed (`-fs hdfs://MyCluster1/user`) to target fs 
(`hdfs://MyCluster2/user`).
+This way users can get the access to all hdfs child file systems in 
ViewFsOverloadScheme.
+If there is no `-fs` option provided, then it will try to connect to the 
configured fs.defaultFS cluster if a cluster running with the fs.defaultFS uri.
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
index f851ef6..52ad49c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
@@ -361,6 +361,12 @@ resume its work, it's a good idea to provision some sort 
of cron job to purge su
 
 Delegation tokens for the cluster to which you are submitting the job 
(including all mounted volumes for that cluster’s mount table), and for input 
and output paths to your map-reduce job (including all volumes mounted via 
mount tables for the specified input and output paths) are all handled 
automatically. In addition, there is a way to add additional delegation tokens 
to the base cluster configuration for special circumstances.
 
+Don't want to change scheme or difficult to copy mount-table configurations to 
all clients?
+---
+
+Please refer to the [View File System Overload Scheme

[hadoop] branch branch-3.1 updated: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new f7e590a  HDFS-15389. DFSAdmin should close filesystem and dfsadmin 
-setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by 
Ayush Saxena
f7e590a is described below

commit f7e590aaffd03ed4308a7a90f862e876ea89f641
Author: Ayush Saxena 
AuthorDate: Sat Jun 6 10:49:38 2020 +0530

HDFS-15389. DFSAdmin should close filesystem and dfsadmin 
-setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by 
Ayush Saxena

(cherry picked from commit cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
(cherry picked from commit bee2846bee4ae676bdc14585f8a3927a9dd7df37)
---
 .../java/org/apache/hadoop/hdfs/tools/DFSAdmin.java  | 13 +++--
 ...TestViewFileSystemOverloadSchemeWithDFSAdmin.java | 20 
 2 files changed, 23 insertions(+), 10 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index ab243f3..163c147 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -479,9 +479,9 @@ public class DFSAdmin extends FsShell {
   public DFSAdmin(Configuration conf) {
 super(conf);
   }
-  
+
   protected DistributedFileSystem getDFS() throws IOException {
-return AdminHelper.getDFS(getConf());
+return AdminHelper.checkAndGetDFS(getFS(), getConf());
   }
   
   /**
@@ -1036,14 +1036,7 @@ public class DFSAdmin extends FsShell {
   System.err.println("Bandwidth should be a non-negative integer");
   return exitCode;
 }
-
-FileSystem fs = getFS();
-if (!(fs instanceof DistributedFileSystem)) {
-  System.err.println("FileSystem is " + fs.getUri());
-  return exitCode;
-}
-
-DistributedFileSystem dfs = (DistributedFileSystem) fs;
+DistributedFileSystem dfs = getDFS();
 try{
   dfs.setBalancerBandwidth(bandwidth);
   System.out.println("Balancer bandwidth is set to " + bandwidth);
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
index 1961dc2..a9475dd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
@@ -263,4 +263,24 @@ public class TestViewFileSystemOverloadSchemeWithDFSAdmin {
 assertOutMsg("Disallowing snapshot on / succeeded", 1);
 assertEquals(0, ret);
   }
+
+  /**
+   * Tests setBalancerBandwidth with ViewFSOverloadScheme.
+   */
+  @Test
+  public void testSetBalancerBandwidth() throws Exception {
+final Path hdfsTargetPath = new Path(defaultFSURI + HDFS_USER_FOLDER);
+addMountLinks(defaultFSURI.getAuthority(),
+new String[] {HDFS_USER_FOLDER, LOCAL_FOLDER },
+new String[] {hdfsTargetPath.toUri().toString(),
+localTargetDir.toURI().toString() },
+conf);
+final DFSAdmin dfsAdmin = new DFSAdmin(conf);
+redirectStream();
+int ret = ToolRunner.run(dfsAdmin,
+new String[] {"-fs", defaultFSURI.toString(), "-setBalancerBandwidth",
+"1000"});
+assertOutMsg("Balancer bandwidth is set to 1000", 0);
+assertEquals(0, ret);
+  }
 }
\ No newline at end of file


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.1 updated: HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 2e1dfc1  HDFS-15321. Make DFSAdmin tool to work with 
ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
2e1dfc1 is described below

commit 2e1dfc152b36d628c17c66de11741e8828312bd4
Author: Uma Maheswara Rao G 
AuthorDate: Tue Jun 2 11:09:26 2020 -0700

HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
(cherry picked from commit 0b5e202614f0bc20a0db6656f924fa4d2741d00c)
---
 .../fs/viewfs/ViewFileSystemOverloadScheme.java|  29 +++
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |   2 +-
 .../org/apache/hadoop/hdfs/tools/AdminHelper.java  |  25 +-
 .../org/apache/hadoop/hdfs/tools/DFSAdmin.java |  13 +-
 ...stViewFileSystemOverloadSchemeWithDFSAdmin.java | 266 +
 5 files changed, 317 insertions(+), 18 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
index f5952d5..36f9cd1 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
@@ -17,6 +17,7 @@
  */
 package org.apache.hadoop.fs.viewfs;
 
+import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
@@ -27,6 +28,7 @@ import org.apache.hadoop.classification.InterfaceStability;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.UnsupportedFileSystemException;
 
 /**
@@ -227,4 +229,31 @@ public class ViewFileSystemOverloadScheme extends 
ViewFileSystem {
 
   }
 
+  /**
+   * This is an admin only API to give access to its child raw file system, if
+   * the path is link. If the given path is an internal directory(path is from
+   * mount paths tree), it will initialize the file system of given path uri
+   * directly. If path cannot be resolved to any internal directory or link, it
+   * will throw NotInMountpointException. Please note, this API will not return
+   * chrooted file system. Instead, this API will get actual raw file system
+   * instances.
+   *
+   * @param path - fs uri path
+   * @param conf - configuration
+   * @throws IOException
+   */
+  public FileSystem getRawFileSystem(Path path, Configuration conf)
+  throws IOException {
+InodeTree.ResolveResult res;
+try {
+  res = fsState.resolve(getUriPath(path), true);
+  return res.isInternalDir() ? fsGetter().get(path.toUri(), conf)
+  : ((ChRootedFileSystem) res.targetFileSystem).getMyFs();
+} catch (FileNotFoundException e) {
+  // No link configured with passed path.
+  throw new NotInMountpointException(path,
+  "No link found for the given path.");
+}
+  }
+
 }
\ No newline at end of file
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
index f051c9c..efced73 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
@@ -192,7 +192,7 @@ public class ViewFsTestSetup {
* Adds the given mount links to the configuration. Mount link mappings are
* in sources, targets at their respective index locations.
*/
-  static void addMountLinksToConf(String mountTable, String[] sources,
+  public static void addMountLinksToConf(String mountTable, String[] sources,
   String[] targets, Configuration config) throws URISyntaxException {
 for (int i = 0; i < sources.length; i++) {
   String src = sources[i];
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
index 9cb646b..27cdf70 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
@@ -1,4 +1,5 @@
 /**
+
  * Licensed

[hadoop] branch branch-3.1 updated: HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 6ae9296  HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's 
scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
6ae9296 is described below

commit 6ae92962d9d03d09b67a373831c2d3b53948c497
Author: Uma Maheswara Rao G 
AuthorDate: Thu May 21 21:34:58 2020 -0700

HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and 
target uris schemes are same. Contributed by Uma Maheswara Rao G.

(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
(cherry picked from commit 8e71e85af70c17f2350f794f8bc2475eb1e3acea)
---
 .../org/apache/hadoop/fs/viewfs/ConfigUtil.java|  15 ++-
 .../java/org/apache/hadoop/fs/viewfs/FsGetter.java |  47 
 .../fs/viewfs/HCFSMountTableConfigLoader.java  |   3 +-
 .../org/apache/hadoop/fs/viewfs/NflyFSystem.java   |  29 -
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  24 +---
 .../hadoop/fs/viewfs/ViewFileSystemBaseTest.java   |   1 -
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  28 -
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 121 +
 8 files changed, 230 insertions(+), 38 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
index 4c3dae9..6dd1f65 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
@@ -136,6 +136,17 @@ public class ConfigUtil {
   }
 
   /**
+   * Add nfly link to configuration for the given mount table.
+   */
+  public static void addLinkNfly(Configuration conf, String mountTableName,
+  String src, String settings, final String targets) {
+conf.set(
+getConfigViewFsPrefix(mountTableName) + "."
++ Constants.CONFIG_VIEWFS_LINK_NFLY + "." + settings + "." + src,
+targets);
+  }
+
+  /**
*
* @param conf
* @param mountTableName
@@ -149,9 +160,7 @@ public class ConfigUtil {
 settings = settings == null
 ? "minReplication=2,repairOnRead=true"
 : settings;
-
-conf.set(getConfigViewFsPrefix(mountTableName) + "." +
-Constants.CONFIG_VIEWFS_LINK_NFLY + "." + settings + "." + src,
+addLinkNfly(conf, mountTableName, src, settings,
 StringUtils.uriToString(targets));
   }
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
new file mode 100644
index 000..071af11
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.net.URI;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+/**
+ * File system instance getter.
+ */
+@Private
+class FsGetter {
+
+  /**
+   * Gets new file system instance of given uri.
+   */
+  public FileSystem getNewInstance(URI uri, Configuration conf)
+  throws IOException {
+return FileSystem.newInstance(uri, conf);
+  }
+
+  /**
+   * Gets file system instance of given uri.
+   */
+  public FileSystem get(URI uri, Configuration conf) throws IOException {
+return FileSystem.get(uri, conf);
+  }
+}
\ No newline at end of file
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
index c7e5aab..3968e36 100644
---

[hadoop] branch branch-3.1 updated: HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 7015589  HADOOP-17024. ListStatus on ViewFS root (ls "/") should list 
the linkFallBack root (configured target root). Contributed by Abhishek Das.
7015589 is described below

commit 7015589f5845d2732a8b7ba80af9d40187ad5167
Author: Abhishek Das 
AuthorDate: Mon May 18 22:27:12 2020 -0700

HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the 
linkFallBack root (configured target root). Contributed by Abhishek Das.

(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
(cherry picked from commit 5b248de42d2ae42710531a1514a21d60a1fca4b2)
---
 .../org/apache/hadoop/fs/viewfs/InodeTree.java | 13 +++
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java| 49 ++-
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   | 51 ++-
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 98 ++
 4 files changed, 209 insertions(+), 2 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
index 6992343..50c839b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
@@ -123,6 +123,7 @@ abstract class InodeTree {
 private final Map> children = new HashMap<>();
 private T internalDirFs =  null; //filesystem of this internal directory
 private boolean isRoot = false;
+private INodeLink fallbackLink = null;
 
 INodeDir(final String pathToNode, final UserGroupInformation aUgi) {
   super(pathToNode, aUgi);
@@ -149,6 +150,17 @@ abstract class InodeTree {
   return isRoot;
 }
 
+INodeLink getFallbackLink() {
+  return fallbackLink;
+}
+
+void addFallbackLink(INodeLink link) throws IOException {
+  if (!isRoot) {
+throw new IOException("Fallback link can only be added for root");
+  }
+  this.fallbackLink = link;
+}
+
 Map> getChildren() {
   return Collections.unmodifiableMap(children);
 }
@@ -580,6 +592,7 @@ abstract class InodeTree {
 }
   }
   rootFallbackLink = fallbackLink;
+  getRootDir().addFallbackLink(rootFallbackLink);
 }
 
 if (!gotMountTableEntry) {
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a13b6ea..f626ffe 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1161,10 +1161,19 @@ public class ViewFileSystem extends FileSystem {
 }
 
 
+/**
+ * {@inheritDoc}
+ *
+ * Note: listStatus on root("/") considers listing from fallbackLink if
+ * available. If the same directory name is present in configured mount
+ * path as well as in fallback link, then only the configured mount path
+ * will be listed in the returned result.
+ */
 @Override
 public FileStatus[] listStatus(Path f) throws AccessControlException,
 FileNotFoundException, IOException {
   checkPathIsSlash(f);
+  FileStatus[] fallbackStatuses = listStatusForFallbackLink();
   FileStatus[] result = new 
FileStatus[theInternalDir.getChildren().size()];
   int i = 0;
   for (Entry> iEntry :
@@ -1187,7 +1196,45 @@ public class ViewFileSystem extends FileSystem {
 myUri, null));
 }
   }
-  return result;
+  if (fallbackStatuses.length > 0) {
+return consolidateFileStatuses(fallbackStatuses, result);
+  } else {
+return result;
+  }
+}
+
+private FileStatus[] consolidateFileStatuses(FileStatus[] fallbackStatuses,
+FileStatus[] mountPointStatuses) {
+  ArrayList result = new ArrayList<>();
+  Set pathSet = new HashSet<>();
+  for (FileStatus status : mountPointStatuses) {
+result.add(status);
+pathSet.add(status.getPath().getName());
+  }
+  for (FileStatus status : fallbackStatuses) {
+if (!pathSet.contains(status.getPath().getName())) {
+  result.add(status);
+}
+  }
+  return result.toArray(new FileStatus[0]);
+}
+
+private FileStatus[] listStatusForFallbackLink() throws IOException {
+  if (theInternalDir.isRoot() &&
+  theInternalDir.getFallbackLink() != null) {
+FileSystem linkedFs =
+

[hadoop] branch branch-3.1 updated: HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new c4857bb  HDFS-15306. Make mount-table to read from central place ( 
Let's say from HDFS). Contributed by Uma Maheswara Rao G.
c4857bb is described below

commit c4857bb9c1bd5c937cdc658192172d09e56609bc
Author: Uma Maheswara Rao G 
AuthorDate: Thu May 14 17:29:35 2020 -0700

HDFS-15306. Make mount-table to read from central place ( Let's say from 
HDFS). Contributed by Uma Maheswara Rao G.

(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
(cherry picked from commit 544996c85702af7ae241ef2f18e2597e2b4050be)
---
 .../org/apache/hadoop/fs/viewfs/Constants.java |   5 +
 .../fs/viewfs/HCFSMountTableConfigLoader.java  | 122 ++
 .../hadoop/fs/viewfs/MountTableConfigLoader.java   |  44 +
 .../fs/viewfs/ViewFileSystemOverloadScheme.java| 180 -
 .../org/apache/hadoop/fs/viewfs/package-info.java  |  26 +++
 .../fs/viewfs/TestHCFSMountTableConfigLoader.java  | 165 +++
 ...iewFSOverloadSchemeCentralMountTableConfig.java |  77 +
 ...iewFileSystemOverloadSchemeLocalFileSystem.java |  47 --
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  71 +++-
 ...FSOverloadSchemeWithMountTableConfigInHDFS.java |  68 
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 125 +-
 11 files changed, 797 insertions(+), 133 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
index 37f1a16..0a5d4b4 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
@@ -30,6 +30,11 @@ public interface Constants {
* Prefix for the config variable prefix for the ViewFs mount-table
*/
   public static final String CONFIG_VIEWFS_PREFIX = "fs.viewfs.mounttable";
+
+  /**
+   * Prefix for the config variable for the ViewFs mount-table path.
+   */
+  String CONFIG_VIEWFS_MOUNTTABLE_PATH = CONFIG_VIEWFS_PREFIX + ".path";
  
   /**
* Prefix for the home dir for the mount table - if not specified
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
new file mode 100644
index 000..c7e5aab
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation for Apache Hadoop compatible file system based mount-table
+ * file loading.
+ */
+public class HCFSMountTableConfigLoader implements MountTableConfigLoader {
+  private static final String REGEX_DOT = "[.]";
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(HCFSMountTableConfigLoader.class);
+  private Path mountTable = null;
+
+  /**
+   * Loads the mount-table configuration from hadoop compatible file system and
+   * add the configuration items to given configuration. Mount-table
+   * configuration format should be suffixed with version number.
+   * Format: mount-table..xml
+   * Example: mount-table.1.xml
+   * When user wants to update mount-table, the expectation is to upload new
+   * mount-table configuration file with monotonically increasing integer as
+   * version number.

[hadoop] branch branch-3.2 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 80111fe  HDFS-15427. Merged ListStatus with Fallback target filesystem 
and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
80111fe is described below

commit 80111fe5bb65fc3d6276c007747e8c1c74d636e7
Author: Uma Maheswara Rao G 
AuthorDate: Tue Jun 23 01:42:25 2020 -0700

HDFS-15427. Merged ListStatus with Fallback target filesystem and 
InternalDirViewFS. Contributed by Uma Maheswara Rao G.

(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
---
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |   4 +-
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  89 
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  94 +---
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 251 -
 4 files changed, 360 insertions(+), 78 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
index 50c839b..d1e5d3a 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
@@ -374,7 +374,7 @@ abstract class InodeTree {
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(INodeDir dir)
-  throws URISyntaxException;
+  throws URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(String settings, URI[] mergeFsURIs)
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
@@ -393,7 +393,7 @@ abstract class InodeTree {
 return rootFallbackLink != null;
   }
 
-  private INodeLink getRootFallbackLink() {
+  protected INodeLink getRootFallbackLink() {
 Preconditions.checkState(root.isInternalDir());
 return rootFallbackLink;
   }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a1fd14b..16a5e08 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -288,8 +288,9 @@ public class ViewFileSystem extends FileSystem {
 
 @Override
 protected FileSystem getTargetFileSystem(final INodeDir 
dir)
-  throws URISyntaxException {
-  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, 
config);
+throws URISyntaxException {
+  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, config,
+  this);
 }
 
 @Override
@@ -516,10 +517,10 @@ public class ViewFileSystem extends FileSystem {
   /**
* {@inheritDoc}
*
-   * Note: listStatus on root("/") considers listing from fallbackLink if
-   * available. If the same directory name is present in configured mount path
-   * as well as in fallback link, then only the configured mount path will be
-   * listed in the returned result.
+   * Note: listStatus considers listing from fallbackLink if available. If the
+   * same directory path is present in configured mount path as well as in
+   * fallback fs, then only the fallback path will be listed in the returned
+   * result except for link.
*
* If any of the the immediate children of the given path f is a 
symlink(mount
* link), the returned FileStatus object of that children would be 
represented
@@ -1086,11 +1087,13 @@ public class ViewFileSystem extends FileSystem {
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
 private final boolean showMountLinksAsSymlinks;
+private InodeTree fsState;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
-Configuration config) throws URISyntaxException {
+Configuration config, InodeTree fsState) throws URISyntaxException {
   myUri = uri;
+  this.fsState = fsState;
   try {
 initialize(myUri, config);
   } catch (IOException e) {
@@ -1186,7 +1189,8 @@ public class ViewFileSystem extends FileSystem {
 FileNotFoundException, IOException {
   checkPathIsSlash(f);
   FileStatus[] fallbackStatuses = listStatusForFallbackLink();
-  FileStatus[] result = new 
FileStatus[theInternalDir.getChildren().size()];
+  Set linkStatuses = new HashSet<>();
+  Set internalDirStatuses = new HashSet<>();

[hadoop] branch branch-3.2 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new d030f4b  HDFS-15429. mkdirs should work when parent dir is an 
internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
d030f4b is described below

commit d030f4b2a632b97d0ceda4216880d5690ac5ee14
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 26 01:29:38 2020 -0700

HDFS-15429. mkdirs should work when parent dir is an internalDir and 
fallback configured. Contributed by Uma Maheswara Rao G.

(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  25 ++
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  28 +-
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 229 +---
 .../hadoop/fs/viewfs/TestViewFsLinkFallback.java   | 297 +
 4 files changed, 542 insertions(+), 37 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 16a5e08..c960a21 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1300,6 +1300,31 @@ public class ViewFileSystem extends FileSystem {
   dir.toString().substring(1))) {
 return true; // this is the stupid semantics of FileSystem
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+
+try {
+  return linkedFallbackFs.mkdirs(dirToCreate, permission);
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg =
+new StringBuilder("Failed to create ").append(dirToCreate)
+.append(" at fallback : ")
+.append(linkedFallbackFs.getUri());
+LOG.debug(msg.toString(), e);
+  }
+  return false;
+}
+  }
+
   throw readOnlyMountTable("mkdirs",  dir);
 }
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index b10c897..770f43b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -1127,11 +1127,35 @@ public class ViewFs extends AbstractFileSystem {
 
 @Override
 public void mkdir(final Path dir, final FsPermission permission,
-final boolean createParent) throws AccessControlException,
-FileAlreadyExistsException {
+final boolean createParent) throws IOException {
   if (theInternalDir.isRoot() && dir == null) {
 throw new FileAlreadyExistsException("/ already exits");
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+AbstractFileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+try {
+  // We are here because, the parent dir already exist in the mount
+  // table internal tree. So, let's create parent always in fallback.
+  linkedFallbackFs.mkdir(dirToCreate, permission, true);
+  return;
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg = new StringBuilder("Failed to create {}")
+.append(" at fallback fs : {}");
+LOG.debug(msg.toString(), dirToCreate, linkedFallbackFs.getUri());
+  }
+  throw e;
+}
+  }
+
   throw readOnlyMountTable("mkdir", dir);
 }
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
index

[hadoop] branch branch-3.2 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 6c22210  HDFS-15418. ViewFileSystemOverloadScheme should represent 
mount links as non symlinks. Contributed by Uma Maheswara Rao G.
6c22210 is described below

commit 6c22210baa3591ce4e280d244891399075e47424
Author: Uma Maheswara Rao G 
AuthorDate: Sat Jun 20 00:32:02 2020 -0700

HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as 
non symlinks. Contributed by Uma Maheswara Rao G.

(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
---
 .../org/apache/hadoop/fs/viewfs/Constants.java |   8 ++
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  71 +++
 .../fs/viewfs/ViewFileSystemOverloadScheme.java|  20 +++-
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  80 -
 .../viewfs/TestViewFsOverloadSchemeListStatus.java | 132 +
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java |   4 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  |  42 +++
 ...SystemOverloadSchemeHdfsFileSystemContract.java |   5 +
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java |   9 ++
 9 files changed, 295 insertions(+), 76 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
index 0a5d4b4..f454f63 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
@@ -90,4 +90,12 @@ public interface Constants {
   String CONFIG_VIEWFS_ENABLE_INNER_CACHE = "fs.viewfs.enable.inner.cache";
 
   boolean CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT = true;
+
+  /**
+   * Enable ViewFileSystem to show mountlinks as symlinks.
+   */
+  String CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS =
+  "fs.viewfs.mount.links.as.symlinks";
+
+  boolean CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT = true;
 }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index e2d8eac..a1fd14b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -19,6 +19,8 @@ package org.apache.hadoop.fs.viewfs;
 
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT;
 import static org.apache.hadoop.fs.viewfs.Constants.PERMISSION_555;
 
 import java.io.FileNotFoundException;
@@ -525,10 +527,18 @@ public class ViewFileSystem extends FileSystem {
* the target path FileStatus object. The target path will be available via
* getSymlink on that children's FileStatus object. Since it represents as
* symlink, isDirectory on that children's FileStatus will return false.
+   * This behavior can be changed by setting an advanced configuration
+   * fs.viewfs.mount.links.as.symlinks to false. In this case, mount points 
will
+   * be represented as non-symlinks and all the file/directory attributes like
+   * permissions, isDirectory etc will be assigned from it's resolved target
+   * directory/file.
*
* If you want to get the FileStatus of target path for that children, you 
may
* want to use GetFileStatus API with that children's symlink path. Please 
see
* {@link ViewFileSystem#getFileStatus(Path f)}
+   *
+   * Note: In ViewFileSystem, by default the mount links are represented as
+   * symlinks.
*/
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
@@ -1075,6 +1085,7 @@ public class ViewFileSystem extends FileSystem {
 final long creationTime; // of the the mount table
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
+private final boolean showMountLinksAsSymlinks;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
@@ -1088,6 +1099,9 @@ public class ViewFileSystem extends FileSystem {
   theInternalDir = dir;
   creationTime = cTime;
   this.ugi = ugi;
+  showMountLinksAsSymlinks = config
+  .getBoolean(CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS,

[hadoop] branch branch-3.2 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 4a9ec4d  HADOOP-17060. Clarify listStatus and getFileStatus behaviors 
inconsistent in the case of ViewFs implementation for isDirectory. Contributed 
by Uma Maheswara Rao G.
4a9ec4d is described below

commit 4a9ec4d1431c819870444915a43283089a783443
Author: Uma Maheswara Rao G 
AuthorDate: Wed Jun 10 15:00:02 2020 -0700

HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent 
in the case of ViewFs implementation for isDirectory. Contributed by Uma 
Maheswara Rao G.

(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java| 36 --
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   | 24 +++
 .../src/main/java/org/apache/hadoop/fs/Hdfs.java   | 22 +
 .../apache/hadoop/hdfs/DistributedFileSystem.java  | 25 ---
 4 files changed, 94 insertions(+), 13 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 7552c06..e2d8eac 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -486,6 +486,14 @@ public class ViewFileSystem extends FileSystem {
 : new ViewFsFileStatus(orig, qualified);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a symlink(mount link), the path will be resolved to a
+   * target path and it will get the resolved path's FileStatus object. It will
+   * not be represented as a symlink and isDirectory API returns true if the
+   * resolved path is a directory, false otherwise.
+   */
   @Override
   public FileStatus getFileStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -503,6 +511,25 @@ public class ViewFileSystem extends FileSystem {
 res.targetFileSystem.access(res.remainingPath, mode);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * Note: listStatus on root("/") considers listing from fallbackLink if
+   * available. If the same directory name is present in configured mount path
+   * as well as in fallback link, then only the configured mount path will be
+   * listed in the returned result.
+   *
+   * If any of the the immediate children of the given path f is a 
symlink(mount
+   * link), the returned FileStatus object of that children would be 
represented
+   * as a symlink. It will not be resolved to the target path and will not get
+   * the target path FileStatus object. The target path will be available via
+   * getSymlink on that children's FileStatus object. Since it represents as
+   * symlink, isDirectory on that children's FileStatus will return false.
+   *
+   * If you want to get the FileStatus of target path for that children, you 
may
+   * want to use GetFileStatus API with that children's symlink path. Please 
see
+   * {@link ViewFileSystem#getFileStatus(Path f)}
+   */
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -1135,20 +1162,11 @@ public class ViewFileSystem extends FileSystem {
   checkPathIsSlash(f);
   return new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-
   new Path(theInternalDir.fullPath).makeQualified(
   myUri, ROOT_PATH));
 }
 
 
-/**
- * {@inheritDoc}
- *
- * Note: listStatus on root("/") considers listing from fallbackLink if
- * available. If the same directory name is present in configured mount
- * path as well as in fallback link, then only the configured mount path
- * will be listed in the returned result.
- */
 @Override
 public FileStatus[] listStatus(Path f) throws AccessControlException,
 FileNotFoundException, IOException {
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index 8cebc76..5d06b30 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -351,6 +351,14 @@ public class ViewFs extends AbstractFileSystem {
 return res.targetFileSystem.getFileChecksum(res.remainingPath);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a

[hadoop] branch branch-3.2 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new fc9445a  HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.
fc9445a is described below

commit fc9445abbddac906b3258df2414a39ac72426885
Author: Ayush Saxena 
AuthorDate: Mon Jun 8 01:59:10 2020 +0530

HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.

(cherry picked from commit a8610c15c498531bf3c011f1b0ace8ef61f2)
---
 .../src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java  | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a19366e..7552c06 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1163,6 +1163,9 @@ public class ViewFileSystem extends FileSystem {
   INodeLink link = (INodeLink) inode;
   try {
 String linkedPath = link.getTargetFileSystem().getUri().getPath();
+if("".equals(linkedPath)) {
+  linkedPath = "/";
+}
 FileStatus status =
 ((ChRootedFileSystem)link.getTargetFileSystem())
 .getMyFs().getFileStatus(new Path(linkedPath));


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.2 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new db5c32d  HADOOP-17029. Return correct permission and owner for listing 
on internal directories in ViewFs. Contributed by Abhishek Das.
db5c32d is described below

commit db5c32d2a56bad58cc4403305e60469aa7f1d854
Author: Abhishek Das 
AuthorDate: Fri Jun 5 14:56:51 2020 -0700

HADOOP-17029. Return correct permission and owner for listing on internal 
directories in ViewFs. Contributed by Abhishek Das.

(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  27 +++--
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  41 +--
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java | 118 -
 3 files changed, 146 insertions(+), 40 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index bdf429e..a19366e 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1161,13 +1161,26 @@ public class ViewFileSystem extends FileSystem {
 INode inode = iEntry.getValue();
 if (inode.isLink()) {
   INodeLink link = (INodeLink) inode;
-
-  result[i++] = new FileStatus(0, false, 0, 0,
-creationTime, creationTime, PERMISSION_555,
-ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-link.getTargetLink(),
-new Path(inode.fullPath).makeQualified(
-myUri, null));
+  try {
+String linkedPath = link.getTargetFileSystem().getUri().getPath();
+FileStatus status =
+((ChRootedFileSystem)link.getTargetFileSystem())
+.getMyFs().getFileStatus(new Path(linkedPath));
+result[i++] = new FileStatus(status.getLen(), false,
+  status.getReplication(), status.getBlockSize(),
+  status.getModificationTime(), status.getAccessTime(),
+  status.getPermission(), status.getOwner(), status.getGroup(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  } catch (FileNotFoundException ex) {
+result[i++] = new FileStatus(0, false, 0, 0,
+  creationTime, creationTime, PERMISSION_555,
+  ugi.getShortUserName(), ugi.getPrimaryGroupName(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  }
 } else {
   result[i++] = new FileStatus(0, true, 0, 0,
 creationTime, creationTime, PERMISSION_555,
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index dde6649..8cebc76 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -910,11 +910,25 @@ public class ViewFs extends AbstractFileSystem {
   if (inode.isLink()) {
 INodeLink inodelink = 
   (INodeLink) inode;
-result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
+try {
+  String linkedPath = inodelink.getTargetFileSystem()
+  .getUri().getPath();
+  FileStatus status = ((ChRootedFs)inodelink.getTargetFileSystem())
+  .getMyFs().getFileStatus(new Path(linkedPath));
+  result = new FileStatus(status.getLen(), false,
+status.getReplication(), status.getBlockSize(),
+status.getModificationTime(), status.getAccessTime(),
+status.getPermission(), status.getOwner(), status.getGroup(),
+inodelink.getTargetLink(),
+new Path(inode.fullPath).makeQualified(
+myUri, null));
+} catch (FileNotFoundException ex) {
+  result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
 PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
 inodelink.getTargetLink(),
 new Path(inode.fullPath).makeQualified(
 myUri, null));
+}
   } else {
 result = new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
@@ -969,12 +983,25 @@ public class ViewFs extends

[hadoop] branch branch-3.2 updated: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new e7cead1  HDFS-15330. Document the ViewFSOverloadScheme details in 
ViewFS guide. Contributed by Uma Maheswara Rao G.
e7cead1 is described below

commit e7cead114314412d461ed1c4b28ba33726c65b9b
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 5 10:58:21 2020 -0700

HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
(cherry picked from commit 418580446b65be3a0674762e76fc2cb9a1e5629a)
---
 .../hadoop-hdfs/src/site/markdown/HDFSCommands.md  |  40 -
 .../hadoop-hdfs/src/site/markdown/ViewFs.md|   6 +
 .../src/site/markdown/ViewFsOverloadScheme.md  | 163 +
 .../site/resources/images/ViewFSOverloadScheme.png | Bin 0 -> 190004 bytes
 hadoop-project/src/site/site.xml   |   1 +
 5 files changed, 209 insertions(+), 1 deletion(-)

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index 306884c..32e1a7b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -672,4 +672,42 @@ Usage: `hdfs debug recoverLease -path  [-retries 
]`
 | [`-path` *path*] | HDFS path for which to recover the lease. |
 | [`-retries` *num-retries*] | Number of times the client will retry calling 
recoverLease. The default number of retries is 1. |
 
-Recover the lease on the specified path. The path must reside on an HDFS 
filesystem. The default number of retries is 1.
+Recover the lease on the specified path. The path must reside on an HDFS file 
system. The default number of retries is 1.
+
+dfsadmin with ViewFsOverloadScheme
+--
+
+Usage: `hdfs dfsadmin -fs  `
+
+| COMMAND\_OPTION | Description |
+|: |: |
+| `-fs` *child fs mount link URI* | Its a logical mount link path to child 
file system in ViewFS world. This uri typically formed as src mount link 
prefixed with fs.defaultFS. Please note, this is not an actual child file 
system uri, instead its a logical mount link uri pointing to actual child file 
system|
+
+Example command usage:
+   `hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
+
+In ViewFsOverloadScheme, we may have multiple child file systems as mount 
point mappings as shown in [ViewFsOverloadScheme 
Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic 
parameter supported by dfsadmin. When users want to execute commands on one of 
the child file system, they need to pass that file system mount mapping link 
uri to -fs option. Let's take an example mount link configuration and dfsadmin 
command below.
+
+Mount link:
+
+```xml
+
+  fs.defaultFS
+  hdfs://MyCluster1
+
+
+
+  fs.viewfs.mounttable.MyCluster1./user
+  hdfs://MyCluster2/user
+   hdfs://MyCluster2/user
+   mount link path: /user
+   mount link uri: hdfs://MyCluster1/user
+   mount target uri for /user: hdfs://MyCluster2/user -->
+
+```
+
+If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option 
(`-fs hdfs://MyCluster1/user`)
+Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve 
the passed (`-fs hdfs://MyCluster1/user`) to target fs 
(`hdfs://MyCluster2/user`).
+This way users can get the access to all hdfs child file systems in 
ViewFsOverloadScheme.
+If there is no `-fs` option provided, then it will try to connect to the 
configured fs.defaultFS cluster if a cluster running with the fs.defaultFS uri.
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
index f851ef6..52ad49c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
@@ -361,6 +361,12 @@ resume its work, it's a good idea to provision some sort 
of cron job to purge su
 
 Delegation tokens for the cluster to which you are submitting the job 
(including all mounted volumes for that cluster’s mount table), and for input 
and output paths to your map-reduce job (including all volumes mounted via 
mount tables for the specified input and output paths) are all handled 
automatically. In addition, there is a way to add additional delegation tokens 
to the base cluster configuration for special circumstances.
 
+Don't want to change scheme or difficult to copy mount-table configurations to 
all clients?
+---
+
+Please refer to the [View File System Overload Scheme

[hadoop] branch branch-3.2 updated: HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new aa8de2f  HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme 
in processPath. Contributed by Uma Maheswara Rao G.
aa8de2f is described below

commit aa8de2f43b49ac9ad328de39fd5e69dd703ba461
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 12 14:32:19 2020 -0700

HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit 785b1def959fab6b8b766410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
---
 .../java/org/apache/hadoop/fs/shell/FsUsage.java   |   3 +-
 .../hadoop/fs/viewfs/ViewFileSystemUtil.java   |  14 +-
 ...ViewFileSystemOverloadSchemeWithFSCommands.java | 173 +
 3 files changed, 188 insertions(+), 2 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
index 6596527..64aade3 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
@@ -128,7 +128,8 @@ class FsUsage extends FsCommand {
 
 @Override
 protected void processPath(PathData item) throws IOException {
-  if (ViewFileSystemUtil.isViewFileSystem(item.fs)) {
+  if (ViewFileSystemUtil.isViewFileSystem(item.fs)
+  || ViewFileSystemUtil.isViewFileSystemOverloadScheme(item.fs)) {
 ViewFileSystem viewFileSystem = (ViewFileSystem) item.fs;
 Map  fsStatusMap =
 ViewFileSystemUtil.getStatus(viewFileSystem, item.path);
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
index c8a1d78..f486a10 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemUtil.java
@@ -52,6 +52,17 @@ public final class ViewFileSystemUtil {
   }
 
   /**
+   * Check if the FileSystem is a ViewFileSystemOverloadScheme.
+   *
+   * @param fileSystem
+   * @return true if the fileSystem is ViewFileSystemOverloadScheme
+   */
+  public static boolean isViewFileSystemOverloadScheme(
+  final FileSystem fileSystem) {
+return fileSystem instanceof ViewFileSystemOverloadScheme;
+  }
+
+  /**
* Get FsStatus for all ViewFsMountPoints matching path for the given
* ViewFileSystem.
*
@@ -93,7 +104,8 @@ public final class ViewFileSystemUtil {
*/
   public static Map getStatus(
   FileSystem fileSystem, Path path) throws IOException {
-if (!isViewFileSystem(fileSystem)) {
+if (!(isViewFileSystem(fileSystem)
+|| isViewFileSystemOverloadScheme(fileSystem))) {
   throw new UnsupportedFileSystemException("FileSystem '"
   + fileSystem.getUri() + "'is not a ViewFileSystem.");
 }
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
new file mode 100644
index 000..a974377
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithFSCommands.java
@@ -0,0 +1,173 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.tools;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;

[hadoop] branch branch-3.2 updated: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new e804218  HDFS-15389. DFSAdmin should close filesystem and dfsadmin 
-setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by 
Ayush Saxena
e804218 is described below

commit e80421820a2dde7a59a470dbae60882e55102e73
Author: Ayush Saxena 
AuthorDate: Sat Jun 6 10:49:38 2020 +0530

HDFS-15389. DFSAdmin should close filesystem and dfsadmin 
-setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by 
Ayush Saxena

(cherry picked from commit cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
(cherry picked from commit bee2846bee4ae676bdc14585f8a3927a9dd7df37)
---
 .../java/org/apache/hadoop/hdfs/tools/DFSAdmin.java  | 13 +++--
 ...TestViewFileSystemOverloadSchemeWithDFSAdmin.java | 20 
 2 files changed, 23 insertions(+), 10 deletions(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index 5fdc835..ae66064 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -479,9 +479,9 @@ public class DFSAdmin extends FsShell {
   public DFSAdmin(Configuration conf) {
 super(conf);
   }
-  
+
   protected DistributedFileSystem getDFS() throws IOException {
-return AdminHelper.getDFS(getConf());
+return AdminHelper.checkAndGetDFS(getFS(), getConf());
   }
   
   /**
@@ -1036,14 +1036,7 @@ public class DFSAdmin extends FsShell {
   System.err.println("Bandwidth should be a non-negative integer");
   return exitCode;
 }
-
-FileSystem fs = getFS();
-if (!(fs instanceof DistributedFileSystem)) {
-  System.err.println("FileSystem is " + fs.getUri());
-  return exitCode;
-}
-
-DistributedFileSystem dfs = (DistributedFileSystem) fs;
+DistributedFileSystem dfs = getDFS();
 try{
   dfs.setBalancerBandwidth(bandwidth);
   System.out.println("Balancer bandwidth is set to " + bandwidth);
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
index 1961dc2..a9475dd 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
@@ -263,4 +263,24 @@ public class TestViewFileSystemOverloadSchemeWithDFSAdmin {
 assertOutMsg("Disallowing snapshot on / succeeded", 1);
 assertEquals(0, ret);
   }
+
+  /**
+   * Tests setBalancerBandwidth with ViewFSOverloadScheme.
+   */
+  @Test
+  public void testSetBalancerBandwidth() throws Exception {
+final Path hdfsTargetPath = new Path(defaultFSURI + HDFS_USER_FOLDER);
+addMountLinks(defaultFSURI.getAuthority(),
+new String[] {HDFS_USER_FOLDER, LOCAL_FOLDER },
+new String[] {hdfsTargetPath.toUri().toString(),
+localTargetDir.toURI().toString() },
+conf);
+final DFSAdmin dfsAdmin = new DFSAdmin(conf);
+redirectStream();
+int ret = ToolRunner.run(dfsAdmin,
+new String[] {"-fs", defaultFSURI.toString(), "-setBalancerBandwidth",
+"1000"});
+assertOutMsg("Balancer bandwidth is set to 1000", 0);
+assertEquals(0, ret);
+  }
 }
\ No newline at end of file


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.2 updated: HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 84ceb6d  HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's 
scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
84ceb6d is described below

commit 84ceb6d5204b08e0044a592d4b21625b57e015f6
Author: Uma Maheswara Rao G 
AuthorDate: Thu May 21 21:34:58 2020 -0700

HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and 
target uris schemes are same. Contributed by Uma Maheswara Rao G.

(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
(cherry picked from commit 8e71e85af70c17f2350f794f8bc2475eb1e3acea)
---
 .../org/apache/hadoop/fs/viewfs/ConfigUtil.java|  15 ++-
 .../java/org/apache/hadoop/fs/viewfs/FsGetter.java |  47 
 .../fs/viewfs/HCFSMountTableConfigLoader.java  |   3 +-
 .../org/apache/hadoop/fs/viewfs/NflyFSystem.java   |  29 -
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  24 +---
 .../hadoop/fs/viewfs/ViewFileSystemBaseTest.java   |   1 -
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  28 -
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 121 +
 8 files changed, 230 insertions(+), 38 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
index 4c3dae9..6dd1f65 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
@@ -136,6 +136,17 @@ public class ConfigUtil {
   }
 
   /**
+   * Add nfly link to configuration for the given mount table.
+   */
+  public static void addLinkNfly(Configuration conf, String mountTableName,
+  String src, String settings, final String targets) {
+conf.set(
+getConfigViewFsPrefix(mountTableName) + "."
++ Constants.CONFIG_VIEWFS_LINK_NFLY + "." + settings + "." + src,
+targets);
+  }
+
+  /**
*
* @param conf
* @param mountTableName
@@ -149,9 +160,7 @@ public class ConfigUtil {
 settings = settings == null
 ? "minReplication=2,repairOnRead=true"
 : settings;
-
-conf.set(getConfigViewFsPrefix(mountTableName) + "." +
-Constants.CONFIG_VIEWFS_LINK_NFLY + "." + settings + "." + src,
+addLinkNfly(conf, mountTableName, src, settings,
 StringUtils.uriToString(targets));
   }
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
new file mode 100644
index 000..071af11
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/FsGetter.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.net.URI;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+/**
+ * File system instance getter.
+ */
+@Private
+class FsGetter {
+
+  /**
+   * Gets new file system instance of given uri.
+   */
+  public FileSystem getNewInstance(URI uri, Configuration conf)
+  throws IOException {
+return FileSystem.newInstance(uri, conf);
+  }
+
+  /**
+   * Gets file system instance of given uri.
+   */
+  public FileSystem get(URI uri, Configuration conf) throws IOException {
+return FileSystem.get(uri, conf);
+  }
+}
\ No newline at end of file
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
index c7e5aab..3968e36 100644
---

[hadoop] branch branch-3.2 updated: HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 852ee71  HDFS-15321. Make DFSAdmin tool to work with 
ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
852ee71 is described below

commit 852ee713548744a0709e03a8a743cbe69e436764
Author: Uma Maheswara Rao G 
AuthorDate: Tue Jun 2 11:09:26 2020 -0700

HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. 
Contributed by Uma Maheswara Rao G.

(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
(cherry picked from commit 0b5e202614f0bc20a0db6656f924fa4d2741d00c)
---
 .../fs/viewfs/ViewFileSystemOverloadScheme.java|  29 +++
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |   2 +-
 .../org/apache/hadoop/hdfs/tools/AdminHelper.java  |  25 +-
 .../org/apache/hadoop/hdfs/tools/DFSAdmin.java |  13 +-
 ...stViewFileSystemOverloadSchemeWithDFSAdmin.java | 266 +
 5 files changed, 317 insertions(+), 18 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
index f5952d5..36f9cd1 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystemOverloadScheme.java
@@ -17,6 +17,7 @@
  */
 package org.apache.hadoop.fs.viewfs;
 
+import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
@@ -27,6 +28,7 @@ import org.apache.hadoop.classification.InterfaceStability;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.UnsupportedFileSystemException;
 
 /**
@@ -227,4 +229,31 @@ public class ViewFileSystemOverloadScheme extends 
ViewFileSystem {
 
   }
 
+  /**
+   * This is an admin only API to give access to its child raw file system, if
+   * the path is link. If the given path is an internal directory(path is from
+   * mount paths tree), it will initialize the file system of given path uri
+   * directly. If path cannot be resolved to any internal directory or link, it
+   * will throw NotInMountpointException. Please note, this API will not return
+   * chrooted file system. Instead, this API will get actual raw file system
+   * instances.
+   *
+   * @param path - fs uri path
+   * @param conf - configuration
+   * @throws IOException
+   */
+  public FileSystem getRawFileSystem(Path path, Configuration conf)
+  throws IOException {
+InodeTree.ResolveResult res;
+try {
+  res = fsState.resolve(getUriPath(path), true);
+  return res.isInternalDir() ? fsGetter().get(path.toUri(), conf)
+  : ((ChRootedFileSystem) res.targetFileSystem).getMyFs();
+} catch (FileNotFoundException e) {
+  // No link configured with passed path.
+  throw new NotInMountpointException(path,
+  "No link found for the given path.");
+}
+  }
+
 }
\ No newline at end of file
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
index f051c9c..efced73 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java
@@ -192,7 +192,7 @@ public class ViewFsTestSetup {
* Adds the given mount links to the configuration. Mount link mappings are
* in sources, targets at their respective index locations.
*/
-  static void addMountLinksToConf(String mountTable, String[] sources,
+  public static void addMountLinksToConf(String mountTable, String[] sources,
   String[] targets, Configuration config) throws URISyntaxException {
 for (int i = 0; i < sources.length; i++) {
   String src = sources[i];
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
index 9cb646b..27cdf70 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/AdminHelper.java
@@ -1,4 +1,5 @@
 /**
+
  * Licensed

[hadoop] branch branch-3.2 updated: HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 7cf9601  HADOOP-17024. ListStatus on ViewFS root (ls "/") should list 
the linkFallBack root (configured target root). Contributed by Abhishek Das.
7cf9601 is described below

commit 7cf96019870a3042ae7dcf1a3575c625703404a7
Author: Abhishek Das 
AuthorDate: Mon May 18 22:27:12 2020 -0700

HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the 
linkFallBack root (configured target root). Contributed by Abhishek Das.

(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
(cherry picked from commit 5b248de42d2ae42710531a1514a21d60a1fca4b2)
---
 .../org/apache/hadoop/fs/viewfs/InodeTree.java | 13 +++
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java| 49 ++-
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   | 51 ++-
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 98 ++
 4 files changed, 209 insertions(+), 2 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
index 6992343..50c839b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
@@ -123,6 +123,7 @@ abstract class InodeTree {
 private final Map> children = new HashMap<>();
 private T internalDirFs =  null; //filesystem of this internal directory
 private boolean isRoot = false;
+private INodeLink fallbackLink = null;
 
 INodeDir(final String pathToNode, final UserGroupInformation aUgi) {
   super(pathToNode, aUgi);
@@ -149,6 +150,17 @@ abstract class InodeTree {
   return isRoot;
 }
 
+INodeLink getFallbackLink() {
+  return fallbackLink;
+}
+
+void addFallbackLink(INodeLink link) throws IOException {
+  if (!isRoot) {
+throw new IOException("Fallback link can only be added for root");
+  }
+  this.fallbackLink = link;
+}
+
 Map> getChildren() {
   return Collections.unmodifiableMap(children);
 }
@@ -580,6 +592,7 @@ abstract class InodeTree {
 }
   }
   rootFallbackLink = fallbackLink;
+  getRootDir().addFallbackLink(rootFallbackLink);
 }
 
 if (!gotMountTableEntry) {
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index a13b6ea..f626ffe 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1161,10 +1161,19 @@ public class ViewFileSystem extends FileSystem {
 }
 
 
+/**
+ * {@inheritDoc}
+ *
+ * Note: listStatus on root("/") considers listing from fallbackLink if
+ * available. If the same directory name is present in configured mount
+ * path as well as in fallback link, then only the configured mount path
+ * will be listed in the returned result.
+ */
 @Override
 public FileStatus[] listStatus(Path f) throws AccessControlException,
 FileNotFoundException, IOException {
   checkPathIsSlash(f);
+  FileStatus[] fallbackStatuses = listStatusForFallbackLink();
   FileStatus[] result = new 
FileStatus[theInternalDir.getChildren().size()];
   int i = 0;
   for (Entry> iEntry :
@@ -1187,7 +1196,45 @@ public class ViewFileSystem extends FileSystem {
 myUri, null));
 }
   }
-  return result;
+  if (fallbackStatuses.length > 0) {
+return consolidateFileStatuses(fallbackStatuses, result);
+  } else {
+return result;
+  }
+}
+
+private FileStatus[] consolidateFileStatuses(FileStatus[] fallbackStatuses,
+FileStatus[] mountPointStatuses) {
+  ArrayList result = new ArrayList<>();
+  Set pathSet = new HashSet<>();
+  for (FileStatus status : mountPointStatuses) {
+result.add(status);
+pathSet.add(status.getPath().getName());
+  }
+  for (FileStatus status : fallbackStatuses) {
+if (!pathSet.contains(status.getPath().getName())) {
+  result.add(status);
+}
+  }
+  return result.toArray(new FileStatus[0]);
+}
+
+private FileStatus[] listStatusForFallbackLink() throws IOException {
+  if (theInternalDir.isRoot() &&
+  theInternalDir.getFallbackLink() != null) {
+FileSystem linkedFs =
+

[hadoop] branch branch-3.2 updated: HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
 new 3dcc9ae  HDFS-15306. Make mount-table to read from central place ( 
Let's say from HDFS). Contributed by Uma Maheswara Rao G.
3dcc9ae is described below

commit 3dcc9aed4d7ccd4fd495a206dbe614a9706db4e1
Author: Uma Maheswara Rao G 
AuthorDate: Thu May 14 17:29:35 2020 -0700

HDFS-15306. Make mount-table to read from central place ( Let's say from 
HDFS). Contributed by Uma Maheswara Rao G.

(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
(cherry picked from commit 544996c85702af7ae241ef2f18e2597e2b4050be)
---
 .../org/apache/hadoop/fs/viewfs/Constants.java |   5 +
 .../fs/viewfs/HCFSMountTableConfigLoader.java  | 122 ++
 .../hadoop/fs/viewfs/MountTableConfigLoader.java   |  44 +
 .../fs/viewfs/ViewFileSystemOverloadScheme.java| 180 -
 .../org/apache/hadoop/fs/viewfs/package-info.java  |  26 +++
 .../fs/viewfs/TestHCFSMountTableConfigLoader.java  | 165 +++
 ...iewFSOverloadSchemeCentralMountTableConfig.java |  77 +
 ...iewFileSystemOverloadSchemeLocalFileSystem.java |  47 --
 .../apache/hadoop/fs/viewfs/ViewFsTestSetup.java   |  71 +++-
 ...FSOverloadSchemeWithMountTableConfigInHDFS.java |  68 
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java | 125 +-
 11 files changed, 797 insertions(+), 133 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
index 37f1a16..0a5d4b4 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
@@ -30,6 +30,11 @@ public interface Constants {
* Prefix for the config variable prefix for the ViewFs mount-table
*/
   public static final String CONFIG_VIEWFS_PREFIX = "fs.viewfs.mounttable";
+
+  /**
+   * Prefix for the config variable for the ViewFs mount-table path.
+   */
+  String CONFIG_VIEWFS_MOUNTTABLE_PATH = CONFIG_VIEWFS_PREFIX + ".path";
  
   /**
* Prefix for the home dir for the mount table - if not specified
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
new file mode 100644
index 000..c7e5aab
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation for Apache Hadoop compatible file system based mount-table
+ * file loading.
+ */
+public class HCFSMountTableConfigLoader implements MountTableConfigLoader {
+  private static final String REGEX_DOT = "[.]";
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(HCFSMountTableConfigLoader.class);
+  private Path mountTable = null;
+
+  /**
+   * Loads the mount-table configuration from hadoop compatible file system and
+   * add the configuration items to given configuration. Mount-table
+   * configuration format should be suffixed with version number.
+   * Format: mount-table..xml
+   * Example: mount-table.1.xml
+   * When user wants to update mount-table, the expectation is to upload new
+   * mount-table configuration file with monotonically increasing integer as
+   * version number.

[hadoop] branch branch-3.3 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 81e33d2  HDFS-15429. mkdirs should work when parent dir is an 
internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
81e33d2 is described below

commit 81e33d22a0d83abc88e3cd2411a5f198430800c4
Author: Uma Maheswara Rao G 
AuthorDate: Fri Jun 26 01:29:38 2020 -0700

HDFS-15429. mkdirs should work when parent dir is an internalDir and 
fallback configured. Contributed by Uma Maheswara Rao G.

(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  25 ++
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  28 +-
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 229 +---
 .../hadoop/fs/viewfs/TestViewFsLinkFallback.java   | 297 +
 4 files changed, 542 insertions(+), 37 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 06052b8..56448cb 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1339,6 +1339,31 @@ public class ViewFileSystem extends FileSystem {
   dir.toString().substring(1))) {
 return true; // this is the stupid semantics of FileSystem
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+FileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+
+try {
+  return linkedFallbackFs.mkdirs(dirToCreate, permission);
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg =
+new StringBuilder("Failed to create ").append(dirToCreate)
+.append(" at fallback : ")
+.append(linkedFallbackFs.getUri());
+LOG.debug(msg.toString(), e);
+  }
+  return false;
+}
+  }
+
   throw readOnlyMountTable("mkdirs",  dir);
 }
 
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index d18233a..c769003 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -1134,11 +1134,35 @@ public class ViewFs extends AbstractFileSystem {
 
 @Override
 public void mkdir(final Path dir, final FsPermission permission,
-final boolean createParent) throws AccessControlException,
-FileAlreadyExistsException {
+final boolean createParent) throws IOException {
   if (theInternalDir.isRoot() && dir == null) {
 throw new FileAlreadyExistsException("/ already exits");
   }
+
+  if (this.fsState.getRootFallbackLink() != null) {
+AbstractFileSystem linkedFallbackFs =
+this.fsState.getRootFallbackLink().getTargetFileSystem();
+Path parent = Path.getPathWithoutSchemeAndAuthority(
+new Path(theInternalDir.fullPath));
+String leafChild = (InodeTree.SlashPath.equals(dir)) ?
+InodeTree.SlashPath.toString() :
+dir.getName();
+Path dirToCreate = new Path(parent, leafChild);
+try {
+  // We are here because, the parent dir already exist in the mount
+  // table internal tree. So, let's create parent always in fallback.
+  linkedFallbackFs.mkdir(dirToCreate, permission, true);
+  return;
+} catch (IOException e) {
+  if (LOG.isDebugEnabled()) {
+StringBuilder msg = new StringBuilder("Failed to create {}")
+.append(" at fallback fs : {}");
+LOG.debug(msg.toString(), dirToCreate, linkedFallbackFs.getUri());
+  }
+  throw e;
+}
+  }
+
   throw readOnlyMountTable("mkdir", dir);
 }
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java
index

[hadoop] branch branch-3.3 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 5f67c3f  HDFS-15418. ViewFileSystemOverloadScheme should represent 
mount links as non symlinks. Contributed by Uma Maheswara Rao G.
5f67c3f is described below

commit 5f67c3f3ca49d718d8fc7c1914c3d2b77b3462f0
Author: Uma Maheswara Rao G 
AuthorDate: Sat Jun 20 00:32:02 2020 -0700

HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as 
non symlinks. Contributed by Uma Maheswara Rao G.

(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
---
 .../org/apache/hadoop/fs/viewfs/Constants.java |   8 ++
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  71 +++
 .../fs/viewfs/ViewFileSystemOverloadScheme.java|  20 +++-
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  80 -
 .../viewfs/TestViewFsOverloadSchemeListStatus.java | 132 +
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java |   4 +-
 .../src/site/markdown/ViewFsOverloadScheme.md  |  42 +++
 ...SystemOverloadSchemeHdfsFileSystemContract.java |   5 +
 ...ViewFileSystemOverloadSchemeWithHdfsScheme.java |   9 ++
 9 files changed, 295 insertions(+), 76 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
index 0a5d4b4..f454f63 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
@@ -90,4 +90,12 @@ public interface Constants {
   String CONFIG_VIEWFS_ENABLE_INNER_CACHE = "fs.viewfs.enable.inner.cache";
 
   boolean CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT = true;
+
+  /**
+   * Enable ViewFileSystem to show mountlinks as symlinks.
+   */
+  String CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS =
+  "fs.viewfs.mount.links.as.symlinks";
+
+  boolean CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT = true;
 }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 895edc0..1ee06e0 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -20,6 +20,8 @@ package org.apache.hadoop.fs.viewfs;
 import static 
org.apache.hadoop.fs.impl.PathCapabilitiesSupport.validatePathCapabilityArgs;
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
 import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE_DEFAULT;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS;
+import static 
org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_MOUNT_LINKS_AS_SYMLINKS_DEFAULT;
 import static org.apache.hadoop.fs.viewfs.Constants.PERMISSION_555;
 
 import java.io.FileNotFoundException;
@@ -527,10 +529,18 @@ public class ViewFileSystem extends FileSystem {
* the target path FileStatus object. The target path will be available via
* getSymlink on that children's FileStatus object. Since it represents as
* symlink, isDirectory on that children's FileStatus will return false.
+   * This behavior can be changed by setting an advanced configuration
+   * fs.viewfs.mount.links.as.symlinks to false. In this case, mount points 
will
+   * be represented as non-symlinks and all the file/directory attributes like
+   * permissions, isDirectory etc will be assigned from it's resolved target
+   * directory/file.
*
* If you want to get the FileStatus of target path for that children, you 
may
* want to use GetFileStatus API with that children's symlink path. Please 
see
* {@link ViewFileSystem#getFileStatus(Path f)}
+   *
+   * Note: In ViewFileSystem, by default the mount links are represented as
+   * symlinks.
*/
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
@@ -1114,6 +1124,7 @@ public class ViewFileSystem extends FileSystem {
 final long creationTime; // of the the mount table
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
+private final boolean showMountLinksAsSymlinks;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
@@ -1127,6 +1138,9 @@ public class ViewFileSystem extends FileSystem {
   theInternalDir = dir;
   creationTime = cTime;
   this.ugi = ugi;
+

[hadoop] branch branch-3.3 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 29a8ee4  HDFS-15427. Merged ListStatus with Fallback target filesystem 
and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
29a8ee4 is described below

commit 29a8ee4be639a18648df34a64bee6b413d1dcaf7
Author: Uma Maheswara Rao G 
AuthorDate: Tue Jun 23 01:42:25 2020 -0700

HDFS-15427. Merged ListStatus with Fallback target filesystem and 
InternalDirViewFS. Contributed by Uma Maheswara Rao G.

(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
---
 .../org/apache/hadoop/fs/viewfs/InodeTree.java |   4 +-
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  89 
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  94 +---
 .../fs/viewfs/TestViewFileSystemLinkFallback.java  | 251 -
 4 files changed, 360 insertions(+), 78 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
index 50c839b..d1e5d3a 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
@@ -374,7 +374,7 @@ abstract class InodeTree {
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(INodeDir dir)
-  throws URISyntaxException;
+  throws URISyntaxException, IOException;
 
   protected abstract T getTargetFileSystem(String settings, URI[] mergeFsURIs)
   throws UnsupportedFileSystemException, URISyntaxException, IOException;
@@ -393,7 +393,7 @@ abstract class InodeTree {
 return rootFallbackLink != null;
   }
 
-  private INodeLink getRootFallbackLink() {
+  protected INodeLink getRootFallbackLink() {
 Preconditions.checkState(root.isInternalDir());
 return rootFallbackLink;
   }
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 1ee06e0..06052b8 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -290,8 +290,9 @@ public class ViewFileSystem extends FileSystem {
 
 @Override
 protected FileSystem getTargetFileSystem(final INodeDir 
dir)
-  throws URISyntaxException {
-  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, 
config);
+throws URISyntaxException {
+  return new InternalDirOfViewFs(dir, creationTime, ugi, myUri, config,
+  this);
 }
 
 @Override
@@ -518,10 +519,10 @@ public class ViewFileSystem extends FileSystem {
   /**
* {@inheritDoc}
*
-   * Note: listStatus on root("/") considers listing from fallbackLink if
-   * available. If the same directory name is present in configured mount path
-   * as well as in fallback link, then only the configured mount path will be
-   * listed in the returned result.
+   * Note: listStatus considers listing from fallbackLink if available. If the
+   * same directory path is present in configured mount path as well as in
+   * fallback fs, then only the fallback path will be listed in the returned
+   * result except for link.
*
* If any of the the immediate children of the given path f is a 
symlink(mount
* link), the returned FileStatus object of that children would be 
represented
@@ -1125,11 +1126,13 @@ public class ViewFileSystem extends FileSystem {
 final UserGroupInformation ugi; // the user/group of user who created 
mtable
 final URI myUri;
 private final boolean showMountLinksAsSymlinks;
+private InodeTree fsState;
 
 public InternalDirOfViewFs(final InodeTree.INodeDir dir,
 final long cTime, final UserGroupInformation ugi, URI uri,
-Configuration config) throws URISyntaxException {
+Configuration config, InodeTree fsState) throws URISyntaxException {
   myUri = uri;
+  this.fsState = fsState;
   try {
 initialize(myUri, config);
   } catch (IOException e) {
@@ -1225,7 +1228,8 @@ public class ViewFileSystem extends FileSystem {
 FileNotFoundException, IOException {
   checkPathIsSlash(f);
   FileStatus[] fallbackStatuses = listStatusForFallbackLink();
-  FileStatus[] result = new 
FileStatus[theInternalDir.getChildren().size()];
+  Set linkStatuses = new HashSet<>();
+  Set internalDirStatuses = new HashSet<>();

[hadoop] branch branch-3.3 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 3cddd0b  HADOOP-17060. Clarify listStatus and getFileStatus behaviors 
inconsistent in the case of ViewFs implementation for isDirectory. Contributed 
by Uma Maheswara Rao G.
3cddd0b is described below

commit 3cddd0be29afc3405b33e59e45c17e7564239b74
Author: Uma Maheswara Rao G 
AuthorDate: Wed Jun 10 15:00:02 2020 -0700

HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent 
in the case of ViewFs implementation for isDirectory. Contributed by Uma 
Maheswara Rao G.

(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java| 36 --
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   | 24 +++
 .../src/main/java/org/apache/hadoop/fs/Hdfs.java   | 22 +
 .../apache/hadoop/hdfs/DistributedFileSystem.java  | 25 ---
 4 files changed, 94 insertions(+), 13 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 56d0fc5..895edc0 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -488,6 +488,14 @@ public class ViewFileSystem extends FileSystem {
 : new ViewFsFileStatus(orig, qualified);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a symlink(mount link), the path will be resolved to a
+   * target path and it will get the resolved path's FileStatus object. It will
+   * not be represented as a symlink and isDirectory API returns true if the
+   * resolved path is a directory, false otherwise.
+   */
   @Override
   public FileStatus getFileStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -505,6 +513,25 @@ public class ViewFileSystem extends FileSystem {
 res.targetFileSystem.access(res.remainingPath, mode);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * Note: listStatus on root("/") considers listing from fallbackLink if
+   * available. If the same directory name is present in configured mount path
+   * as well as in fallback link, then only the configured mount path will be
+   * listed in the returned result.
+   *
+   * If any of the the immediate children of the given path f is a 
symlink(mount
+   * link), the returned FileStatus object of that children would be 
represented
+   * as a symlink. It will not be resolved to the target path and will not get
+   * the target path FileStatus object. The target path will be available via
+   * getSymlink on that children's FileStatus object. Since it represents as
+   * symlink, isDirectory on that children's FileStatus will return false.
+   *
+   * If you want to get the FileStatus of target path for that children, you 
may
+   * want to use GetFileStatus API with that children's symlink path. Please 
see
+   * {@link ViewFileSystem#getFileStatus(Path f)}
+   */
   @Override
   public FileStatus[] listStatus(final Path f) throws AccessControlException,
   FileNotFoundException, IOException {
@@ -1174,20 +1201,11 @@ public class ViewFileSystem extends FileSystem {
   checkPathIsSlash(f);
   return new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-
   new Path(theInternalDir.fullPath).makeQualified(
   myUri, ROOT_PATH));
 }
 
 
-/**
- * {@inheritDoc}
- *
- * Note: listStatus on root("/") considers listing from fallbackLink if
- * available. If the same directory name is present in configured mount
- * path as well as in fallback link, then only the configured mount path
- * will be listed in the returned result.
- */
 @Override
 public FileStatus[] listStatus(Path f) throws AccessControlException,
 FileNotFoundException, IOException {
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index df10dce..4578a4c 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -351,6 +351,14 @@ public class ViewFs extends AbstractFileSystem {
 return res.targetFileSystem.getFileChecksum(res.remainingPath);
   }
 
+  /**
+   * {@inheritDoc}
+   *
+   * If the given path is a

[hadoop] branch branch-3.3 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new c3bef49  HADOOP-17029. Return correct permission and owner for listing 
on internal directories in ViewFs. Contributed by Abhishek Das.
c3bef49 is described below

commit c3bef4906c34d7a97493f2bbdd3dc35e08324520
Author: Abhishek Das 
AuthorDate: Fri Jun 5 14:56:51 2020 -0700

HADOOP-17029. Return correct permission and owner for listing on internal 
directories in ViewFs. Contributed by Abhishek Das.

(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
---
 .../apache/hadoop/fs/viewfs/ViewFileSystem.java|  27 +++--
 .../java/org/apache/hadoop/fs/viewfs/ViewFs.java   |  41 +--
 .../hadoop/fs/viewfs/TestViewfsFileStatus.java | 118 -
 3 files changed, 146 insertions(+), 40 deletions(-)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index 2fde078..ddb3f2b 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1200,13 +1200,26 @@ public class ViewFileSystem extends FileSystem {
 INode inode = iEntry.getValue();
 if (inode.isLink()) {
   INodeLink link = (INodeLink) inode;
-
-  result[i++] = new FileStatus(0, false, 0, 0,
-creationTime, creationTime, PERMISSION_555,
-ugi.getShortUserName(), ugi.getPrimaryGroupName(),
-link.getTargetLink(),
-new Path(inode.fullPath).makeQualified(
-myUri, null));
+  try {
+String linkedPath = link.getTargetFileSystem().getUri().getPath();
+FileStatus status =
+((ChRootedFileSystem)link.getTargetFileSystem())
+.getMyFs().getFileStatus(new Path(linkedPath));
+result[i++] = new FileStatus(status.getLen(), false,
+  status.getReplication(), status.getBlockSize(),
+  status.getModificationTime(), status.getAccessTime(),
+  status.getPermission(), status.getOwner(), status.getGroup(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  } catch (FileNotFoundException ex) {
+result[i++] = new FileStatus(0, false, 0, 0,
+  creationTime, creationTime, PERMISSION_555,
+  ugi.getShortUserName(), ugi.getPrimaryGroupName(),
+  link.getTargetLink(),
+  new Path(inode.fullPath).makeQualified(
+  myUri, null));
+  }
 } else {
   result[i++] = new FileStatus(0, true, 0, 0,
 creationTime, creationTime, PERMISSION_555,
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
index 607bdb8..df10dce 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
@@ -917,11 +917,25 @@ public class ViewFs extends AbstractFileSystem {
   if (inode.isLink()) {
 INodeLink inodelink = 
   (INodeLink) inode;
-result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
+try {
+  String linkedPath = inodelink.getTargetFileSystem()
+  .getUri().getPath();
+  FileStatus status = ((ChRootedFs)inodelink.getTargetFileSystem())
+  .getMyFs().getFileStatus(new Path(linkedPath));
+  result = new FileStatus(status.getLen(), false,
+status.getReplication(), status.getBlockSize(),
+status.getModificationTime(), status.getAccessTime(),
+status.getPermission(), status.getOwner(), status.getGroup(),
+inodelink.getTargetLink(),
+new Path(inode.fullPath).makeQualified(
+myUri, null));
+} catch (FileNotFoundException ex) {
+  result = new FileStatus(0, false, 0, 0, creationTime, creationTime,
 PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
 inodelink.getTargetLink(),
 new Path(inode.fullPath).makeQualified(
 myUri, null));
+}
   } else {
 result = new FileStatus(0, true, 0, 0, creationTime, creationTime,
   PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
@@ -976,12 +990,25 @@ public class ViewFs extends

[hadoop] branch branch-3.3 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

2020-06-27 Thread umamahesh

This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
 new 7b29019  HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.
7b29019 is described below

commit 7b29019eeae2622957c086c7240f57681088d622
Author: Ayush Saxena 
AuthorDate: Mon Jun 8 01:59:10 2020 +0530

HDFS-15396. Fix 
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. 
Contributed by Ayush Saxena.

(cherry picked from commit a8610c15c498531bf3c011f1b0ace8ef61f2)
---
 .../src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java  | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
index ddb3f2b..56d0fc5 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
@@ -1202,6 +1202,9 @@ public class ViewFileSystem extends FileSystem {
   INodeLink link = (INodeLink) inode;
   try {
 String linkedPath = link.getTargetFileSystem().getUri().getPath();
+if("".equals(linkedPath)) {
+  linkedPath = "/";
+}
 FileStatus status =
 ((ChRootedFileSystem)link.getTargetFileSystem())
 .getMyFs().getFileStatus(new Path(linkedPath));


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch trunk updated: HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.

2020-06-27 Thread ayushsaxena

This is an automated email from the ASF dual-hosted git repository.

ayushsaxena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
 new 8db38c9  HDFS-15378. 
TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on 
trunk. Contributed by hemanthboyina.
8db38c9 is described below

commit 8db38c98a6c6ce9215ea998a2f544b2eabca4340
Author: Ayush Saxena 
AuthorDate: Sat Jun 27 19:13:01 2020 +0530

HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight 
is failing on trunk. Contributed by hemanthboyina.
---
 .../test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
index b119e78..6156c3d 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
@@ -571,7 +571,8 @@ public class TestReconstructStripedFile {
   DataNodeFaultInjector.set(oldInjector);
   for (final DataNode curDn : cluster.getDataNodes()) {
 GenericTestUtils.waitFor(() -> curDn.getXceiverCount() <= 1, 10, 
6);
-assertEquals(0, curDn.getXmitsInProgress());
+GenericTestUtils.waitFor(() -> curDn.getXmitsInProgress() == 0, 10,
+2500);
   }
 }
   }


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

[hadoop] branch branch-3.1 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

[hadoop] branch branch-3.2 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

[hadoop] branch branch-3.3 updated: HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)

[hadoop] branch branch-3.1 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

[hadoop] branch branch-3.1 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

[hadoop] branch branch-3.1 updated: HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena

[hadoop] branch branch-3.1 updated: HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.1 updated: HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.

[hadoop] branch branch-3.1 updated: HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

[hadoop] branch branch-3.2 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

[hadoop] branch branch-3.2 updated: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena

[hadoop] branch branch-3.2 updated: HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.2 updated: HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.

[hadoop] branch branch-3.2 updated: HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.3 updated: HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.3 updated: HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.3 updated: HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.3 updated: HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.

[hadoop] branch branch-3.3 updated: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.

[hadoop] branch branch-3.3 updated: HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.

[hadoop] branch trunk updated: HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.

36 matches

Site Navigation

Mail list logo

Footer information