[
https://issues.apache.org/jira/browse/HDFS-15578?focusedWorklogId=485426&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485426
]
ASF GitHub Bot logged work on HDFS-15578:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Sep/20 22:06
Start Date: 16/Sep/20 22:06
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on a change in pull request
#2305:
URL: https://github.com/apache/hadoop/pull/2305#discussion_r489780767
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestViewDistributedFileSystemWithMountLinks.java
##########
@@ -61,4 +64,55 @@ public void testCreateOnRoot() throws Exception {
public void testMountLinkWithNonExistentLink() throws Exception {
testMountLinkWithNonExistentLink(false);
}
+
+ @Test
+ public void testRenameOnInternalDirWithFallback() throws Exception {
+ Configuration conf = getConf();
+ URI defaultFSURI =
+ URI.create(conf.get(CommonConfigurationKeys.FS_DEFAULT_NAME_KEY));
+ final Path hdfsTargetPath1 = new Path(defaultFSURI + "/HDFSUser");
+ final Path hdfsTargetPath2 = new Path(defaultFSURI + "/NewHDFSUser/next");
+ ViewFsTestSetup.addMountLinksToConf(defaultFSURI.getAuthority(),
+ new String[] {"/HDFSUser", "/NewHDFSUser/next"},
+ new String[] {hdfsTargetPath1.toUri().toString(),
+ hdfsTargetPath2.toUri().toString()}, conf);
+ //Making sure parent dir structure as mount points available in fallback.
+ try (DistributedFileSystem dfs = new DistributedFileSystem()) {
+ dfs.initialize(defaultFSURI, conf);
+ dfs.mkdirs(hdfsTargetPath1);
+ dfs.mkdirs(hdfsTargetPath2);
+ }
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ Path src = new Path("/newFileOnRoot");
+ Path dst = new Path("/newFileOnRoot1");
+ fs.create(src).close();
+ verifyRename(fs, src, dst);
+
+ src = new Path("/newFileOnRoot1");
+ dst = new Path("/NewHDFSUser/newFileOnRoot");
+ fs.mkdirs(dst.getParent());
+ verifyRename(fs, src, dst);
+
+ src = new Path("/NewHDFSUser/newFileOnRoot");
+ dst = new Path("/NewHDFSUser/newFileOnRoot1");
+ verifyRename(fs, src, dst);
+
+ src = new Path("/NewHDFSUser/newFileOnRoot1");
+ dst = new Path("/newFileOnRoot");
+ verifyRename(fs, src, dst);
+
+ src = new Path("/HDFSUser/newFileOnRoot1");
+ dst = new Path("/HDFSUser/newFileOnRoot");
+ fs.create(src).close();
+ verifyRename(fs, src, dst);
+ }
+ }
+
+ private void verifyRename(FileSystem fs, Path src, Path dst)
+ throws IOException {
+ fs.rename(src, dst);
+ Assert.assertFalse(fs.exists(src));
+ Assert.assertTrue(fs.exists(dst));
+ }
Review comment:
What's your proposal for rename here? Could you elaborate a bit?
I am thinking this way: If API nature is to create parent dirs, it will
create. If the API nature is not create parents, I am ok to fail and let the
users know the fact that it's redirected to fallback and they don't have
structure?
In your above example, you made additional call to createFile, so that next
rename succeeded.
I see in ViewFS.java create, mkdir we made createParent by default true. I
am ok to take the user passed flag itself. That way we can be consistent on API
perspective. Also if I am correct, this problem only exist with ViewFs.java. In
ViewFileSystem.java as we will compliant to API as mkdirs by default create
parent. This way, users expected things only we do. What do you? If someone
really wants to createParents bydefault in ViewFs.java, that discussion can be
taken into separate JIRA for create,mkdir behavior. To summarize: in
ViewFs.java, create, mkdir creates the parent in fallback if they don't exist
(irrespective of createParent flag. ) We did this because, for the users
perspective, we provide transparency and exist of parent dir returns true.
However that fact is that, children actually goes to fallback. When users
create mount points with respective to fallback cluster, then only this will be
true. Otherwise it can be any random mount point without any relation to
fallback. When API does not say createParent, then we will keep that semantics
to fallback as well. Whats your thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 485426)
Time Spent: 2h (was: 1h 50m)
> Fix the rename issues with fallback fs enabled
> ----------------------------------------------
>
> Key: HDFS-15578
> URL: https://issues.apache.org/jira/browse/HDFS-15578
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, viewfsOverloadScheme
> Affects Versions: 3.4.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When we enable fallback, rename should success if the src.parent or
> dst.parent on inernalDir.
> {noformat}
> org.apache.hadoop.security.AccessControlException: InternalDir of
> ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
> of ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683) at
> org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
