[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315941#comment-15315941 ] Hadoop QA commented on HDFS-10488: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 5s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s {color} | {color:red} hadoop-hdfs-project: The patch generated 23 new + 125 unchanged - 0 fixed = 148 total (was 125) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s {color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 38s {color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 94m 3s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestRenameWhileOpen | | | hadoop.hdfs.web.TestWebHdfsFileSystemContract | | | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:2c91fd8 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12808212/HDFS-10488.patch | | JIRA Issue | HDFS-10488 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e34a3157025f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 106234d | | Default Java | 1.8.0_91 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit test logs |
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316096#comment-15316096 ] Hadoop QA commented on HDFS-10488: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 56s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s {color} | {color:red} hadoop-hdfs-project: The patch generated 3 new + 161 unchanged - 1 fixed = 164 total (was 162) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s {color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 38s {color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 99m 13s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:2c91fd8 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12808220/HDFS-10488.002.patch | | JIRA Issue | HDFS-10488 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 12e2a27e107e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 106234d | | Default Java | 1.8.0_91 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15657/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15657/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316676#comment-15316676 ] Hadoop QA commented on HDFS-10488: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s {color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 161 unchanged - 1 fixed = 161 total (was 162) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s {color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 23s {color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 105m 11s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:2c91fd8 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12808388/HDFS-10488.003.patch | | JIRA Issue | HDFS-10488 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 289ed45f7f75 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 35f255b | | Default Java | 1.8.0_91 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15660/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15660/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/j
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316919#comment-15316919 ] John Zhuge commented on HDFS-10488: --- [~wchevreuil] Shouldn't you take the ownership of this jira? > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316976#comment-15316976 ] Wei-Chiu Chuang commented on HDFS-10488: Hi [~wchevreuil] thanks for the great finding and the patch. The fix itself looks good to me in general and I just have 1 question about the fix and some quick comments regarding the test: Regarding the fix: I am a little confused by the following line: seems you always call getDirFsPermission() regardless whether it's a dir or file? {code} np.setPermission(fullpath, permission.getDirFsPermission()); {code} # The convention of HDFS unit testing is to start a MiniDFSCluster using {{@Before}} annotation, and tear down the cluster using {{@After}}. You can take a look at other HDFS test files for reference. # Would you mind to add additional tests to verify it works as expected if the mask {{fs.permissions.umask-mode}} is set differently? This will be helpful in detecting other webhfs permission bugs if they exist. # You have fixed the WebHDFS CREATESYMLINK operation. Could you also add tests to verify this operation as well? > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316995#comment-15316995 ] Wei-Chiu Chuang commented on HDFS-10488: I believe he's not on the contributor list which is why he can't assign this to himself. > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317387#comment-15317387 ] Chris Nauroth commented on HDFS-10488: -- -1 for the proposed change. The umask ({{fs.permissions.umask-mode}}) is a concept that is applied at the client side by individual applications, usually via their usage of the FileSystem subclasses that implement a particular file system client. The umask is not applied by the API/protocol layer such as WebHDFS or NameNode RPC. As such, the behavior of the shell, which applies umask, is not always going to look consistent with the behavior of a direct curl WebHDFS call, which does not apply the umask. Using the shell to access WebHDFS gives consistent results, because the logic of the WebHdfsFileSystem class used by the shell will apply the umask. If this patch were committed, then it would become basically impossible to create files and directories with absolute permissions through WebHDFS. For example, suppose {{fs.permissions.umask-mode}} is set to 022, but an individual application has a desire to create a file with 775 permissions. This wouldn't work as expected, because server-side enforcement of the umask would restrict permissions on the resulting file to 755. The only way to work around this would be to reconfigure {{fs.permissions.umask-mode}} and restart the NameNode, which isn't operationally desirable. Worse than that, this would likely have the long-term effect of reducing {{fs.permissions.umask-mode}} to lowest common denominator, perhaps even 000, to accommodate all possible permissions at file creation time, thus weakening the benefit of umask as applied by client applications like the shell. As a final point against this change, please note that it could be considered backwards-incompatible. In my example above trying to create a file with 775 permissions, but the server-side umask forcing it to 755, it means that subsequent write actions by users in the same group will be unauthorized. This may break certain workflows. The area where there is a possibility for change is documentation to help raise user awareness of this. That could potentially go into the HDFS Permissions Guide page or the WebHDFS REST API page, or perhaps some combination of both. I would be happy to help review and +1 documentation changes. [~wchevreuil], despite my -1, thank you for writing up your experience with this and posting a patch. If you'd like to proceed with a documentation patch, please let me know, and I'll assign the issue to you. > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318433#comment-15318433 ] Wellington Chevreuil commented on HDFS-10488: - Thanks for the suggestions, [~jojochuang] and [~cnauroth] for the detailed explanation. I was going to upload another patch with pointers made by [~jojochuang], then I saw [~cnauroth] comments. So, [~cnauroth], summarizing, *fs.permissions.umask-mode* should not be applied for WebHDFS created directories/files. I think the explained intentions behind this is not clear on the documentation, so yes, I would like to work on a documentation patch for this. Additional points: 1) While working on this, I had found out the default permission (if no permission is specified while calling the method) for both directories and files created by WebHDFS currently is *755*. However, defining "execution" permissions for HDFS files don't have any value. Should this be changed to give different default permissions for files and directories? 2) Still on the default values, setting *755* as default can lead to confusion about umask being used. Since default umask is *022*, users can conclude that the umask is being applied when they see newly created directories got *755*. Should this be changed to more permissive permissions such as *777*? 3) When working on tests for WebHDFS CREATESYMLINK as suggested by [~jojochuang], I realized this method is no longer supported. Should we simply remove from WebHDFS, or only document this is not supported anymore and leave it giving the current error? > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319384#comment-15319384 ] Chris Nauroth commented on HDFS-10488: -- bq. So, Chris Nauroth, summarizing, fs.permissions.umask-mode should not be applied for WebHDFS created directories/files. I think a slight refinement of this is to say that it should not be applied by the WebHDFS server side (the NameNode). It may be applied by the WebHDFS client side. For example, the {{WebHdfsFileSystem}} class that ships in Hadoop does apply {{fs.permissions.umask-mode}} from the client side before calling the WebHDFS server side. bq. While working on this, I had found out the default permission (if no permission is specified while calling the method) for both directories and files created by WebHDFS currently is 755. However, defining "execution" permissions for HDFS files don't have any value. Should this be changed to give different default permissions for files and directories? This part is admittedly odd, and there is a long-standing open JIRA requesting a change to 644 as the default for files. That is HDFS-6434. This change is potentially backwards-incompatible, such as if someone has an existing workflow that round-trips a file through HDFS and expects it to be executable after getting it back out, though that's likely a remote edge case. If you'd like to proceed with HDFS-6434, then I'd suggest targeting trunk/Hadoop 3.x, where we currently can make backwards-incompatible changes. bq. Still on the default values, setting 755 as default can lead to confusion about umask being used. Since default umask is 022, users can conclude that the umask is being applied when they see newly created directories got 755. Should this be changed to more permissive permissions such as 777? I do think 777 makes sense from one perspective, but there is also a trade-off with providing behavior that is secure by default. In HDFS-2427, the project made the choice to go with 755, favoring secure default behavior (755) over the possibly more intuitive behavior (777). bq. When working on tests for WebHDFS CREATESYMLINK as suggested by Wei-Chiu Chuang, I realized this method is no longer supported. Should we simply remove from WebHDFS, or only document this is not supported anymore and leave it giving the current error? HDFS symlinks are currently in a state where the code is partially completed but dormant due to unresolved problems with backwards-compatibility and security. We might get past those hurdles someday, so I suggest leaving that code as is. We still run tests against the symlink code paths. This works by having the tests call the private {{FileSystem#enableSymlinks}} method to toggle on the dormant symlink code. > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353453#comment-15353453 ] Wellington Chevreuil commented on HDFS-10488: - Anyone has any additional comment on the last patch? > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.005.patch, HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[ https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353717#comment-15353717 ] Andrew Wang commented on HDFS-10488: The current 005 patch looks okay for branch-2, but since we just committed HDFS-6434, does this patch need to be updated for trunk? It mentions a file having default 755 permissions, but HDFS-6434 makes the default 644. The other note is that there's a "Permission" section in this doc that mentions 755 as the default for both files and dirs, this should also be updated. Would also appreciate if you could update the JIRA summary to reflect the contents of the patch. Thanks Wellington. > WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating > files/directories without specifying permissions > -- > > Key: HDFS-10488 > URL: https://issues.apache.org/jira/browse/HDFS-10488 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch, > HDFS-10488.005.patch, HDFS-10488.patch > > > WebHDFS methods for creating file/directories are always creating it with 755 > permissions as default, even ignoring any configured > *fs.permissions.umask-mode* in the case of directories. > Dfs CLI, however, applies the configured umask to 777 permission for > directories, or 666 permission for files. > Example below shows the different behaviour when creating directory via CLI > and WebHDFS: > {noformat} > 1) Creating a directory under '/test/' as 'test-user'. Configured > fs.permissions.umask-mode is 000: > $ sudo -u test-user hdfs dfs -mkdir /test/test-user1 > $ sudo -u test-user hdfs dfs -getfacl /test/test-user1 > # file: /test/test-user1 > # owner: test-user > # group: supergroup > user::rwx > group::rwx > other::rwx > 4) Doing the same via WebHDFS does not get the proper ACLs: > $ curl -i -X PUT > "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS"; > > $ sudo -u test-user hdfs dfs -getfacl /test/test-user2 > # file: /test/test-user2 > # owner: test-user > # group: supergroup > user::rwx > group::r-x > other::r-x > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org