[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386887#comment-15386887 ] John Zhuge edited comment on HDFS-6962 at 7/21/16 5:41 AM: --- Thanks [~eddyxu] for the review! Great comments. I will fix the issues.. bq. Is it true that after creation, the createMode is not useful for INode, as its mode has already been established? I feel that we do not need to store it as a feature in INode. Yes it is true, create modes are only needed during inode creation. It is a stretch to call CreateModes a feature, but it has be stored somewhere in inode or another data structure 1-1 tied to inode. Suggestions are welcome. was (Author: jzhuge): Thanks [~eddyxu] for the review! Great comments. I will fix the issues.. bq. Is it true that after creation, the createMode is not useful for INode, as its mode has already been established? I feel that we do not need to store it as a feature in INode. Yes it is true, create modes are only needed during inode creation. It is a stretch to call CreateModes a feature, but it has be stored somewhere in something hanging off inode or another data structure 1-1 tied to inode. Suggestions are welcome. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386399#comment-15386399 ] John Zhuge edited comment on HDFS-6962 at 7/20/16 10:40 PM: [~cnauroth] and [~eddyxu] Please review patch 007. Diff from 006: * Add CLI tests {{TestAclCLIWithPosixAclInheritance}} based on {{TestAclCLI}} * No longer add field {{createModes}} to {{INodeWithAdditionalFields}}. Instead, add new feature {{CreateModesFeature}} to store create modes. In this way, no penalty when POSIX ACL inheritance is disable or for any inode not being created. * Remove the feature {{CreateModesFeature}} once default ACL has been processed, so the feature only exists in inode for a short period of time. * There is cost of adding and removing the new feature. TODO: * Support webhdfs was (Author: jzhuge): [~cnauroth] and [~eddyxu] Please review patch 007. Diff from 006: * Add CLI tests {{TestAclCLIWithPosixAclInheritance}} based on {{TestAclCLI}} * No longer add field {{createModes}} to {{INodeWithAdditionalFields}}. Instead, add new feature {{CreateModesFeature}} to store create modes. In this way, no penalty when POSIX ACL inheritance is disable or for any inode not associated with a create request. * Remove the feature {{CreateModesFeature}} once default ACL has been processed. * There is added cost of adding and removing the new feature. TODO: * Support webhdfs > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: h
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359304#comment-15359304 ] John Zhuge edited comment on HDFS-6962 at 7/4/16 8:02 AM: -- Saw {{TestDataNodeLifeline}} and {{TestOfflineEditsViewer}} failures on my local Mac with stock trunk, so they are not regressions. {{TestMover,TestNNThroughputBenchmark,TestNNThroughputBenchmark,TestStripedINodeFile,TestRetryCacheWithHA}} failures are regressions. was (Author: jzhuge): Looking into the unit test failures. They are regressions. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400 ] John Zhuge edited comment on HDFS-6962 at 6/29/16 2:56 AM: --- Patch 004 passed system and compatibility tests on a pseudo cluster. Upload test logs. Focus on group WRITE permission. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) was (Author: jzhuge): Patch 004 passed system and compatibility tests on a pseudo cluster. Upload test logs. Focus on group permission, especially WRITE. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log, > disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: ha
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400 ] John Zhuge edited comment on HDFS-6962 at 6/29/16 2:48 AM: --- Patch 004 passed system and compatibility tests on a pseudo cluster. Upload test logs. Focus on group permission, especially WRITE. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) was (Author: jzhuge): Pass system and compatibility tests with Patch 004 on a pseudo cluster. Upload test logs. Focus on group permission, especially WRITE. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log, > disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdf
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400 ] John Zhuge edited comment on HDFS-6962 at 6/29/16 2:40 AM: --- Pass system and compatibility tests with Patch 004 on a pseudo cluster. Upload test logs. Focus on group permission, especially WRITE. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) was (Author: jzhuge): Upload system and compatibility test logs for Patch 004 on a pseudo cluster. Pay attention to group permission, especially WRITE bit. The test script is {{run}}, the command line is {code} test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag {code} Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance either enabled or disabled. There are 2 versions of test clients. The POSIX ACL inheritance is only in effect when the flag is true and the requests come from a compatible client. The following table shows the matrix for the log files. || Client Version\POSIX ACL Inheritance || Enabled || Disabled || | 2.6.0 | enabled_old_client.log | disabled_old_client.log | | 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log | In each log file, there is a test matrix of the following dimensions: * Parent dir has default ACL or not * Umask 002 or 027 * Create a file (create request) or a sub-directory (mkdirs request) > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log, > disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: h
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345214#comment-15345214 ] John Zhuge edited comment on HDFS-6962 at 6/24/16 5:20 AM: --- [~cnauroth] Are there automated tests for section {{Default Acls}} in test plan {{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new section in test plan to cover ACL inheritance. Currently these are ACL related tests: TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl. Will add a new test class {{namenode/TestAclInheritance}}. was (Author: jzhuge): [~cnauroth] Are there automated tests for section {{Default Acls}} in test plan {{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new section in test plan to cover ACL inheritance. Currently these are ACL related tests: TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,FSAclBaseTest,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl. Will add a new test class {{namenode/TestAclInheritance}}. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345214#comment-15345214 ] John Zhuge edited comment on HDFS-6962 at 6/22/16 9:44 PM: --- [~cnauroth] Are there automated tests for section {{Default Acls}} in test plan {{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new section in test plan to cover ACL inheritance. Currently these are ACL related tests: TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,FSAclBaseTest,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl. Will add a new test class {{namenode/TestAclInheritance}}. was (Author: jzhuge): [~cnauroth] Are there automated tests for section {{Default Acls}} in test plan {{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new section in test plan to cover ACL inheritance. Will add a new test class {{namenode/TestAclInheritance}}. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941 ] John Zhuge edited comment on HDFS-6962 at 6/11/16 6:41 AM: --- Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. Questions: * How to query NN config in {{copyINodeDefaultAcl}}? * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? TODO: * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * Add and run unit tests * Run system compatibility tests * Update {{HdfsPermissionsGuide.md}} [~cnauroth] Could you please take a look at Patch 002 to check whether backwards compatibility can be achieved with an NN flag and the changes are going in the right direction? Thanks. was (Author: jzhuge): Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. Questions: * How to query NN config in {{copyINodeDefaultAcl}}? * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? TODO: * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * Add and run unit tests * Run system compatibility tests * Update {{HdfsPermissionsGuide.md}} > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941 ] John Zhuge edited comment on HDFS-6962 at 6/11/16 6:39 AM: --- Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. Questions: * How to query NN config in {{copyINodeDefaultAcl}}? * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? TODO: * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * Add and run unit tests * Run system compatibility tests * Update {{HdfsPermissionsGuide.md}} was (Author: jzhuge): Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. Questions: * How to query NN config in {{copyINodeDefaultAcl}}? * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? TODO: * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * Add and run unit tests * Run system compatibility tests > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a s
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624 ] John Zhuge edited comment on HDFS-6962 at 6/11/16 6:37 AM: --- Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * Preserve the existing meaning of field {{FsPermissionProto masked}}: mode + umask. * Please note this approach is slightly different from what you suggested above. I am ok either way but find this a little less possibility for misunderstanding. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} // TODO: How to query NameNode config? boolean isPosixAclInheritanceEnabled = true; FsPermission masked = child.getMaskedPermission(); FsPermission unmasked = child.getUnmaskedPermission(); if (isPosixAclInheritanceEnabled && unmasked != null) { // // HDFS-6962: POSIX ACL inheritance // // Set permission to unmasked child.setPermission(unmasked); if (!copyINodeDefaultAclInternal(child)) { // No default ACL in parent dir // Set permission to masked child.setPermission(masked); } } else { // // Old behavior before HDFS-6962 // assert child.getFsPermission().equals(masked); copyINodeDefaultAclInternal(child); } {code} Here {{INode#getUnmaskedPermission}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedPermission}} returns the masked mode. was (Author: jzhuge): Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * Preserve the existing meaning of field {{FsPermissionProto masked}}: mode + umask. * Please note this approach is slightly different from what you suggested above. I am ok either way but find this a little less possibility for misunderstanding. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} if (child.isPosixAclInheritanceEnabled() == true) { // // HDFS-6962. POSIX ACL inheritance. // // Set permission to unmasked if necessary if (!child.getFsPermission().equals(child.getUnmaskedMode())) { child.setPermission(child.getUnmaskedMode()); } if (copyINodeDefaultAclInternal(child) == false) { // No default ACL in parent dir // Set permission to masked child.setPermission(child.getMaskedMode()); } } else { // // Old behavior before HDFS-6962. // // Set permission to masked if necessary if (!child.getFsPermission().equals(child.getMaskedMode())) { child.setPermission(child.getMaskedMode()); } copyINodeDefaultAclInternal(child); } {code} Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedMode}} returns the masked mode; {{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir request comes from a client with HDFS-6962 fix and {{dfs.namenode.posix.acl.inheritance.enabled}} is true. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everythin
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941 ] John Zhuge edited comment on HDFS-6962 at 6/10/16 6:20 AM: --- Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. Questions: * How to query NN config in {{copyINodeDefaultAcl}}? * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? TODO: * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * Add and run unit tests * Run system compatibility tests was (Author: jzhuge): Patch 002 (prototype): * Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store both masked and unmasked permissions. HDFS client uses it to sneak unmasked permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create (placed into PermissionStatus) -> FSNamesystem.startFile -> FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields. * Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance behavior. TODO: * Add and run unit tests * Run system compatibility tests * Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}? * Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories call tree * {{PermissionStatus#applyUMask}} never used, remove it? * {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if permission is null. Should use dir default permission? > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624 ] John Zhuge edited comment on HDFS-6962 at 6/7/16 5:07 AM: -- Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * Preserve the existing meaning of field {{FsPermissionProto masked}}: mode + umask. * Please note this approach is slightly different from what you suggested above. I am ok either way but find this a little less possibility for misunderstanding. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} if (child.isPosixAclInheritanceEnabled() == true) { // // HDFS-6962. POSIX ACL inheritance. // // Set permission to unmasked if necessary if (!child.getFsPermission().equals(child.getUnmaskedMode())) { child.setPermission(child.getUnmaskedMode()); } if (copyINodeDefaultAclInternal(child) == false) { // No default ACL in parent dir // Set permission to masked child.setPermission(child.getMaskedMode()); } } else { // // Old behavior before HDFS-6962. // // Set permission to masked if necessary if (!child.getFsPermission().equals(child.getMaskedMode())) { child.setPermission(child.getMaskedMode()); } copyINodeDefaultAclInternal(child); } {code} Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedMode}} returns the masked mode; {{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir request comes from a client with HDFS-6962 fix and {{dfs.namenode.posix.acl.inheritance.enabled}} is true. was (Author: jzhuge): Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * Preserve the existing meaning of field {{FsPermissionProto masked}}: mode + umask. * Please note this approach is slightly different from what you suggested above. I am ok either way but find this a little less possibility for misunderstanding. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} if (child.isPosixAclInheritanceEnabled() == true) { /* * HDFS-6962. POSIX ACL inheritance. */ // Set permission to unmasked if necessary if (!child.getFsPermission().equals(child.getUnmaskedMode())) { child.setPermission(child.getUnmaskedMode()); } if (AclStorage.copyINodeDefaultAclInternal(child) == false) { // No default ACL in parent dir // Set permission to masked child.setPermission(child.getMaskedMode()); } } else { /* * Old behavior before HDFS-6962. */ // Set permission to masked if necessary if (!child.getFsPermission().equals(child.getMaskedMode())) { child.setPermission(child.getMaskedMode()); } AclStorage.copyINodeDefaultAclInternal(child); } {code} Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedMode}} returns the masked mode; {{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir request come from a client that supports POSIX ACL inheritance and {{dfs.namenode.posix.acl.inheritance.enabled}} is true. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624 ] John Zhuge edited comment on HDFS-6962 at 6/7/16 5:04 AM: -- Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * Preserve the existing meaning of field {{FsPermissionProto masked}}: mode + umask. * Please note this approach is slightly different from what you suggested above. I am ok either way but find this a little less possibility for misunderstanding. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} if (child.isPosixAclInheritanceEnabled() == true) { /* * HDFS-6962. POSIX ACL inheritance. */ // Set permission to unmasked if necessary if (!child.getFsPermission().equals(child.getUnmaskedMode())) { child.setPermission(child.getUnmaskedMode()); } if (AclStorage.copyINodeDefaultAclInternal(child) == false) { // No default ACL in parent dir // Set permission to masked child.setPermission(child.getMaskedMode()); } } else { /* * Old behavior before HDFS-6962. */ // Set permission to masked if necessary if (!child.getFsPermission().equals(child.getMaskedMode())) { child.setPermission(child.getMaskedMode()); } AclStorage.copyINodeDefaultAclInternal(child); } {code} Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedMode}} returns the masked mode; {{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir request come from a client that supports POSIX ACL inheritance and {{dfs.namenode.posix.acl.inheritance.enabled}} is true. was (Author: jzhuge): Hi [~cnauroth], In order to maintain backwards compatibility in Hadoop 2.x, could we add a new NameNode flag {{dfs.namenode.posix.acl.inheritance}}, default false in Hadoop 2.x, default true in Hadoop 3? Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}} * Add optional field {{FsPermissionProto unmasked}} to store unmasked mode parameter * The meaning of the existing field {{FsPermissionProto masked}} stays the same: mode + umask, for both old and new client. * Please note this approach is slightly different alternative what you suggested. I am ok either way. Wrap around {{AclStorage#copyINodeDefaultAcl}}: {code:java} public static void copyINodeDefaultAcl(INode child) { // Current permission is masked assert child.getFsPermission().equals(child.getMaskedMode()); if (child.getPosixAclInheritance()) { // Set permission to unmasked child.setPermission(child.getUnmaskedMode()); } if (AclStorage.copyINodeDefaultAclInternal(child) == false) { // No default ACL in parent dir if (child.getPosixAclInheritance()) { // Set permission to masked child.setPermission(child.getMaskedMode()); } } } {code} Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient; {{INode#getMaskedMode}} returns the masked mode; {{INode#getPosixAclInheritance}} returns true when from a new client and the flag is on. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto