[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFS-9276.18.patch
Patch 18:
* Change {{Token}} copy constructor to do deep copies.
* No need for the extra field {{PrivateToken#publicService}}, thus no change to
{{PrivateToken}} at all.
* Change {{Credentials.addToken}} to use {{getService()}}.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFS-9276.17.patch,
> HDFS-9276.18.patch, HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.
[jira] [Commented] (HDFS-10684) WebHDFS calls fail when boolean parameters not provided
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392575#comment-15392575
]
John Zhuge commented on HDFS-10684:
---
[~loungerdork], I was not able to reproduce on 2.7.1 pseudo cluster, trunk
pseudo cluster, and 2.6.0 cluster. Is it possible that your DataNode
{{hadoop-datanode1}} is running a newer version than NameNode
{{hadoop-primarynamenode}}?
> WebHDFS calls fail when boolean parameters not provided
> ---
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
>Assignee: John Zhuge
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE&overwrite=false";
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false";
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10684) WebHDFS calls fail when boolean parameters not provided
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10684:
--
Labels: (was: newbie)
> WebHDFS calls fail when boolean parameters not provided
> ---
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE&overwrite=false";
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false";
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10684) WebHDFS calls fail when boolean parameters not provided
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HDFS-10684:
-
Assignee: John Zhuge
> WebHDFS calls fail when boolean parameters not provided
> ---
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
>Assignee: John Zhuge
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE&overwrite=false";
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false";
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391400#comment-15391400
]
John Zhuge commented on HDFS-10650:
---
Keep in mind, only when the callers pass in a null permission, this patch
changes default permission from 666 to 777 for directories. Check the call
hierarchy, do not see any issue. Potential problems are test code that expect
wrong dir permission when it passes null permission to mkdir, or scripts that
expect certain wrong default directory permissions. The second case is unlikely
while we can cover the first case by running all unit tests.
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10684) WebHDFS calls fail when boolean parameters not provided
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10684:
--
Labels: newbie (was: )
> WebHDFS calls fail when boolean parameters not provided
> ---
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
> Labels: newbie
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE&overwrite=false";
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020";
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10684) WebHDFS calls fail when boolean parameters not provided
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391383#comment-15391383
]
John Zhuge commented on HDFS-10684:
---
Hi [~loungerdork], thanks for reporting the problem with great details.
Following the redirect, you didn't use the exact DataNode URL printed by
NameNode {{TEMPORARY_REDIRECT}} response, as specified by Section "Create and
Write to a File" Step 2 in
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html.
I am leaning towards closing this jira as "Invalid", however there seems to be
a simple fix to handle this kind of user error by changing
{code}
public OverwriteParam(final String str) {
this(DOMAIN.parse(str));
}
{code}
to
{code}
public OverwriteParam(final String str) {
this(DOMAIN.parse(str == null ? DEFAULT : str));
}
{code}
Waiting for more feedbacks from the community.
> WebHDFS calls fail when boolean parameters not provided
> ---
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE&overwrite=false";
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE&namenoderpcaddress=hadoop-primarynamenode:8020";
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391367#comment-15391367
]
John Zhuge commented on HDFS-9276:
--
Hi [~marsishandsome], hope you don't mind we have been pushing this jira
forward. A customer is in urgent need of a patch.
Could you or [~hitliuyi] explain why {{PrivateToken#publicService}} was added?
Was it because {{PrivateToken.getService()}} could not be trusted? If so, can
we solve it by deep-copying {{service}} in {{Token}} copy constructor?
{code}
public Token(Token other) {
this.identifier = other.identifier;
this.password = other.password;
this.kind = other.kind;
this.service = new Text(other.service);
}
{code}
Or even deep-copying every field?
{code}
public Token(Token other) {
this.identifier = other.identifier.clone();
this.password = other.password.clone();
this.kind = new Text(other.kind);
this.service = new Text(other.service);
}
{code}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFS-9276.17.patch,
> HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFS-9276.17.patch
Patch 17:
* Incorporate review comments
* Make a deep copy of {{token.getService()}} to {{publicService}}
* Revert {{Credentials.addToken}} back to use {{instanceof Token.PrivateToken}}
and typecasting as in Patch 13
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFS-9276.17.patch,
> HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method
[jira] [Created] (HDFS-10683) Refactor Token.PrivateToken
John Zhuge created HDFS-10683:
-
Summary: Refactor Token.PrivateToken
Key: HDFS-10683
URL: https://issues.apache.org/jira/browse/HDFS-10683
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.0.0-alpha2
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
Avoid {{instanceof}} or typecasting of {{Toke.PrivateToken}} by introducing an
interface method in {{Token}}. Make class {{Toke.PrivateToken}} private. Use a
factory method instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391197#comment-15391197
]
John Zhuge edited comment on HDFS-9276 at 7/24/16 11:13 PM:
Another thing that bothers me is {{Token}} copy constructor performs shallow
copy of all 4 fields. Is this by design? Could [~jnp] or [~owen.omalley] please
comment since it is added by MAPREDUCE-2764?
{code}
public Token(Token other) {
this.identifier = other.identifier; <<< byte[]
this.password = other.password; <<< byte[]
this.kind = other.kind; <<< Text
this.service = other.service; <<< Text
}
{code}
was (Author: jzhuge):
Another thing that bothers me is {{Token}} copy constructor performs shallow
copy of all 4 fields. Is this by design?
{code}
public Token(Token other) {
this.identifier = other.identifier;
this.password = other.password;
this.kind = other.kind;
this.service = other.service;
}
{code}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apac
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391197#comment-15391197
]
John Zhuge commented on HDFS-9276:
--
Another thing that bothers me is {{Token}} copy constructor performs shallow
copy of all 4 fields. Is this by design?
{code}
public Token(Token other) {
this.identifier = other.identifier;
this.password = other.password;
this.kind = other.kind;
this.service = other.service;
}
{code}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391195#comment-15391195
]
John Zhuge commented on HDFS-9276:
--
Thanks [~xiaochen] for the great comments. Another patch is coming with the
fixes.
I discover a bug in Patch 13:
{code}
public PrivateToken(Token token) {
super(token);
publicService = token.getService();
}
{code}
{{publicService}} will point to the same object as {{token.service}}. Whenever
the content of {{token.service}} is changed, {{publicService}} will reflect the
change. I don't think this is what we want. It should perform a deep copy:
{code}
public PrivateToken(Token token) {
super(token);
publicService = new Text(token.getService());
}
{code}
bq. IMHO the 'public service' concept is only meaningful for PrivateToken, so
the following in Token would be confusing. I think a typecast is more readable
for future.
I'd avoid {{instanceof}} or typecasting. Whenever the need for that kind of
code pops up, it indicates something not right in the class design. However,
this jira is not the right place for refactoring. I will file a separate jira
to:
* Avoid {{instanceof}} or typecasting of {{Toke.PrivateToken}} with an
interface method technique
* Make class {{Toke.PrivateToken}} private. Use a factory method instead.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
>
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390824#comment-15390824
]
John Zhuge commented on HDFS-9276:
--
All unit tests seemed to have passed:
{noformat}
Tests run: 3462, Failures: 0, Errors: 0, Skipped: 84
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 19:18.435s
[INFO] Finished at: Sat Jul 23 17:38:22 UTC 2016
[INFO] Final Memory: 25M/302M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on
project hadoop-common:
There was a timeout or other error in the fork -> [Help 1]
[ERROR]
{noformat}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(
[jira] [Comment Edited] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390762#comment-15390762
]
John Zhuge edited comment on HDFS-9276 at 7/23/16 4:36 PM:
---
Patch 16:
* Fix test {{TestUserGroupInformation}} failure
* Fix javac unchecked warning by replacing {{new Token.PrivateToken(t)}} with
{{new Token.PrivateToken<>(t)}}. No need to suppress the warning.
{noformat}
[unchecked] unchecked call to PrivateToken(Token) as a member of the raw
type PrivateToken
{noformat}
* Pass these unit tests:
{{TestUserGroupInformation,TestDelegationTokenForProxyUser,TestDataNodeLifeline,TestCrcCorruption,TestBlockTokenWithDFSStriped,TestHttpServerLifecycle,TestLeaseRecovery2}}
was (Author: jzhuge):
Patch 16:
* Fix test {{TestUserGroupInformation}} failure
* Fix javac unchecked warning without suppressing it:
{noformat}
[unchecked] unchecked call to PrivateToken(Token) as a member of the raw
type PrivateToken
{noformat}
* Pass these unit tests:
{{TestUserGroupInformation,TestDelegationTokenForProxyUser,TestDataNodeLifeline,TestCrcCorruption,TestBlockTokenWithDFSStriped,TestHttpServerLifecycle,TestLeaseRecovery2}}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> tok
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFS-9276.16.patch
Patch 16:
* Fix test {{TestUserGroupInformation}} failure
* Fix javac unchecked warning without suppressing it:
{noformat}
[unchecked] unchecked call to PrivateToken(Token) as a member of the raw
type PrivateToken
{noformat}
* Pass these unit tests:
{{TestUserGroupInformation,TestDelegationTokenForProxyUser,TestDataNodeLifeline,TestCrcCorruption,TestBlockTokenWithDFSStriped,TestHttpServerLifecycle,TestLeaseRecovery2}}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFS-9276.16.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invo
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390740#comment-15390740
]
John Zhuge commented on HDFS-6962:
--
Ignore the 3 checkstyle {{ParameterNumber}} errors for now. I will file a jira
for a new method to suppress them individually, e.g., using annotation
{{SuppressWarning("checkstyle:parameternumber")}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch,
> HDFS-6962.009.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: HDFS-6962.009.patch
Patch 009:
* No longer store anything in {{INode}}
* Modify {{addINode}} and {{addLastINode}} to pass create modes from RPC to
{{copyINodeDefaultAcl}} on NameNode
* Do not modify {{addLastINodeNoQuotaCheck}} because all its callers (rename
related) do not have create modes to pass
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch,
> HDFS-6962.009.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFS-9276.15.patch
Patch 15:
* Revert back to {{getPublicService}} API that is more flexible than
{{isPrivateClone}} and equally clean
* To test a private clone: {{alias.equals(token.getPublicService())}}
* To test a private token: {{token.getPublicService() == null}}
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFS-9276.15.patch, HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15390398#comment-15390398
]
John Zhuge commented on HDFS-6962:
--
As suggested by [~eddyxu], I found a way not to store {{FsCreateModes}} in
{{INode}}. Only have to change {{addINode}} and {{addLastINode}} signatures.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch,
> HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log,
> enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389736#comment-15389736 ] John Zhuge commented on HDFS-6962: -- [~cnauroth] and [~eddyxu], could you please review patch 008 that has new and isolated changes to support webhdfs? > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch, > HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log, > enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389708#comment-15389708
]
John Zhuge commented on HDFS-9276:
--
Ignore the unrelated TestGangliaMetrics failure. What other tests should we run?
+1 (non-binding) Looks good to me. Thanks [~marsishandsome] for working on the
patch, [~hitliuyi] for the reviews. Could someone else please take a look? This
issue has been hit by many customers.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflec
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFS-9276.14.patch
Patch 14:
* Minor code cleanups
* Remove the use of {{instanceof}}
The patch (applied on 2.6.0) passed 866 {{hadoop-common}} and {{hadoop-hdfs}}
unit tests.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFS-9276.14.patch,
> HDFSReadLoop.scala, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>
[jira] [Comment Edited] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388803#comment-15388803
]
John Zhuge edited comment on HDFS-9276 at 7/22/16 3:57 AM:
---
I was able to reproduce with a simple long running Spark Scala application
{{HDFSReadLoop.scala}} (attached) that performs HDFS read periodically. It hits
the delegation toke expired exception after a little over 3 minutes. The
cluster is HA with custom delegation properties specified in Description.
{{HDFS-9276.13.patch}} does fix the issue.
was (Author: jzhuge):
I was able to reproduce with a simple long running Spark application
{{HDFSReadLoop}} that performs HDFS read periodically. It hits the delegation
toke expired exception after a little over 3 minutes. The cluster is HA with
custom delegation properties specified in Description. {{HDFS-9276.13.patch}}
does fix the issue.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo
[jira] [Comment Edited] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388803#comment-15388803
]
John Zhuge edited comment on HDFS-9276 at 7/22/16 3:14 AM:
---
I was able to reproduce with a simple long running Spark application
{{HDFSReadLoop}} that performs HDFS read periodically. It hits the delegation
toke expired exception after a little over 3 minutes. The cluster is HA with
custom delegation properties specified in Description. {{HDFS-9276.13.patch}}
does fix the issue.
was (Author: jzhuge):
I was able to reproduce with a simple long running Spark application
{{HDFSReadLoop}} that performs HDFS read periodically. It hits the delegation
toke expired exception after a little over 3 minutes. {{HDFS-9276.13.patch}}
does fix the issue.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.ge
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-9276:
-
Attachment: HDFSReadLoop.scala
I was able to reproduce with a simple long running Spark application
{{HDFSReadLoop}} that performs HDFS read periodically. It hits the delegation
toke expired exception after a little over 3 minutes. {{HDFS-9276.13.patch}}
does fix the issue.
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, HDFS-9276.08.patch,
> HDFS-9276.09.patch, HDFS-9276.10.patch, HDFS-9276.11.patch,
> HDFS-9276.12.patch, HDFS-9276.13.patch, HDFSReadLoop.scala, debug1.PNG,
> debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.Re
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387990#comment-15387990
]
John Zhuge commented on HDFS-6962:
--
Unit test failures seem unrelated. There is nothing I can do about the 3
checkstyle {{ParameterNumber}} errors. Anyway to suppress warnings? I will
research some ways to do it. As for "FsCreateModes.java:0:: Missing
package-info.java file.", I guess I will add a {{package-info.java}} file.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch,
> HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log,
> enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: HDFS-6962.008.patch Patch 008: * Support webhdfs * Incorporate review comments > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch, > HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log, > enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386887#comment-15386887 ] John Zhuge edited comment on HDFS-6962 at 7/21/16 5:41 AM: --- Thanks [~eddyxu] for the review! Great comments. I will fix the issues.. bq. Is it true that after creation, the createMode is not useful for INode, as its mode has already been established? I feel that we do not need to store it as a feature in INode. Yes it is true, create modes are only needed during inode creation. It is a stretch to call CreateModes a feature, but it has be stored somewhere in inode or another data structure 1-1 tied to inode. Suggestions are welcome. was (Author: jzhuge): Thanks [~eddyxu] for the review! Great comments. I will fix the issues.. bq. Is it true that after creation, the createMode is not useful for INode, as its mode has already been established? I feel that we do not need to store it as a feature in INode. Yes it is true, create modes are only needed during inode creation. It is a stretch to call CreateModes a feature, but it has be stored somewhere in something hanging off inode or another data structure 1-1 tied to inode. Suggestions are welcome. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386887#comment-15386887 ] John Zhuge commented on HDFS-6962: -- Thanks [~eddyxu] for the review! Great comments. I will fix the issues.. bq. Is it true that after creation, the createMode is not useful for INode, as its mode has already been established? I feel that we do not need to store it as a feature in INode. Yes it is true, create modes are only needed during inode creation. It is a stretch to call CreateModes a feature, but it has be stored somewhere in something hanging off inode or another data structure 1-1 tied to inode. Suggestions are welcome. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386399#comment-15386399
]
John Zhuge edited comment on HDFS-6962 at 7/20/16 10:40 PM:
[~cnauroth] and [~eddyxu] Please review patch 007.
Diff from 006:
* Add CLI tests {{TestAclCLIWithPosixAclInheritance}} based on {{TestAclCLI}}
* No longer add field {{createModes}} to {{INodeWithAdditionalFields}}.
Instead, add new feature {{CreateModesFeature}} to store create modes. In this
way, no penalty when POSIX ACL inheritance is disable or for any inode not
being created.
* Remove the feature {{CreateModesFeature}} once default ACL has been
processed, so the feature only exists in inode for a short period of time.
* There is cost of adding and removing the new feature.
TODO:
* Support webhdfs
was (Author: jzhuge):
[~cnauroth] and [~eddyxu] Please review patch 007.
Diff from 006:
* Add CLI tests {{TestAclCLIWithPosixAclInheritance}} based on {{TestAclCLI}}
* No longer add field {{createModes}} to {{INodeWithAdditionalFields}}.
Instead, add new feature {{CreateModesFeature}} to store create modes. In this
way, no penalty when POSIX ACL inheritance is disable or for any inode not
associated with a create request.
* Remove the feature {{CreateModesFeature}} once default ACL has been processed.
* There is added cost of adding and removing the new feature.
TODO:
* Support webhdfs
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch,
> disabled_new_client.log, disabled_old_client.log, enabled_new_client.log,
> enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: h
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: HDFS-6962.007.patch
[~cnauroth] and [~eddyxu] Please review patch 007.
Diff from 006:
* Add CLI tests {{TestAclCLIWithPosixAclInheritance}} based on {{TestAclCLI}}
* No longer add field {{createModes}} to {{INodeWithAdditionalFields}}.
Instead, add new feature {{CreateModesFeature}} to store create modes. In this
way, no penalty when POSIX ACL inheritance is disable or for any inode not
associated with a create request.
* Remove the feature {{CreateModesFeature}} once default ACL has been processed.
* There is added cost of adding and removing the new feature.
TODO:
* Support webhdfs
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.1.patch,
> disabled_new_client.log, disabled_old_client.log, enabled_new_client.log,
> enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10661) Make MiniDFSCluster AutoCloseable
[ https://issues.apache.org/jira/browse/HDFS-10661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved HDFS-10661. --- Resolution: Duplicate [~ajisakaa] Looks like a dup of HDFS-10287. Please re-open if you think otherwise. > Make MiniDFSCluster AutoCloseable > - > > Key: HDFS-10661 > URL: https://issues.apache.org/jira/browse/HDFS-10661 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Akira Ajisaka > > If we make MiniDFSCluster AutoCloseable, we can create MiniDFSCluster > instance using try-with-resources statement. That way we don't have to > shutdown the cluster in finally clause every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-10657) testAclCLI.xml inherit default ACL to dir test should expect mask r-x
[
https://issues.apache.org/jira/browse/HDFS-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-10657 started by John Zhuge.
-
> testAclCLI.xml inherit default ACL to dir test should expect mask r-x
> -
>
> Key: HDFS-10657
> URL: https://issues.apache.org/jira/browse/HDFS-10657
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> The following test case should expect {{mask::r-x}} ACL entry instead of
> {{mask::rwx}}:
> {code:xml}
> setfacl : check inherit default ACL to dir
>
> -fs NAMENODE -mkdir /dir1
> -fs NAMENODE -setfacl -m
> default:user:charlie:r-x,default:group:admin:rwx /dir1
> -fs NAMENODE -mkdir /dir1/dir2
> -fs NAMENODE -getfacl /dir1/dir2
> ...
>
> SubstringComparator
> mask::rwx
>
> {code}
> But why does it pass? Because the comparator type is {{SubstringComparator}}
> and it matches the wrong line {{default:mask::rwx}} in the output of
> {{getfacl}}:
> {noformat}
> # file: /dir1/dir2
> # owner: jzhuge
> # group: supergroup
> user::rwx
> user:charlie:r-x
> group::r-x
> group:admin:rwx #effective:r-x
> mask::r-x
> other::r-x
> default:user::rwx
> default:user:charlie:r-x
> default:group::r-x
> default:group:admin:rwx
> default:mask::rwx
> default:other::r-x
> {noformat}
> The comparator should match the entire line instead of just substring. Other
> comparators in {{testAclCLI.xml}} have the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10657) testAclCLI.xml inherit default ACL to dir test should expect mask r-x
[
https://issues.apache.org/jira/browse/HDFS-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385536#comment-15385536
]
John Zhuge commented on HDFS-10657:
---
Change the type to {{RegexpComparator}} and set {{expected-output}} to
{{^mask::r-x$}}.
Is it worthwhile to create a new type {{ExactLineComparator}}? With the new
type, {{expected-output}} can be {{mask::r-x}}
> testAclCLI.xml inherit default ACL to dir test should expect mask r-x
> -
>
> Key: HDFS-10657
> URL: https://issues.apache.org/jira/browse/HDFS-10657
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> The following test case should expect {{mask::r-x}} ACL entry instead of
> {{mask::rwx}}:
> {code:xml}
> setfacl : check inherit default ACL to dir
>
> -fs NAMENODE -mkdir /dir1
> -fs NAMENODE -setfacl -m
> default:user:charlie:r-x,default:group:admin:rwx /dir1
> -fs NAMENODE -mkdir /dir1/dir2
> -fs NAMENODE -getfacl /dir1/dir2
> ...
>
> SubstringComparator
> mask::rwx
>
> {code}
> But why does it pass? Because the comparator type is {{SubstringComparator}}
> and it matches the wrong line {{default:mask::rwx}} in the output of
> {{getfacl}}:
> {noformat}
> # file: /dir1/dir2
> # owner: jzhuge
> # group: supergroup
> user::rwx
> user:charlie:r-x
> group::r-x
> group:admin:rwx #effective:r-x
> mask::r-x
> other::r-x
> default:user::rwx
> default:user:charlie:r-x
> default:group::r-x
> default:group:admin:rwx
> default:mask::rwx
> default:other::r-x
> {noformat}
> The comparator should match the entire line instead of just substring. Other
> comparators in {{testAclCLI.xml}} have the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10657) testAclCLI.xml inherit default ACL to dir test should expect mask r-x
John Zhuge created HDFS-10657:
-
Summary: testAclCLI.xml inherit default ACL to dir test should
expect mask r-x
Key: HDFS-10657
URL: https://issues.apache.org/jira/browse/HDFS-10657
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
The following test case should expect {{mask::r-x}} ACL entry instead of
{{mask::rwx}}:
{code:xml}
setfacl : check inherit default ACL to dir
-fs NAMENODE -mkdir /dir1
-fs NAMENODE -setfacl -m
default:user:charlie:r-x,default:group:admin:rwx /dir1
-fs NAMENODE -mkdir /dir1/dir2
-fs NAMENODE -getfacl /dir1/dir2
...
SubstringComparator
mask::rwx
{code}
But why does it pass? Because the comparator type is {{SubstringComparator}}
and it matches the wrong line {{default:mask::rwx}} in the output of
{{getfacl}}:
{noformat}
# file: /dir1/dir2
# owner: jzhuge
# group: supergroup
user::rwx
user:charlie:r-x
group::r-x
group:admin:rwx #effective:r-x
mask::r-x
other::r-x
default:user::rwx
default:user:charlie:r-x
default:group::r-x
default:group:admin:rwx
default:mask::rwx
default:other::r-x
{noformat}
The comparator should match the entire line instead of just substring. Other
comparators in {{testAclCLI.xml}} have the same problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10650:
--
Attachment: HDFS-10650.002.patch
Patch 002:
* Fix a regression in 001
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384940#comment-15384940
]
John Zhuge commented on HDFS-10650:
---
Thanks [~xiaochen].
This jira came out of discussion
https://issues.apache.org/jira/browse/HDFS-6962?focusedCommentId=15383249&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15383249.
[~cnauroth], do you know the current behavior is a bug or intentional?
bq. why remove the null check?
My bug. I will fix it.
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10650:
--
Status: Patch Available (was: Open)
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10650:
--
Attachment: HDFS-10650.001.patch
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10650:
--
Attachment: (was: HADOOP-13191.002.patch)
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HDFS-10650.001.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10650:
--
Attachment: HADOOP-13191.002.patch
Patch 001:
* These 2 methods should use default dir permission instead of default
permission when the permission parameter is {{null}}
* Add unit tests
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-13191.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383376#comment-15383376
]
John Zhuge commented on HDFS-10650:
---
In {{trunk}} branch, things were consolidated into this private method
{{DFSClient#applyUMask}}:
{code:java}
private FsPermission applyUMask(FsPermission permission) {
if (permission == null) {
permission = FsPermission.getFileDefault();
}
return permission.applyUMask(dfsClientConf.getUMask());
}
{code}
However, same problem when this is called by {{mkdirs}} and {{primitiveMkdir}}.
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
John Zhuge created HDFS-10650:
-
Summary: DFSClient#mkdirs and DFSClient#primitiveMkdir should use
default directory permission
Key: HDFS-10650
URL: https://issues.apache.org/jira/browse/HDFS-10650
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
These 2 DFSClient methods should use default directory permission to create a
directory.
{code:java}
public boolean mkdirs(String src, FsPermission permission,
boolean createParent) throws IOException {
if (permission == null) {
permission = FsPermission.getDefault();
}
{code}
{code:java}
public boolean primitiveMkdir(String src, FsPermission absPermission,
boolean createParent)
throws IOException {
checkOpen();
if (absPermission == null) {
absPermission =
FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10649) Remove unused PermissionStatus#applyUMask
[
https://issues.apache.org/jira/browse/HDFS-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10649:
--
Description: Class {{PermissionStatus}} is LimitedPrivate("HDFS",
"MapReduce") and Unstable. (was: Class {{PermissionStatus}} is
LimitedPrivate({"HDFS", "MapReduce"}) and Unstable.)
> Remove unused PermissionStatus#applyUMask
> -
>
> Key: HDFS-10649
> URL: https://issues.apache.org/jira/browse/HDFS-10649
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Trivial
> Labels: newbie
>
> Class {{PermissionStatus}} is LimitedPrivate("HDFS", "MapReduce") and
> Unstable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10649) Remove unused PermissionStatus#applyUMask
John Zhuge created HDFS-10649:
-
Summary: Remove unused PermissionStatus#applyUMask
Key: HDFS-10649
URL: https://issues.apache.org/jira/browse/HDFS-10649
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: John Zhuge
Priority: Trivial
Class {{PermissionStatus}} is LimitedPrivate({"HDFS", "MapReduce"}) and
Unstable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Target Version/s: 3.0.0-alpha2 (was: 2.8.0) Hadoop Flags: Incompatible change > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.1.patch, disabled_new_client.log, > disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383312#comment-15383312
]
John Zhuge commented on HDFS-6962:
--
Thank you very much [~cnauroth]. Valid points.
One additional question before responding to your comments. I added
{{getMasked}} and {{getUnmasked}} with default implementations to
{{FsPermission}} which is public and stable. Is that ok? The alternative to
this approach is to use {{instanceof}} to detect {{FsCreateModes}} object with
an {{FsPermission}} reference.
bq. target it to the 3.x line
I think it is ok. Will it affect our plan to backport the fix to CDH branches
based on 2.6.0?
bq. WebHDFS support
Will implement it in this jira.
bq. adding the {{createModes}} member to {{INodeWithAdditionalFields}}
Yes, I made the trade-off between adding a field to
{{INodeWithAdditionalFields}} and changing method signatures along the NN stack
from RPC to {{FSDirectory#copyINodeDefaultAcl}}. I will revisit the decision.
bq. LOG.warn("Received create request without unmasked create mode");
Changed to {{debug}}.
bq. "comppatible"
Fixed.
bq. add a clarifying statement that umask would be ignored
Done.
bq. make a new test suite, similar to TestAclCLI, but with
dfs.namenode.posix.acl.inheritance.enabled set to true.
Will do.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.006.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-8897) Loadbalancer always exits with : java.io.IOException: Another Balancer is running.. Exiting ...
[
https://issues.apache.org/jira/browse/HDFS-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8897 started by John Zhuge.
> Loadbalancer always exits with : java.io.IOException: Another Balancer is
> running.. Exiting ...
>
>
> Key: HDFS-8897
> URL: https://issues.apache.org/jira/browse/HDFS-8897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.7.1
> Environment: Centos 6.6
>Reporter: LINTE
>Assignee: John Zhuge
>
> When balancer is launched, it should test if there is already a
> /system/balancer.id file in HDFS.
> When the file doesn't exist, the balancer don't want to run :
> 15/08/14 16:35:12 INFO balancer.Balancer: namenodes = [hdfs://sandbox/,
> hdfs://sandbox]
> 15/08/14 16:35:12 INFO balancer.Balancer: parameters =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration
> = 5, number of nodes to be excluded = 0, number of nodes to be included = 0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> java.io.IOException: Another Balancer is running.. Exiting ...
> Aug 14, 2015 4:35:14 PM Balancing took 2.408 seconds
> Looking at the audit log file when trying to run the balancer, the balancer
> create the /system/balancer.id and then delete it on exiting ...
> 2015-08-14 16:37:45,844 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:45,900 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=create
> src=/system/balancer.id dst=nullperm=hdfs:hadoop:rw-r-
> proto=rpc
> 2015-08-14 16:37:45,919 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,090 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,112 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,117 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=delete
> src=/system/balancer.id dst=nullperm=null proto=rpc
> The error seems to be located in
> org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> The function checkAndMarkRunning return null even if the /system/balancer.id
> doesn't exist before entering this function; if it exists, then it is deleted
> and the balancer exit with the same error.
>
> private OutputStream checkAndMarkRunning() throws IOException {
> try {
> if (fs.exists(idPath)) {
> // try appending to it so that it will fail fast if another balancer
> is
> // running.
> IOUtils.closeStream(fs.append(idPath));
> fs.delete(idPath, true);
> }
> final FSDataOutputStream fsout = fs.create(idPath, false);
> // mark balancer idPath to be deleted during filesystem closure
> fs.deleteOnExit(idPath);
> if (write2IdFile) {
> fsout.writeBytes(InetAddress.getLocalHost().getHostName());
> fsout.hflush();
> }
> return fsout;
> } catch(RemoteException e) {
>
> if(AlreadyBeingCreatedException.class.getName().equals(e.getClassName())){
> return null;
> } else {
> throw e;
> }
> }
> }
>
> Regards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-
[jira] [Assigned] (HDFS-8897) Loadbalancer always exits with : java.io.IOException: Another Balancer is running.. Exiting ...
[
https://issues.apache.org/jira/browse/HDFS-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HDFS-8897:
Assignee: John Zhuge
> Loadbalancer always exits with : java.io.IOException: Another Balancer is
> running.. Exiting ...
>
>
> Key: HDFS-8897
> URL: https://issues.apache.org/jira/browse/HDFS-8897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.7.1
> Environment: Centos 6.6
>Reporter: LINTE
>Assignee: John Zhuge
>
> When balancer is launched, it should test if there is already a
> /system/balancer.id file in HDFS.
> When the file doesn't exist, the balancer don't want to run :
> 15/08/14 16:35:12 INFO balancer.Balancer: namenodes = [hdfs://sandbox/,
> hdfs://sandbox]
> 15/08/14 16:35:12 INFO balancer.Balancer: parameters =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration
> = 5, number of nodes to be excluded = 0, number of nodes to be included = 0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
> 15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> java.io.IOException: Another Balancer is running.. Exiting ...
> Aug 14, 2015 4:35:14 PM Balancing took 2.408 seconds
> Looking at the audit log file when trying to run the balancer, the balancer
> create the /system/balancer.id and then delete it on exiting ...
> 2015-08-14 16:37:45,844 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:45,900 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=create
> src=/system/balancer.id dst=nullperm=hdfs:hadoop:rw-r-
> proto=rpc
> 2015-08-14 16:37:45,919 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,090 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,112 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
> src=/system/balancer.id dst=nullperm=null proto=rpc
> 2015-08-14 16:37:46,117 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=delete
> src=/system/balancer.id dst=nullperm=null proto=rpc
> The error seems to be located in
> org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> The function checkAndMarkRunning return null even if the /system/balancer.id
> doesn't exist before entering this function; if it exists, then it is deleted
> and the balancer exit with the same error.
>
> private OutputStream checkAndMarkRunning() throws IOException {
> try {
> if (fs.exists(idPath)) {
> // try appending to it so that it will fail fast if another balancer
> is
> // running.
> IOUtils.closeStream(fs.append(idPath));
> fs.delete(idPath, true);
> }
> final FSDataOutputStream fsout = fs.create(idPath, false);
> // mark balancer idPath to be deleted during filesystem closure
> fs.deleteOnExit(idPath);
> if (write2IdFile) {
> fsout.writeBytes(InetAddress.getLocalHost().getHostName());
> fsout.hflush();
> }
> return fsout;
> } catch(RemoteException e) {
>
> if(AlreadyBeingCreatedException.class.getName().equals(e.getClassName())){
> return null;
> } else {
> throw e;
> }
> }
> }
>
> Regards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: h
[jira] [Commented] (HDFS-10387) DataTransferProtocol#writeBlock missing some javadocs
[
https://issues.apache.org/jira/browse/HDFS-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373964#comment-15373964
]
John Zhuge commented on HDFS-10387:
---
Thanks [~eddyxu].
> DataTransferProtocol#writeBlock missing some javadocs
> -
>
> Key: HDFS-10387
> URL: https://issues.apache.org/jira/browse/HDFS-10387
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs
>Reporter: Yongjun Zhang
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-10387.001.patch
>
>
> DataTransferProtocol#writeBlock's javadocs has the following parameters
> missing:
> {code}
> final DataChecksum requestedChecksum,
> final CachingStrategy cachingStrategy,
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10300) TestDistCpSystem should share MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372025#comment-15372025 ] John Zhuge commented on HDFS-10300: --- Thanks [~andrew.wang]. > TestDistCpSystem should share MiniDFSCluster > > > Key: HDFS-10300 > URL: https://issues.apache.org/jira/browse/HDFS-10300 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Trivial > Labels: quality, test > Fix For: 2.8.0 > > Attachments: HDFS-10300.001.patch, HDFS-10300.002.patch > > > The test cases in this class should share MiniDFSCluster if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10300) TestDistCpSystem should share MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10300:
--
Attachment: HDFS-10300.002.patch
Patch 002:
* Check not null before calling {{cluster.shutdown}}
* Fix 2 checkstyle errors
> TestDistCpSystem should share MiniDFSCluster
>
>
> Key: HDFS-10300
> URL: https://issues.apache.org/jira/browse/HDFS-10300
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Trivial
> Labels: quality, test
> Attachments: HDFS-10300.001.patch, HDFS-10300.002.patch
>
>
> The test cases in this class should share MiniDFSCluster if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9805) TCP_NODELAY not set before SASL handshake in data transfer pipeline
[
https://issues.apache.org/jira/browse/HDFS-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362916#comment-15362916
]
John Zhuge commented on HDFS-9805:
--
[~cmccabe]: You committed the patch into trunk on 6/29. Do you plan to resolve
the jira?
> TCP_NODELAY not set before SASL handshake in data transfer pipeline
> ---
>
> Key: HDFS-9805
> URL: https://issues.apache.org/jira/browse/HDFS-9805
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Attachments: HDFS-9805.002.patch, HDFS-9805.003.patch,
> HDFS-9805.004.patch, HDFS-9805.005.patch
>
>
> There are a few places in the DN -> DN block transfer pipeline where
> TCP_NODELAY is not set before doing a SASL handshake:
> * in {{DataNode.DataTransfer::run()}}
> * in {{DataXceiver::replaceBlock()}}
> * in {{DataXceiver::writeBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: HDFS-6962.006.patch Patch 006 over 005: * Remove the ugly calls of "instanceof FsCreateModes" > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.1.patch, disabled_new_client.log, > disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361628#comment-15361628
]
John Zhuge commented on HDFS-6962:
--
{{TestBalancer}} failures seem unrelated.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch,
> HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log,
> enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: HDFS-6962.005.patch Patch 005 over 004: * Fix test regressions > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.1.patch, disabled_new_client.log, disabled_old_client.log, > enabled_new_client.log, enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359304#comment-15359304
]
John Zhuge edited comment on HDFS-6962 at 7/4/16 8:02 AM:
--
Saw {{TestDataNodeLifeline}} and {{TestOfflineEditsViewer}} failures on my
local Mac with stock trunk, so they are not regressions.
{{TestMover,TestNNThroughputBenchmark,TestNNThroughputBenchmark,TestStripedINodeFile,TestRetryCacheWithHA}}
failures are regressions.
was (Author: jzhuge):
Looking into the unit test failures. They are regressions.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch,
> disabled_new_client.log, disabled_old_client.log, enabled_new_client.log,
> enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5086) Support RPCSEC_GSS authentication in NFSv3 gateway
[ https://issues.apache.org/jira/browse/HDFS-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359681#comment-15359681 ] John Zhuge commented on HDFS-5086: -- Hi [~jingzhao] Is your patch working to some degree? All code written from scratch? Thanks, John > Support RPCSEC_GSS authentication in NFSv3 gateway > -- > > Key: HDFS-5086 > URL: https://issues.apache.org/jira/browse/HDFS-5086 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0-alpha1 >Reporter: Brandon Li >Assignee: Jing Zhao > Attachments: HDFS-5086.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359304#comment-15359304 ] John Zhuge commented on HDFS-6962: -- Looking into the unit test failures. They are regressions. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Hadoop Flags: (was: Incompatible change) Target Version/s: 2.8.0 (was: ) Status: Patch Available (was: In Progress) > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: HDFS-6962.004.patch
Patch 004 (passed system and compatibility tests):
* Create a new class {{FsCreateModes}} that extends {{FsPermission}} to store
both masked and unmasked create modes. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
* Add 2 unit tests to {{FSAclBaseTest}}
Questions:
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Create a separate jira to to support WebHDFSAcl and NFS if necessary
* More updates to {{HdfsPermissionsGuide.md}}
[~cnauroth] and [~atm], please review patch 004.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch,
> disabled_new_client.log, disabled_old_client.log, enabled_new_client.log,
> enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400
]
John Zhuge edited comment on HDFS-6962 at 6/29/16 2:56 AM:
---
Patch 004 passed system and compatibility tests on a pseudo cluster. Upload
test logs. Focus on group WRITE permission. The test script is {{run}}, the
command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
was (Author: jzhuge):
Patch 004 passed system and compatibility tests on a pseudo cluster. Upload
test logs. Focus on group permission, especially WRITE. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: ha
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354430#comment-15354430
]
John Zhuge commented on HDFS-6962:
--
Patch 004 passed the following ACL related unit tests including the new
{{FSAclBaseTest#testUMaskDefaultAclNewFile}} and
{{FSAclBaseTest#testUMaskDefaultAclNewDir}}: TestAcl, TestAclCommands,
TestAclCLI, TestViewFileSystemWithAcls, TestViewFsWithAcls,
TestAclWithSnapshot, TestAclConfigFlag, TestAclTransformation,
TestFileContextAcl, TestFSImageWithAcl, TestNameNodeAcl, TestAclsEndToEnd,
TestOfflineImageViewerForAcl, and TestWebHDFSAcl.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400
]
John Zhuge edited comment on HDFS-6962 at 6/29/16 2:48 AM:
---
Patch 004 passed system and compatibility tests on a pseudo cluster. Upload
test logs. Focus on group permission, especially WRITE. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
was (Author: jzhuge):
Pass system and compatibility tests with Patch 004 on a pseudo cluster. Upload
test logs. Focus on group permission, especially WRITE. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdf
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354400#comment-15354400
]
John Zhuge edited comment on HDFS-6962 at 6/29/16 2:40 AM:
---
Pass system and compatibility tests with Patch 004 on a pseudo cluster. Upload
test logs. Focus on group permission, especially WRITE. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
was (Author: jzhuge):
Upload system and compatibility test logs for Patch 004 on a pseudo cluster.
Pay attention to group permission, especially WRITE bit. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: h
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: run
enabled_old_client.log
enabled_new_client.log
disabled_old_client.log
disabled_new_client.log
Upload system and compatibility test logs for Patch 004 on a pseudo cluster.
Pay attention to group permission, especially WRITE bit. The test script is
{{run}}, the command line is
{code}
test_tag=disabled_new_client ; script ~/test/$test_tag.log ~/test/run $test_tag
{code}
Test server is running {{3.0.0-alpha1 + Patch 004}} with POSIX ACL Inheritance
either enabled or disabled. There are 2 versions of test clients. The POSIX ACL
inheritance is only in effect when the flag is true and the requests come from
a compatible client. The following table shows the matrix for the log files.
|| Client Version\POSIX ACL Inheritance || Enabled || Disabled ||
| 2.6.0 | enabled_old_client.log | disabled_old_client.log |
| 3.0.0-alpha1 + Patch 004 | enabled_new_client.log | disabled_new_client.log |
In each log file, there is a test matrix of the following dimensions:
* Parent dir has default ACL or not
* Umask 002 or 027
* Create a file (create request) or a sub-directory (mkdirs request)
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch, disabled_new_client.log,
> disabled_old_client.log, enabled_new_client.log, enabled_old_client.log, run
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353446#comment-15353446
]
John Zhuge commented on HDFS-6962:
--
{{FSDirMkdirOp.createAncestorDirectories}} is unrelated to the current inode
being created, thus should not use the current create modes.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352115#comment-15352115
]
John Zhuge commented on HDFS-6962:
--
To support webhdfs, the REST api for {{CREATE}} and {{MKDIRS}} must be extended
with a new parameter for unmasked permission. Let us add the support in a
separate jira if deemed necessary. For now, the 2 new unit tests are disabled
for {{TestWebHDFSAcl}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: HDFS-6962.003.patch
Patch 003 (almost complete, unit tests passed):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
* Add 2 unit tests to {{FSAclBaseTest}}
Questions:
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
* Better name for {{FsPermissionDuo}}?
TODO:
* Investigate why TestWebHDFSAcl does not support the 2 new unit tests
* Run system tests and compatibility tests
* Update {{HdfsPermissionsGuide.md}}
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
[~cnauroth], please take a look at 003 which plugs some loose ends and adds 2
unit tests.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.003.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345214#comment-15345214
]
John Zhuge edited comment on HDFS-6962 at 6/24/16 5:20 AM:
---
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Currently these are ACL related
tests:
TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl.
Will add a new test class {{namenode/TestAclInheritance}}.
was (Author: jzhuge):
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Currently these are ACL related
tests:
TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,FSAclBaseTest,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl.
Will add a new test class {{namenode/TestAclInheritance}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345258#comment-15345258 ] John Zhuge commented on HDFS-6962: -- [~cnauroth], Thanks for the info. I will work with the existing test classes as much as possible. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345214#comment-15345214
]
John Zhuge edited comment on HDFS-6962 at 6/22/16 9:44 PM:
---
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Currently these are ACL related
tests:
TestAcl,TestAclCommands,TestAclCLI,TestViewFileSystemWithAcls,TestViewFsWithAcls,FSAclBaseTest,TestAclWithSnapshot,TestAclConfigFlag,TestAclTransformation,TestFileContextAcl,TestFSImageWithAcl,TestNameNodeAcl,TestAclsEndToEnd,TestOfflineImageViewerForAcl,TestWebHDFSAcl.
Will add a new test class {{namenode/TestAclInheritance}}.
was (Author: jzhuge):
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Will add a new test class
{{namenode/TestAclInheritance}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345214#comment-15345214
]
John Zhuge commented on HDFS-6962:
--
[~cnauroth] Are there automated tests for section {{Default Acls}} in test plan
{{Test-Plan-for-Extended-Acls-2.pdf}}? Either extend that section or add a new
section in test plan to cover ACL inheritance. Will add a new test class
{{namenode/TestAclInheritance}}.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337480#comment-15337480 ] John Zhuge commented on HDFS-6962: -- What is your exact scenario? What are possible parent dir's default ACLs? Is it ok for you to set umask to 0 or something that will not affect any default ACL? > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336361#comment-15336361 ] John Zhuge commented on HDFS-6962: -- [~cnauroth] I will start working on a fully functional patch. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941
]
John Zhuge edited comment on HDFS-6962 at 6/11/16 6:41 AM:
---
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
Questions:
* How to query NN config in {{copyINodeDefaultAcl}}?
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* Add and run unit tests
* Run system compatibility tests
* Update {{HdfsPermissionsGuide.md}}
[~cnauroth] Could you please take a look at Patch 002 to check whether
backwards compatibility can be achieved with an NN flag and the changes are
going in the right direction? Thanks.
was (Author: jzhuge):
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
Questions:
* How to query NN config in {{copyINodeDefaultAcl}}?
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* Add and run unit tests
* Run system compatibility tests
* Update {{HdfsPermissionsGuide.md}}
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941
]
John Zhuge edited comment on HDFS-6962 at 6/11/16 6:39 AM:
---
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
Questions:
* How to query NN config in {{copyINodeDefaultAcl}}?
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* Add and run unit tests
* Run system compatibility tests
* Update {{HdfsPermissionsGuide.md}}
was (Author: jzhuge):
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
Questions:
* How to query NN config in {{copyINodeDefaultAcl}}?
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* Add and run unit tests
* Run system compatibility tests
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a s
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624
]
John Zhuge edited comment on HDFS-6962 at 6/11/16 6:37 AM:
---
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
// TODO: How to query NameNode config?
boolean isPosixAclInheritanceEnabled = true;
FsPermission masked = child.getMaskedPermission();
FsPermission unmasked = child.getUnmaskedPermission();
if (isPosixAclInheritanceEnabled && unmasked != null) {
//
// HDFS-6962: POSIX ACL inheritance
//
// Set permission to unmasked
child.setPermission(unmasked);
if (!copyINodeDefaultAclInternal(child)) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(masked);
}
} else {
//
// Old behavior before HDFS-6962
//
assert child.getFsPermission().equals(masked);
copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedPermission}} returns the unmasked mode sent by
DFSClient; {{INode#getMaskedPermission}} returns the masked mode.
was (Author: jzhuge):
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
//
// HDFS-6962. POSIX ACL inheritance.
//
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
//
// Old behavior before HDFS-6962.
//
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request comes from a client with HDFS-6962 fix and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everythin
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323941#comment-15323941
]
John Zhuge edited comment on HDFS-6962 at 6/10/16 6:20 AM:
---
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
Questions:
* How to query NN config in {{copyINodeDefaultAcl}}?
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
TODO:
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* Add and run unit tests
* Run system compatibility tests
was (Author: jzhuge):
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
TODO:
* Add and run unit tests
* Run system compatibility tests
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-6962:
-
Attachment: HDFS-6962.002.patch
Patch 002 (prototype):
* Create a new class {{FsPermissionDuo}} that extends {{FsPermission}} to store
both masked and unmasked permissions. HDFS client uses it to sneak unmasked
permission from FileContext/FileSystem -> AFS -> Hdfs -> DFSClient -> RPC. NN
uses it to sneak unmasked permission from RPC -> NameNodeRpcServer.create
(placed into PermissionStatus) -> FSNamesystem.startFile ->
FSDirWriterFileOp.startFile/addFile -> INodeFile -> INodeWithAdditionalFields.
* Add field {{unmasked}} to protobuf message {{CreateRequestProto}} and
{{MkdirsRequestProto}}
* Modify {{copyINodeDefaultAcl}} to switch between old and new ACL inheritance
behavior.
TODO:
* Add and run unit tests
* Run system compatibility tests
* Anyway to avoid adding field FsPermissionDuo to {{INodeWithAdditionalFields}}?
* Investigate the use of permissions in FSDirMkdirOp.createAncestorDirectories
call tree
* {{PermissionStatus#applyUMask}} never used, remove it?
* {{DFSClient#mkdirs}} and DFSClient#primitiveMkdir}} use file default if
permission is null. Should use dir default permission?
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch,
> HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4821) It's possible to create files with special characters in the filenames, but 'hadoop fs -ls' gives no indication
[ https://issues.apache.org/jira/browse/HDFS-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved HDFS-4821. -- Resolution: Duplicate > It's possible to create files with special characters in the filenames, but > 'hadoop fs -ls' gives no indication > --- > > Key: HDFS-4821 > URL: https://issues.apache.org/jira/browse/HDFS-4821 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Stephen Fritz > > For example: > -bash-4.1$ hadoop fs -mkdir /user/hdfs > -bash-4.1$ hadoop fs -touchz /user/hdfs/dupfile > -bash-4.1$ hadoop fs -touchz /user/hdfs/dupfile^M > -bash-4.1$ hadoop fs -ls /user/hdfs > Found 2 items > -rw-r--r-- 3 hdfs supergroup 0 2013-05-14 07:13 /user/hdfs/dupfile > -rw-r--r-- 3 hdfs supergroup 0 2013-05-14 07:13 /user/hdfs/dupfile > By way of comparison, bash will print a '?' at the end of the file name, > indicating there's a special character in the filename, which isn't perfect > but at least gives an indication that there's more to the filename than just > alphanumeric characters -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624
]
John Zhuge edited comment on HDFS-6962 at 6/7/16 5:07 AM:
--
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
//
// HDFS-6962. POSIX ACL inheritance.
//
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
//
// Old behavior before HDFS-6962.
//
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request comes from a client with HDFS-6962 fix and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
was (Author: jzhuge):
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
/*
* HDFS-6962. POSIX ACL inheritance.
*/
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
/*
* Old behavior before HDFS-6962.
*/
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
AclStorage.copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request come from a client that supports POSIX ACL inheritance and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624
]
John Zhuge edited comment on HDFS-6962 at 6/7/16 5:04 AM:
--
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
/*
* HDFS-6962. POSIX ACL inheritance.
*/
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
/*
* Old behavior before HDFS-6962.
*/
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
AclStorage.copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request come from a client that supports POSIX ACL inheritance and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
was (Author: jzhuge):
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance}}, default false in Hadoop
2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* The meaning of the existing field {{FsPermissionProto masked}} stays the
same: mode + umask, for both old and new client.
* Please note this approach is slightly different alternative what you
suggested. I am ok either way.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
public static void copyINodeDefaultAcl(INode child) {
// Current permission is masked
assert child.getFsPermission().equals(child.getMaskedMode());
if (child.getPosixAclInheritance()) {
// Set permission to unmasked
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
if (child.getPosixAclInheritance()) {
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
}
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#getPosixAclInheritance}} returns true when from a new client and the
flag is on.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317440#comment-15317440
]
John Zhuge commented on HDFS-10375:
---
Looks good. BTW, it shoulda been named 004.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317358#comment-15317358
]
John Zhuge edited comment on HDFS-10375 at 6/6/16 10:09 PM:
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003.
Please merge them into 1 commit by running {{git rebase -i HEAD~2}} (fixup the
newer commit) then run format-patch again.
was (Author: jzhuge):
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003. In
this case, please run {{git format-patch --stdout HEAD~2 >
HDFS-10375.NNN.patch}}.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317358#comment-15317358
]
John Zhuge commented on HDFS-10375:
---
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003. In
this case, please run {{git format-patch --stdout HEAD~2 >
HDFS-10375.NNN.patch}}.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316975#comment-15316975
]
John Zhuge commented on HDFS-10375:
---
You can upload multiple patches as long as they are all numbered sequentially
which you did.
On you local machines, just run {{git commit --amend}} then {{git format-patch
--stdout HEAD~ > HDFS-10375.NNN.patch}} (increment NNN each time).
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316960#comment-15316960
]
John Zhuge commented on HDFS-10375:
---
Thanks [~clouderajiayi] for contributing the patches. There is no need to
remove the the previous patch before adding a new one.
+1(non-binding) LGTM.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Hadoop Flags: Reviewed
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316919#comment-15316919
]
John Zhuge commented on HDFS-10488:
---
[~wchevreuil] Shouldn't you take the ownership of this jira?
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user&op=MKDIRS";
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Comment: was deleted
(was: +1(non-binding) after reverting the change in {{ValueQueue.java}}. File a
new jira to fix that typo.)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Summary: Remove redundant TestMiniDFSCluster.testDualClusters (was: Remove
redundent TestMiniDFSCluster.testDualClusters)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316910#comment-15316910
]
John Zhuge commented on HDFS-10375:
---
+1(non-binding) after reverting the change in {{ValueQueue.java}}. File a new
jira to fix that typo.
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316911#comment-15316911
]
John Zhuge commented on HDFS-10375:
---
+1(non-binding) after reverting the change in {{ValueQueue.java}}. File a new
jira to fix that typo.
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314624#comment-15314624
]
John Zhuge commented on HDFS-6962:
--
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance}}, default false in Hadoop
2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* The meaning of the existing field {{FsPermissionProto masked}} stays the
same: mode + umask, for both old and new client.
* Please note this approach is slightly different alternative what you
suggested. I am ok either way.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
public static void copyINodeDefaultAcl(INode child) {
// Current permission is masked
assert child.getFsPermission().equals(child.getMaskedMode());
if (child.getPosixAclInheritance()) {
// Set permission to unmasked
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
if (child.getPosixAclInheritance()) {
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
}
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#getPosixAclInheritance}} returns true when from a new client and the
flag is on.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx access for inhéritance.
> default:mask::rwx inhéritance mask is rwx, so no mask
> 4/ Create a subdir to test inheritance of ACL
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs
> 5/ check ACLs /tmp/ACLS/hdfs
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs
> # file: /tmp/ACLS/hdfs
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:r-x
> group::r-x
> group:readwrite:rwx #effective:r-x
> mask::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> Here we can see that the readwrite group has rwx ACL bu only r-x is effective
> because the mask is r-x (mask::r-x) in spite of default mask for inheritance
> is set to default:mask::rwx on /tmp/ACLS/
> 6/ Modifiy hdfs-site.xml et restart namenode
>
> dfs.umaskmode
> 010
>
> 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode
> bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2
> 8/ Check ACL on /tmp/ACLS/hdfs2
> bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2
> # file: /tmp/ACLS/hdfs2
> # owner: hdfs
> # group: hadoop
> user::rwx
> user:toto:rwx #effective:rw-
> group::r-x #effective:r--
> group:readwrite:rwx #effective:rw-
> mask::rw-
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> So HDFS masks the ACL value (user, group and other -- exepted the POSIX
> owner -- ) with the group mask of dfs.umaskmode properties when creating
> directory with inherited ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: HDFS-6962.001.patch Patch 001: * Just a rebase of HDFS-6962.1.patch * Pass build > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10370) Allow DataNode to be started with numactl
[
https://issues.apache.org/jira/browse/HDFS-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306974#comment-15306974
]
John Zhuge edited comment on HDFS-10370 at 5/30/16 11:27 PM:
-
Thanks [~dlmarion] for uploading the new patches.
Comments for file {{hdfs}} in {{HDFS-10370-branch-2.004.patch}}.
* 151-158: How about replacing the block with
{code}
if [[ ${HADOOP_DN_NUMACTL_ENABLE} == true ]] ; then
NUMACTL_ARGS=${HADOOP_DN_NUMACTL_ARGS:-"--interleave=all"}
NUMACTL_CMD="$(command -v numactl)"
fi
{code}
* Line 154: It is discouraged to use {{which}} to detect executable. See
explanation in
http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script/677212#677212.
* 315 and 317: These 2 lines are almost identical, any way to remove redundancy?
* 317: Where is {{NUMACTL_ARGS}} initialized? Is it supposed to take the value
from {{HADOOP_DN_NUMACTL_ARGS}}?
* 317: Inconsistent prefix between {{NUMA_CMD}} and {{NUMACTL_ARGS}}
[~aw] Any generic way to run numactl for Hadoop daemons with different numactl
args? How to suppot JSVC? Maybe using JNI {{set_mempolicy}}?
was (Author: jzhuge):
Thanks [~dlmarion] for uploading the new patches.
Comments for file {{hdfs}} in {{HDFS-10370-branch-2.004.patch}}.
* 151-158: How about replacing the block with
{code}
if [[ ${HADOOP_DN_NUMACTL_ENABLE == true ]] ; then
NUMACTL_ARGS=${HADOOP_DN_NUMACTL_ARGS:-"--interleave=all"}
NUMACTL_CMD=$(command -v numactl)
fi
{code}
* Line 154: It is discouraged to use {{which}} to detect executable. See
explanation in
http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script/677212#677212.
* 315 and 317: These 2 lines are almost identical, any way to remove redundancy?
* 317: Where is {{NUMACTL_ARGS}} initialized? Is it supposed to take the value
from {{HADOOP_DN_NUMACTL_ARGS}}?
* 317: Inconsistent prefix between {{NUMA_CMD}} and {{NUMACTL_ARGS}}
[~aw] Any generic way to run numactl for Hadoop daemons with different numactl
args? How to suppot JSVC? Maybe using JNI {{set_mempolicy}}?
> Allow DataNode to be started with numactl
> -
>
> Key: HDFS-10370
> URL: https://issues.apache.org/jira/browse/HDFS-10370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Dave Marion
>Assignee: Dave Marion
> Attachments: HDFS-10370-1.patch, HDFS-10370-2.patch,
> HDFS-10370-3.patch, HDFS-10370-branch-2.004.patch, HDFS-10370.004.patch
>
>
> Allow numactl constraints to be applied to the datanode process. The
> implementation I have in mind involves two environment variables (enable and
> parameters) in the datanode startup process. Basically, if enabled and
> numactl exists on the system, then start the java process using it. Provide a
> default set of parameters, and allow the user to override the default. Wiring
> this up for the non-jsvc use case seems straightforward. Not sure how this
> can be supported using jsvc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10370) Allow DataNode to be started with numactl
[
https://issues.apache.org/jira/browse/HDFS-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306974#comment-15306974
]
John Zhuge commented on HDFS-10370:
---
Thanks [~dlmarion] for uploading the new patches.
Comments for file {{hdfs}} in {{HDFS-10370-branch-2.004.patch}}.
* 151-158: How about replacing the block with
{code}
if [[ ${HADOOP_DN_NUMACTL_ENABLE == true ]] ; then
NUMACTL_ARGS=${HADOOP_DN_NUMACTL_ARGS:-"--interleave=all"}
NUMACTL_CMD=$(command -v numactl)
fi
{code}
* Line 154: It is discouraged to use {{which}} to detect executable. See
explanation in
http://stackoverflow.com/questions/592620/check-if-a-program-exists-from-a-bash-script/677212#677212.
* 315 and 317: These 2 lines are almost identical, any way to remove redundancy?
* 317: Where is {{NUMACTL_ARGS}} initialized? Is it supposed to take the value
from {{HADOOP_DN_NUMACTL_ARGS}}?
* 317: Inconsistent prefix between {{NUMA_CMD}} and {{NUMACTL_ARGS}}
[~aw] Any generic way to run numactl for Hadoop daemons with different numactl
args? How to suppot JSVC? Maybe using JNI {{set_mempolicy}}?
> Allow DataNode to be started with numactl
> -
>
> Key: HDFS-10370
> URL: https://issues.apache.org/jira/browse/HDFS-10370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Dave Marion
>Assignee: Dave Marion
> Attachments: HDFS-10370-1.patch, HDFS-10370-2.patch,
> HDFS-10370-3.patch, HDFS-10370-branch-2.004.patch, HDFS-10370.004.patch
>
>
> Allow numactl constraints to be applied to the datanode process. The
> implementation I have in mind involves two environment variables (enable and
> parameters) in the datanode startup process. Basically, if enabled and
> numactl exists on the system, then start the java process using it. Provide a
> default set of parameters, and allow the user to override the default. Wiring
> this up for the non-jsvc use case seems straightforward. Not sure how this
> can be supported using jsvc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10370) Allow DataNode to be started with numactl
[ https://issues.apache.org/jira/browse/HDFS-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298631#comment-15298631 ] John Zhuge commented on HDFS-10370: --- Thanks for the clarifications. I understand the current focus is on "interleave memory allocations", do you have any performance test results? > Allow DataNode to be started with numactl > - > > Key: HDFS-10370 > URL: https://issues.apache.org/jira/browse/HDFS-10370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Dave Marion >Assignee: Dave Marion > Attachments: HDFS-10370-1.patch, HDFS-10370-2.patch, > HDFS-10370-3.patch > > > Allow numactl constraints to be applied to the datanode process. The > implementation I have in mind involves two environment variables (enable and > parameters) in the datanode startup process. Basically, if enabled and > numactl exists on the system, then start the java process using it. Provide a > default set of parameters, and allow the user to override the default. Wiring > this up for the non-jsvc use case seems straightforward. Not sure how this > can be supported using jsvc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
