[jira] [Commented] (HDFS-8337) httpfs doesn't work with creates from a jar with kerberos
[
https://issues.apache.org/jira/browse/HDFS-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578475#comment-14578475
]
Alejandro Abdelnur commented on HDFS-8337:
--
I'll look. BTW, a workaround until there is a release with this fix is adding
the following to the {{httpfs-site.xml}}:
{code}
httpfs.authentication.type
org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticationHandler
httpfs.authentication.delegation-token.token-kind
WEBHDFS delegation
{code}
> httpfs doesn't work with creates from a jar with kerberos
> -
>
> Key: HDFS-8337
> URL: https://issues.apache.org/jira/browse/HDFS-8337
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, hdfs-client, security, webhdfs
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-8337.001.patch, HDFS-8337.002.patch,
> HDFS-8337.003.patch
>
>
> In a secure cluster, running a simple program:
> {code}
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.fs.*;
> import org.apache.hadoop.security.*;
> class Foo {
> public static void main(String args[]) throws Exception {
> FileSystem fs = FileSystem.get(new
> java.net.URI("webhdfs://:14000/"), new Configuration());
> System.out.println(fs.listStatus(new Path("/"))[0]);
> java.io.OutputStream os = fs.create(new Path("/tmp/foo"));
> os.write('a');
> os.close();
> }
> }
> {code}
> Basically to access httpfs via webhdfs, the following exception is thrown:
> {code}
> [systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop
> classpath):. Foo
> 15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException):
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
> at Foo.main(Foo.java:7)
> {code}
> Thanks [~qwertymaniac] and [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8337) httpfs doesn't work with creates from a jar with kerberos
[
https://issues.apache.org/jira/browse/HDFS-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578356#comment-14578356
]
Alejandro Abdelnur commented on HDFS-8337:
--
[~yzhangal], thanks for looking into this. After digging a bit in the code, I
think the intended way for this to work is that HTTPFS authentication filter
should do the following in its {{getConfiguration()}} method (KMS does exactly
this):
{code}
String authType = props.getProperty(AUTH_TYPE);
if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
props.setProperty(AUTH_TYPE,
PseudoDelegationTokenAuthenticationHandler.class.getName());
} else if (authType.equals(KerberosAuthenticationHandler.TYPE)) {
props.setProperty(AUTH_TYPE,
KerberosDelegationTokenAuthenticationHandler.class.getName());
}
props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND,
WebHdfsConstants.WEBHDFS_TOKEN_KIND.toString());
{code}
> httpfs doesn't work with creates from a jar with kerberos
> -
>
> Key: HDFS-8337
> URL: https://issues.apache.org/jira/browse/HDFS-8337
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, hdfs-client, security, webhdfs
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-8337.001.patch, HDFS-8337.002.patch,
> HDFS-8337.003.patch
>
>
> In a secure cluster, running a simple program:
> {code}
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.fs.*;
> import org.apache.hadoop.security.*;
> class Foo {
> public static void main(String args[]) throws Exception {
> FileSystem fs = FileSystem.get(new
> java.net.URI("webhdfs://:14000/"), new Configuration());
> System.out.println(fs.listStatus(new Path("/"))[0]);
> java.io.OutputStream os = fs.create(new Path("/tmp/foo"));
> os.write('a');
> os.close();
> }
> }
> {code}
> Basically to access httpfs via webhdfs, the following exception is thrown:
> {code}
> [systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop
> classpath):. Foo
> 15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException):
> java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
> at Foo.main(Foo.java:7)
> {code}
> Thanks [~qwertymaniac]
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350786#comment-14350786 ] Alejandro Abdelnur commented on HDFS-6200: -- Haohui, Could you please list the actual set of dependencies the hdfs-client will carry? > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350774#comment-14350774 ] Alejandro Abdelnur commented on HDFS-6200: -- Haohui, Doing what hadoop-client wont solve the problems you want to tackle, it will just remove the JARs used on the HDFS server side only. If you just care about those server side dependencies, hadoop-client should be enough and you could exclude YARN/MR artifacts in your dependency. If you want take care of guava, commons-*, etc, etc, you'll need to classloader magic for the filesystem impls, and this should be done in common where the Hadoop FileSystem API lives so all Hadoop FileSystem implementations get this kind of isolation. > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313630#comment-14313630 ] Alejandro Abdelnur commented on HDFS-6826: -- [~asuresh], sure no prob. Thanks for following up with it. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, > HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, > HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, > HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFS-6826v9.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14149886#comment-14149886
]
Alejandro Abdelnur commented on HDFS-6826:
--
[~daryn], thanks for the detailed review. A few comments on it.
On #2, I don’t see how this would happen unless you are adding a new
authorization property or a new way of checking permissions independent of the
current one.
On #4, my bad, I’ve missed the special entry point of webhdfs, that is easily
fixable.
ON #5, AFAIK the performance of thread locals has improved significantly since
its inception. I don’t think this should be an issue.
On #8,the {{DefaultAuthorizationProvider}} is the implementation provide by
Hadoop. Also, its implementation is not ‘public’.
On #9, I think is a reasonable assumption for now to have a single authz
provider. If more than one, that could be done with a multiplexor
implementation (as you could assume they will have zero intersection on the
HDFS sub-trees they manage.
Also, I think that because v7.6 works on resolved INodes, we don’t need to
worry about snapshots paths or inodes paths.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch,
> HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch,
> HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch,
> HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch,
> HDFS-6826v8.patch, HDFS-6826v9.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7086) httpfs create files default overwrite behavior is set to true
[ https://issues.apache.org/jira/browse/HDFS-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned HDFS-7086: Assignee: (was: Alejandro Abdelnur) > httpfs create files default overwrite behavior is set to true > - > > Key: HDFS-7086 > URL: https://issues.apache.org/jira/browse/HDFS-7086 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 2.1.0-beta, 2.2.0, 2.3.0, 2.4.1, 2.5.1 > Environment: Linux, Java >Reporter: Eric Yang > > WebHDFS documentation says overwrite flag is default to false, but httpfs set > the flag to true by default. This can be different from user's expectation > and cause data to be overwritten. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7004) Update KeyProvider instantiation to create by URI
[
https://issues.apache.org/jira/browse/HDFS-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138085#comment-14138085
]
Alejandro Abdelnur commented on HDFS-7004:
--
one NIT, in the {{KMSConfiguration.java}} file, the {{KEY_PROVIDER_URI}} value
seems to miss {{CONFIG_PREFIX + }}.
+1 after that and jenkins.
> Update KeyProvider instantiation to create by URI
> -
>
> Key: HDFS-7004
> URL: https://issues.apache.org/jira/browse/HDFS-7004
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption
>Affects Versions: 2.6.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-7004.001.patch, hdfs-7004.002.patch
>
>
> See HADOOP-11054, would be good to update the NN/DFSClient to fetch via this
> method rather than depending on the URI path lookup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7075) hadoop-fuse-dfs fails because it cannot find JavaKeyStoreProvider$Factory
[
https://issues.apache.org/jira/browse/HDFS-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138000#comment-14138000
]
Alejandro Abdelnur commented on HDFS-7075:
--
+1.
> hadoop-fuse-dfs fails because it cannot find JavaKeyStoreProvider$Factory
> -
>
> Key: HDFS-7075
> URL: https://issues.apache.org/jira/browse/HDFS-7075
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7075.001.patch
>
>
> hadoop-fuse-dfs fails complaining with:
> {code}
> java.util.ServiceConfigurationError:
> org.apache.hadoop.crypto.key.KeyProviderFactory: Provider
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory not found
> {code}
> Here is an example of the hadoop-fuse-dfs debug output.
> {code}
> 14/09/04 13:49:04 WARN crypto.CryptoCodec: Crypto codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
> hdfsBuilderConnect(forceNewInstance=1,
> nn=hdfs://hdfs-cdh5-secure-1.vpc.cloudera.com:8020, port=0,
> kerbTicketCachePath=/tmp/krb5cc_0, userName=root) error:
> java.util.ServiceConfigurationError:
> org.apache.hadoop.crypto.key.KeyProviderFactory: Provider
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:231)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk and branch-2. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.6.0 > > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch, > HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136272#comment-14136272 ] Alejandro Abdelnur commented on HDFS-7006: -- test failure unrelated. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch, > HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7075) hadoop-fuse-dfs fails because it cannot find JavaKeyStoreProvider$Factory
[
https://issues.apache.org/jira/browse/HDFS-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136188#comment-14136188
]
Alejandro Abdelnur commented on HDFS-7075:
--
Well, you could still have the issue that you have multiple keyprovider impls
in different JARs. You have to make sure you are using the ClassLoader that
contains them all.
> hadoop-fuse-dfs fails because it cannot find JavaKeyStoreProvider$Factory
> -
>
> Key: HDFS-7075
> URL: https://issues.apache.org/jira/browse/HDFS-7075
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7075.001.patch
>
>
> hadoop-fuse-dfs fails complaining with:
> {code}
> java.util.ServiceConfigurationError:
> org.apache.hadoop.crypto.key.KeyProviderFactory: Provider
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory not found
> {code}
> Here is an example of the hadoop-fuse-dfs debug output.
> {code}
> 14/09/04 13:49:04 WARN crypto.CryptoCodec: Crypto codec
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
> hdfsBuilderConnect(forceNewInstance=1,
> nn=hdfs://hdfs-cdh5-secure-1.vpc.cloudera.com:8020, port=0,
> kerbTicketCachePath=/tmp/krb5cc_0, userName=root) error:
> java.util.ServiceConfigurationError:
> org.apache.hadoop.crypto.key.KeyProviderFactory: Provider
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:231)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[
https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136155#comment-14136155
]
Alejandro Abdelnur commented on HDFS-6987:
--
Also, when doing this JIRA, in the {{MiniKDC.java}}, we should remove the
{code}
kms.setBoolean(KMSConfiguration.KEY_AUTHORIZATION_ENABLE, false);
{code}
introduced by HDFS-7006.
> Move CipherSuite xattr information up to the encryption zone root
> -
>
> Key: HDFS-6987
> URL: https://issues.apache.org/jira/browse/HDFS-6987
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption
>Affects Versions: 2.6.0
>Reporter: Andrew Wang
>Assignee: Zhe Zhang
>
> All files within a single EZ need to be encrypted with the same CipherSuite.
> Because of this, I think we can store the CipherSuite once in the EZ rather
> than on each file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7074) KMS: KeyAuthorizationKeyProvider should verify the keyversion belongs to the keyname on decrypt
Alejandro Abdelnur created HDFS-7074:
Summary: KMS: KeyAuthorizationKeyProvider should verify the
keyversion belongs to the keyname on decrypt
Key: HDFS-7074
URL: https://issues.apache.org/jira/browse/HDFS-7074
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
when decrypting a EEK the {{KeyAuthorizationKeyProvider}} must verify the
keyversionname belongs to the keyname.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[ https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136118#comment-14136118 ] Alejandro Abdelnur commented on HDFS-6987: -- We need also to get keyname in the same way. Currently DFSClient is re-creating the EEK to send for decryption is sending a NULL keyname. For key based ACLs (HADOOP-10758), the keyname is required. I'll open a JIRA for the key based ACLs KeyProvider to verify the keyversion belongs to the keyname. > Move CipherSuite xattr information up to the encryption zone root > - > > Key: HDFS-6987 > URL: https://issues.apache.org/jira/browse/HDFS-6987 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.6.0 >Reporter: Andrew Wang >Assignee: Zhe Zhang > > All files within a single EZ need to be encrypted with the same CipherSuite. > Because of this, I think we can store the CipherSuite once in the EZ rather > than on each file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Attachment: HDFS-7006.patch After a few hours of debugging (courtesy of Jetty handling webapps completely differently when in directory, JAR and WARs) I've sorted this one out. I've merged HADOOP-11075 in this patch, it required some tweaks. I had to do add a special handling for the webapp in the MiniKDC (because of Jetty's webapp handling). I've tested MiniKMS works when consumed from a directory in the classpath and when consumed from Maven local repo as JAR. Also, I've verified all necessary JARs are published to the maven repo. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch, > HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Status: Patch Available (was: Open) > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch, > HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135635#comment-14135635 ] Alejandro Abdelnur commented on HDFS-6826: -- I've forgot to comment on the replay edits concerns, the v7.6 plugin API provides feedback to the plugin implementation indicating if it is within a client filesystem call or not (not is the case of replaying edit). > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, > HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, > HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, > HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFS-6826v9.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6826:
-
Attachment: HDFS-6826v9.patch
Attached, v9, is a functioning patch based on [~daryn] idea. It requires a
little refining.
A few comments on the approach:
* Instead being invasive on the {{INode}} classes, it is invasive in the
{{PermissionChecker}} and in the {{FSDirectory}}.
* It does not allow to replace the permission check logic (v7.x does) (could be
done in a follow up JIRA)
* It does not allow to replace the setter of authz info logic (v7.x does)
* It seems to be more efficient regarding the times it gets call and how the
full path can be inferred by the plugin.
* Talking with Daryn, he suggested doing some changes in the {{FSDirectory}}
for creating file status by passing. I'm deferring this to the refactoring he
wants to do as it is not that trivial.
Refining required:
* How to handle setting of authz info, meaning fully ignoring setter calls for
a path managed by a plugin. Else setters have effect. The plugin should be able
to prevent those setter calls from happening.
* subAccess check is a TODO, the stack logic there needs to carry more info
for the plugin (but we don't want to do that if there is no plugin).
The v7.6 and the v9 plugins provide the functionality it is needed for
HMS/Sentry. I'm OK either way.
Could others weight on the preferred approach? I would like to get closure on
this ASAP for Sentry to start building on top of it.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch,
> HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch,
> HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch,
> HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch,
> HDFS-6826v8.patch, HDFS-6826v9.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Status: Open (was: Patch Available) testcases pass when run from hadoop source root, they don't when run from hdfs source root (using kms stuff from JARs). Jetty WebAppContext has different behavior when the webapp is working out or an expanded directory or out a JAR. Looking on how to make MiniKMS to work correctly in both scenarios. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6776) Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6776:
-
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks YongJun and thanks Haohui for reviewing it.
Committed to trunk and branch-2.
> Using distcp to copy data between insecure and secure cluster via webdhfs
> doesn't work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Fix For: 2.6.0
>
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, HDFS-6776.010.patch, HDFS-6776.011.patch,
> HDFS-6776.012.nomsgparsing.patch, HDFS-6776.013.patch, HDFS-6776.014.patch,
> HDFS-6776.015.patch, HDFS-6776.016.patch, HDFS-6776.017.patch,
> dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.jav
[jira] [Commented] (HDFS-6776) Using distcp to copy data between insecure and secure cluster via webdhfs doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128003#comment-14128003
]
Alejandro Abdelnur commented on HDFS-6776:
--
I'll commit this momentarily, if there are further discussions on how to make
the testcase more accurate we can follow up in a new JIRA.
> Using distcp to copy data between insecure and secure cluster via webdhfs
> doesn't work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, HDFS-6776.010.patch, HDFS-6776.011.patch,
> HDFS-6776.012.nomsgparsing.patch, HDFS-6776.013.patch, HDFS-6776.014.patch,
> HDFS-6776.015.patch, HDFS-6776.016.patch, HDFS-6776.017.patch,
> dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.Web
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Attachment: COMBO.patch uploading COMBO of HDFS-7006 & HADOOP-11075 to see if this fixes test-patch run. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HDFS-7006.patch, HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127603#comment-14127603
]
Alejandro Abdelnur commented on HDFS-6826:
--
[~jnp], the {{AuthorizationProviderProxyClientProtocol}} does not do any harm
being there the whole time, I would prefer to keep it to avoid having to
propagate to the {{ClientNamenodeProtocolServerSideTranslatorPB}} the knowledge
of the plugin being used. What is your concern in having it always?
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
> HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
> HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch,
> HDFS-6826v7.patch, HDFS-6826v8.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127545#comment-14127545
]
Alejandro Abdelnur commented on HDFS-6776:
--
[~wheat9], do you have any concern with the patch v17? Else I'd like to commit
it by EOD PST today.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work via webhdfs
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, HDFS-6776.010.patch, HDFS-6776.011.patch,
> HDFS-6776.012.nomsgparsing.patch, HDFS-6776.013.patch, HDFS-6776.014.patch,
> HDFS-6776.015.patch, HDFS-6776.016.patch, HDFS-6776.017.patch,
> dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Abstra
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127543#comment-14127543
]
Alejandro Abdelnur commented on HDFS-6776:
--
+1. Patch v17 LGTM.
IMO it will be a pita for users with multiple clusters the fact that we are not
parsing the exception message. Please lets have a follow up JIRA for it.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work via webhdfs
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, HDFS-6776.010.patch, HDFS-6776.011.patch,
> HDFS-6776.012.nomsgparsing.patch, HDFS-6776.013.patch, HDFS-6776.014.patch,
> HDFS-6776.015.patch, HDFS-6776.016.patch, HDFS-6776.017.patch,
> dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.
[jira] [Commented] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125921#comment-14125921
]
Alejandro Abdelnur commented on HDFS-6606:
--
{{DataTransferSaslUtil.java#negotiateCipherOption()}} method, hardcoding 16
bytes (128bits) for keys, any reason this is not configurable so we can use 192
or 256?
Also, if we are transferring data for a file that is in an encryption zone, the
data is already encrypted. Thus, we could do the transfer without encryption
and avoid a the penalty of an unnecessary double encryption. Though we can
tackle this in a follow up JIRA.
> Optimize HDFS Encrypted Transport performance
> -
>
> Key: HDFS-6606
> URL: https://issues.apache.org/jira/browse/HDFS-6606
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, security
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
> HDFS-6606.003.patch, OptimizeHdfsEncryptedTransportperformance.pdf
>
>
> In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
> it was a great work.
> It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
> three security strength:
> * high 3des or rc4 (128bits)
> * medium des or rc4(56bits)
> * low rc4(40bits)
> 3des and rc4 are slow, only *tens of MB/s*,
> http://www.javamex.com/tutorials/cryptography/ciphers.shtml
> http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
> I will give more detailed performance data in future. Absolutely it’s
> bottleneck and will vastly affect the end to end performance.
> AES(Advanced Encryption Standard) is recommended as a replacement of DES,
> it’s more secure; with AES-NI support, the throughput can reach nearly
> *2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
> supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
> a new mode support for AES).
> This JIRA will use AES with AES-NI support as encryption algorithm for
> DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Status: Patch Available (was: Open) > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-7006.patch, HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with KMS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Attachment: HDFS-7006.patch rebased patch. > Test encryption zones with KMS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-7006.patch, HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6986:
-
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Zhe Zhang. Committed to trunk and branch-2.
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
> Fix For: 2.6.0
>
> Attachments: HDFS-6986-20140905-v2.patch,
> HDFS-6986-20140905-v3.patch, HDFS-6986-20140905.patch, HDFS-6986.patch
>
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124335#comment-14124335
]
Alejandro Abdelnur commented on HDFS-6986:
--
+1, test failure seems unrelated.
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
> Attachments: HDFS-6986-20140905-v2.patch,
> HDFS-6986-20140905-v3.patch, HDFS-6986-20140905.patch, HDFS-6986.patch
>
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123578#comment-14123578
]
Alejandro Abdelnur commented on HDFS-6986:
--
can we do the test stronger, asserting we are getting the token set by the mock?
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
> Attachments: HDFS-6986-20140905.patch, HDFS-6986.patch
>
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7006) Test encryption zones with MKS
[ https://issues.apache.org/jira/browse/HDFS-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-7006: - Attachment: HDFS-7006.patch > Test encryption zones with MKS > -- > > Key: HDFS-7006 > URL: https://issues.apache.org/jira/browse/HDFS-7006 > Project: Hadoop HDFS > Issue Type: Test > Components: security, test >Affects Versions: 2.6.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-7006.patch > > > We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7006) Test encryption zones with MKS
Alejandro Abdelnur created HDFS-7006: Summary: Test encryption zones with MKS Key: HDFS-7006 URL: https://issues.apache.org/jira/browse/HDFS-7006 Project: Hadoop HDFS Issue Type: Test Components: security, test Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur We should test EZs with KMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123488#comment-14123488
]
Alejandro Abdelnur commented on HDFS-6986:
--
I've tested the provided patch in a real cluster and it works as advertised.
Please add testcase and we are good to go.
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
> Attachments: HDFS-6986.patch
>
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6905: - Fix Version/s: (was: 3.0.0) 2.6.0 committed to branch-2. > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Fix For: 2.6.0 > > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120335#comment-14120335
]
Alejandro Abdelnur commented on HDFS-6986:
--
yep, {{TestEncryptionZones}} seems the natural place.
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
> Attachments: HDFS-6986.patch
>
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6705) Create an XAttr that disallows the HDFS admin from accessing a file
[ https://issues.apache.org/jira/browse/HDFS-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120217#comment-14120217 ] Alejandro Abdelnur commented on HDFS-6705: -- I would keep this simple, a UNREADABLE_BY_SUPERUSER xattr, that is settable only (cannot be removed) and it works on ALL files (regardless of HDFS encryption), it prevent a superuser from opening the file. > Create an XAttr that disallows the HDFS admin from accessing a file > --- > > Key: HDFS-6705 > URL: https://issues.apache.org/jira/browse/HDFS-6705 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: 3.0.0 >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HDFS-6705.001.patch, HDFS-6705.002.patch, > HDFS-6705.003.patch > > > There needs to be an xattr that specifies that the HDFS admin can not access > a file. This is needed for m/r delegation tokens and data at rest encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6986) DistributedFileSystem must get delegation tokens from configured KeyProvider
[
https://issues.apache.org/jira/browse/HDFS-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118902#comment-14118902
]
Alejandro Abdelnur commented on HDFS-6986:
--
The changes in {{DistributedFileSystem.java}} should be something like:
{code}
@Override
public Token[] addDelegationTokens(String renewer, Credentials
credentials)
throws IOException {
Token[] tokens = super.addDelegationTokens(renewer, credentials);
if (dfs.getKeyProvider() != null) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
Token[] kpTokens = keyProviderDelegationTokenExtension.
addDelegationTokens(renewer, credentials);
if (tokens != null && kpTokens != null) {
Token[] all = new Token[tokens.length + kpTokens.length];
System.arraycopy(tokens, 0, all, 0, tokens.length);
System.arraycopy(kpTokens, 0, all, tokens.length, kpTokens.length);
tokens = all;
} else {
tokens = (tokens != null) ? tokens : kpTokens;
}
}
return tokens;
}
{code}
And {{DFSClient}} should expose the keyprovider via a {{getKeyProvider()}}
method.
> DistributedFileSystem must get delegation tokens from configured KeyProvider
>
>
> Key: HDFS-6986
> URL: https://issues.apache.org/jira/browse/HDFS-6986
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.6.0
>Reporter: Alejandro Abdelnur
>Assignee: Zhe Zhang
>
> {{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides
> delegation tokens. {{DistributedFileSystem}} should augment the HDFS
> delegation tokens with the keyprovider ones so tasks can interact with
> keyprovider when it is a client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-6986) DistributedFileSystem must get deletagiontokens from configured KeyProvider
Alejandro Abdelnur created HDFS-6986:
Summary: DistributedFileSystem must get deletagiontokens from
configured KeyProvider
Key: HDFS-6986
URL: https://issues.apache.org/jira/browse/HDFS-6986
Project: Hadoop HDFS
Issue Type: Sub-task
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
{{KeyProvider}} via {{KeyProviderDelegationTokenExtension}} provides delegation
tokens. {{DistributedFileSystem}} should augment the HDFS delegation tokens
with the keyprovider ones so tasks can interact with keyprovider when it is a
client/server impl (KMS).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6843) Create FSDataInputStream & FSDataOutputStream isEncrypted() method
[
https://issues.apache.org/jira/browse/HDFS-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115980#comment-14115980
]
Alejandro Abdelnur commented on HDFS-6843:
--
errr, don't like it for the following reasons:
* it requires to open a file to see if it is encrypted
* it does not work on directories
* it requires the HDFS client to contact the KMS to decrypt the encryption key
(which won't be used)
Regarding TOCTOU concerns, this is not different than computing the splits for
an MR job and then running the MR job, if the files change between the split
computation and the task execution, the job is doomed. We are already under
immutability assumptions.
I would put it in the FileStatus, most of the time I'll have a filestatus at
hand, so checking if it is encrypted will be 0 cost (no extra RPC).
> Create FSDataInputStream & FSDataOutputStream isEncrypted() method
> --
>
> Key: HDFS-6843
> URL: https://issues.apache.org/jira/browse/HDFS-6843
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, security
>Affects Versions: 3.0.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-6843.001.patch, HDFS-6843.002.patch,
> HDFS-6843.003.patch
>
>
> FileStatus should have a 'boolean isEncrypted()' method. (it was in the
> context of discussing with AndreW about FileStatus being a Writable).
> Having this method would allow MR JobSubmitter do the following:
> -
> BOOLEAN intermediateEncryption = false
> IF jobconf.contains("mr.intermidate.encryption") THEN
> intermediateEncryption = jobConf.getBoolean("mr.intermidate.encryption")
> ELSE
> IF (I/O)Format INSTANCEOF File(I/O)Format THEN
> intermediateEncryption = ANY File(I/O)Format HAS a Path with status
> isEncrypted()==TRUE
> FI
> jobConf.setBoolean("mr.intermidate.encryption", intermediateEncryption)
> FI
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115891#comment-14115891
]
Alejandro Abdelnur commented on HDFS-6776:
--
LGTM
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work via webhdfs
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, HDFS-6776.010.patch, dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.
[jira] [Commented] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115547#comment-14115547
]
Alejandro Abdelnur commented on HDFS-6606:
--
Yi, the approach LGTM. I'll ping [~atm] in case I'm missing something.
My feedback on the patch:
*CryptoInputStream.java*:
* read(BB), the {{if (n >- 0) THEN ELSE}} section at the end, why do we need
this now and not before? or it is a bug in CIS?
*DataTransferSaslUtils.java*:
* requestedQopContainsPrivacy(), should’t we trim() the values in the set?
just in case somebody added a whitespace, or this is not possible?
* sendSaslMessageAnNegotiationCipherOptions(), can payload be NULL? or that is
an error if we get here? if the later we should throw an exception.
* createStreamPair(), we are using the same key and IV for both streams. we
should either use 2 different KEYs or 2 different IVs, not to repeat the use of
a KEY/IV pair. or simply exchanging 2 cipheroptions, one for IN and one for OUT.
* the use wrapping/unwrapping seems too asymmetrical how is implemented, is it
possible to have 2 symmetric methods used at the same level on both sides?
*hdfs.proto*:
* ChiperOption, I don’t think the fields should be optional, they are all
required, no?
*SaslDataTransferServer.java*:
* why do we need 2 confs?, what was wrong with the original one?
> Optimize HDFS Encrypted Transport performance
> -
>
> Key: HDFS-6606
> URL: https://issues.apache.org/jira/browse/HDFS-6606
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client, security
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
> OptimizeHdfsEncryptedTransportperformance.pdf
>
>
> In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
> it was a great work.
> It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
> three security strength:
> * high 3des or rc4 (128bits)
> * medium des or rc4(56bits)
> * low rc4(40bits)
> 3des and rc4 are slow, only *tens of MB/s*,
> http://www.javamex.com/tutorials/cryptography/ciphers.shtml
> http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
> I will give more detailed performance data in future. Absolutely it’s
> bottleneck and will vastly affect the end to end performance.
> AES(Advanced Encryption Standard) is recommended as a replacement of DES,
> it’s more secure; with AES-NI support, the throughput can reach nearly
> *2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
> supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
> a new mode support for AES).
> This JIRA will use AES with AES-NI support as encryption algorithm for
> DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Fix Version/s: (was: 3.0.0) 2.6.0 merged to branch-2. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Fix For: 2.6.0 > > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf, > fs-encryption.2014-08-18.patch, fs-encryption.2014-08-19.patch > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114385#comment-14114385
]
Alejandro Abdelnur commented on HDFS-6826:
--
Regarding my action item #2, based on @daryn’s suggestion, I’ve move all data
authz interception to the {{FsPermissionChecker}}, something like:
*Adding the following methods:*
{code}
private String getUserName(INode[] nodes) {
return AuthorizationProvider.get().getUser(nodes);
}
private String getGroupName(INode[] nodes) {
// wee need to recreate INode[] using the snapshot version of the nodes,
here or before calling
return AuthorizationProvider.get().getGroup(nodes);
}
private FsPermission getFsPermission(INode[] nodes) {
return AuthorizationProvider.get().getFsPermission(nodes);
}
private AclFeature getAclFeature(INode[] nodes) {
return AuthorizationProvider.get().getAclFeature(nodes);
}
{code}
And then replacing all the calls user/group/permissions/acls getters of INodes
calls within {{FsPermissionChecker}} to use the above methods, ie:
The intention was to be able to reuse the already calculated {{INode[]}} chain.
The issues I’m running with this are:
* The {{INode[]}} chain calculated on the entry point of checkPermission does
not take into account snapshots, thus we will need to recalculate the
{{INode[]}} to use the right snapshot for the full chain.
* All the logic in {{FSPermissionChecker}} has to be redone to pass the
{{INode[]}} chain around. This will get tricky as in many places direct array
access like this 'checkStickyBit(inodes[inodes.length - 2], last, snapshotId);'
are being done. And this is not always done on the snapshot version of the
{{INode}}.
* The {{INode[]}} may have null elements, complicating things on the plugin
side.
* We’ll have to do the same in the {{FSDirectory}} to create file status.
Another issue is that doing this, the plugin is only intercepting getter calls,
not setter calls.
Overall, it seems the plugin will have to be more complex than with the v7
approach, it will have less functionality (no writes, no snapshots), and it
will require some serious rewriting of the {{FsPermissionChecker}}.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
> HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
> HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch,
> HDFS-6826v7.patch, HDFS-6826v8.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6826:
-
Attachment: HDFS-6826v7.6.patch
Uploading v7.6, which is based on v7.4 (based on Jitendra's preference).
This patch adds the {{isClientOp()}} method to the provider which indicates if
the current invocation is done as part of a client FS operation or as part of a
NameNode operation (fs image save, fs edit log, etc). I've had to create an
interceptor for the {{ClientProtocol}}.
The testcases do positive and negative testing for this.
This addresses my action item #1 from my discussion with @daryn.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
> HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
> HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch,
> HDFS-6826v7.patch, HDFS-6826v8.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1411#comment-1411 ] Alejandro Abdelnur commented on HDFS-6826: -- [~jnp], then v7.4 is what you would prefer? From yesterday's call I think Daryn and I misunderstood then that the requirement was to be able do add an additional assertion in the authorization path, not to fully replace it. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, > HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v7.5.patch v7.5 version, as per Daryn's (offline) feedback undoing all FsPermissionChecker changes and adding a call to the plugin during permission check. I have 2 additional action items: 1. see if special handling is required during edit log replay with v7. 2. toy with a v9 where interception is done at permission checker and filestatus creation level. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, > HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111578#comment-14111578
]
Alejandro Abdelnur commented on HDFS-6776:
--
IMO, enabling to work with an unpatched cluster (via message parsing) is a
desirable capability as it does not require users to upgrade older clusters if
they are just reading data from them.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work via webhdfs
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> HDFS-6776.009.patch, dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsF
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v7.4.patch Test failure pass locally, scanning the test output does not shield anything related to this patch. Uploading new v7 patch with some refactoring, making the authz provider and abstract class with singleton pattern access. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, > HDFS-6826v7.4.patch, HDFS-6826v7.patch, HDFS-6826v8.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Status: Patch Available (was: Open) > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, > HDFS-6826v7.patch, HDFS-6826v8.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6826:
-
Attachment: HDFS-6826v7.3.patch
Updated version of the v7 patch (v7.3), the problem was the
{{FSNamesystem#stopCommonServices()}} is sometimes called without a previous
call to {{startCommonServices()}}, thus a NPE was happening in the
non-initialized authz provider. I've guarded with an if-not-null the provider
stop() and all test-patch failing tests are now passing.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
> HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
> HDFS-6826v7.patch, HDFS-6826v8.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109835#comment-14109835 ] Alejandro Abdelnur commented on HDFS-6826: -- @daryn, bq. When a partition is added to a hive table, the hive server moves the file into a directory representing the table and chowns the file to itself. To do the chown it would require the HiveMetaStore to be a hdfs superuser. AFAIK, ACLs are not inherited at permission check time. They are inherited at creation time if the parent has DEFAULT ACLs. So, the chown would have to be done recursively. And, if somebody adds a file to a partition dir later on, it would not have the right settings. And still this is not solving my 'single source of authz info' as I could still set things in HDFS or in Sentry. I'm trying to come up with a model that enables authz to be delegated if the file/dir belongs to a table. Please let me know if you want to do another synchronous discussion round. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109640#comment-14109640
]
Alejandro Abdelnur commented on HDFS-6776:
--
Today, with HTTP APIs, we don't have an 'allow fallback to simple auth' switch.
It is something we should add.
Still, I don't think that will help address the usecase this JIRA tries to
solve, distcp-ing from an insecure cluster to a secure cluster. For this
usecase we will need to leave the switch in TRUE, which is today's current
behavior (without the switch).
I would suggest, we fix the current usecase, and we follow it up with a JIRA
adding the switch to the configuration and another to distcp.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
> dummy-token-proxy.js
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
>
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v8.patch @daryn, Thanks for following up on this. I really like your suggestion of using 'additional’ immutable ACLs to expose external permissions. I’ve made a version of the patch to see how it would work (see attachment HDFS-6826v8.patch). But I still have a couple of issues I’m not sure how to address them with your proposed approach: *Permissions remain settable in 2 places for 'table files'*, HDFS and the HiveMetaStore authorization source (i.e. Sentry). One of the motivations was to have a single source of permissions for 'table files’. I can constraint this to regular permissions (by having the plugin fully redacting the ACLs), but still I have 2 sources of authorization. Maybe being able to redact the regular permissions as well? (though you are advocating against that). *It is not possible to do a hand over*, with the previous proposals when a file was added to a table, the owner of the file would become hive:hive. If the original owner did not have GRANTs to the file, then s/he would not have access anymore. With your proposed approach, this is not possible as original owner remains. We also lose the capability of changing the 'permission check' logic Jitendra and Selvamohan wanted. Thoughts? > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.patch, > HDFS-6826v8.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Status: Open (was: Patch Available) canceling patch, there is some special handling for editlogs that has to be done, working on it. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v7.2.patch new patch cleaning up after testcase. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6905: - Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) thanks Charles. Committed to trunk. > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Fix For: 3.0.0 > > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106475#comment-14106475 ] Alejandro Abdelnur commented on HDFS-6905: -- +1 > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v7.1.patch uploaded patch taking care of the testcases failures (needed a bit of defensive coding as testcases are not always initializing everything), fixed javadoc warning. audit warnings are unrelated (the come from the fs-encryption merge and they are being addressed already). > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.1.patch, HDFS-6826v7.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6909) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
Alejandro Abdelnur created HDFS-6909: Summary: KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys Key: HDFS-6909 URL: https://issues.apache.org/jira/browse/HDFS-6909 Project: Hadoop HDFS Issue Type: Task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Currently is using JDK Cipher, with fs-encryption branch merged into trunk we can swap to CryptoCodec. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Status: Patch Available (was: Open) > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v7.patch v7, patch with javadocs and testcases. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFS-6826v7.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-3988) HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request
[ https://issues.apache.org/jira/browse/HDFS-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved HDFS-3988. -- Resolution: Done HADOOP-10771 took care of this. > HttpFS can't do GETDELEGATIONTOKEN without a prior authenticated request > > > Key: HDFS-3988 > URL: https://issues.apache.org/jira/browse/HDFS-3988 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.6.0 > > > A request to obtain a delegation token cannot it initiate an authentication > sequence, it must be accompanied by an auth cookie obtained in a prev request > using a different operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v6.patch uploading v6 patch: * add snapshot lifecycle methods and wires them to the SnapshotManager * it fixes how permissions are handled to be consistent with the other methods (receiving snapshotId on get) * fixes setters of user/group which were not properly wired * some classes/params/vars renaming for consistency Trying to answer on how an external plugin would handle snapshot info: A plugin willing to handle snapshot information would have to have a storage with versioned authorization information. Leveraging the global monotonic characteristics of snapshot IDs. If the provided snapshotId is CURRENT_STATE_ID, the latest version of authorization information must be returned. If the provided snaphotId is not CURRENT_STATE_ID, then the latest-no-greater-than snapshotId version of the authorization information must be provided. The setSnapshotableDirs method is call as NN startup to inform the plugin of the current snapshotable directories and their latest snapshot. This allows a plugin to initialize its storage accordingly (in case it was just being setup) or resync if necessary. The create/remove snapshotableDir methods, allow the plugin to setup and cleanup any necessary information regarding a directory being snapshotable. The create/remove snapshot methods, allow the plugin to tag/version and cleanup any necessary information regarding a snapshot being created or removed. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103321#comment-14103321
]
Alejandro Abdelnur commented on HDFS-6826:
--
[~andrew.wang],
By add/remove I've meant a directory becoming/ending to be snapshot-able.
The {{setSnaphostDirs()}} method would be called by the SnapshotManager on
startup. It would be called with values obtained from
{{getSnapshottableDirs()}} to seed the authzplugin with the current
snapshot-able dirs and their current snapshot info. This would allow the
plugin, on first ever to initialize its snapshot logic and on subsequent
restarts to verify/resync itself.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103164#comment-14103164 ] Alejandro Abdelnur commented on HDFS-6826: -- [~jinzhao], Wiring the authz plugin to the snapshot lifecycle requires the following methods: * setSnaphostDirs(Map) // called at start up time, allowing the plugin to know existing snapshots * addSnapshotDir(INode) * removeSnapshotDir(INode) * createSnapshot(INode, snapshotID) * deleteSnapshot(INode, snapshotID) The authz plugin is given to the SnapshotManager at startup, and the SnapshotManager would call these methods in the corresponding snapshot operations. This will allow the authz plugin (if desired) to keep track of snapshots and to perform its own snapshot management for authz information. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103076#comment-14103076
]
Alejandro Abdelnur commented on HDFS-6826:
--
I've poked [~andrew.wang] earlier today to get some info on snapshots.
My current patch is intercepting {{getPermissionLong()}}, it should intercept
{{getFsPermission()}}, then is consistent with all other getter methods that
takes snapshotId.
The plugin should have 2 additional methods {{createSnapshot()}} and
{{deleteSnapshot()}}, this would make the plugin participate in the snapshot
lifecycle and allow the plugin to implement snapshot logic if desired (it
doesn't have to do it).
I'll update the patch to reflect these changes.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102825#comment-14102825
]
Alejandro Abdelnur edited comment on HDFS-6776 at 8/19/14 9:08 PM:
---
latest patch looks good to me just a minor thing:
Can you please add a comment in the catch of the IOException indicating why we
are doing that (to support older clusters that do not throw the NullToken?
+1 from my side after this.
[~daryn], are you OK with the current approach?
was (Author: tucu00):
latest patch looks good to me just a minor thing:
Can you please add a comment in the catch of the IOException indicating why we
are doing that (to support older clusters that do not throw the NullToken?
+1 from my side after this.
@daryn, are you OK with the current approach?
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102825#comment-14102825
]
Alejandro Abdelnur commented on HDFS-6776:
--
latest patch looks good to me just a minor thing:
Can you please add a comment in the catch of the IOException indicating why we
are doing that (to support older clusters that do not throw the NullToken?
+1 from my side after this.
@daryn, are you OK with the current approach?
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFil
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6826:
-
Attachment: HDFS-6826v5.patch
[~jnp],
uploading v5 patch. I've removed the resolveLink. I've added 2 methods to the
{{INodeAuthorizationInfo}} interface {{getLocalName()}} and {{getParent}}.
IMO, it is up to a plugin to support snapshots or not. The default impl (native
HDFS) would, the current patch. An impl that does not care, can return current
authz info for the node regardless of the provided snapshotId. Using the full
path does not necessary work if supporting snapshots because of rename ops.
Regarding wrapping {{INode}} with a {{INodeAuthorizationInfo}} impl before
passing to the plugin, that would require creating a bunch of new objects (the
wrappers). When crafting the API I've took special care on no creating
additional objects in the authz methods. Thus, I would ike to keep it like that.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101354#comment-14101354
]
Alejandro Abdelnur commented on HDFS-6776:
--
[~yzhangal], how about the following:
* introduce a new exception on the server side
* on the client side, catch this new exception (this will work against a
cluster with the fix) and catch IOException looking for the string occurrence
in the message (this will work against a cluster without the fix).
BTW, testcases seem unrelated.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
> HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
> HDFS-6776.006.NullToken.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> o
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100924#comment-14100924
]
Alejandro Abdelnur commented on HDFS-6826:
--
Forgot to mention, the
{{AuthorizationProvider.INodeAuthorizationInfo.getPermissionLong()}} method
does not take a snapshotId because it seems there is special handling for
permissions and the snapshot-ed INode is queried. I'll take a look to see if it
make sense to tweak this to pass a snapshotId.
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
> HDFS-6826v3.patch, HDFS-6826v4.patch,
> HDFSPluggableAuthorizationProposal-v2.pdf,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v4.patch New patch (v4) based on Selvamohan's & Jitendra's feebdback: * created an interface to only expose getters for authz info from INode * retrofitted INode classes to this interface * created a check in the authz interface that does not take an FSDirectory * did some renaming on proposed new classes/interfaces > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFS-6826v4.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1412#comment-1412 ] Alejandro Abdelnur commented on HDFS-6826: -- [~apurtell], cell level is out of scope from this proposal. This proposal focuses on providing 'synchronized' authorization between data entities and the associated files for the use cases where the files fully belong to a single data entity. If a file contains data for multiple data entities (Hbase cell, columns of a CSV file mapped to a HiveMetaStore table), it is not possible to map authorization to a file in a secure way (enforced by HDFS; you could enforce that a client lib level, but a modified client lib will give you access to the whole file). My take is that, in the case of authorization at cell level, this will always remain in HBase. Otherwise, we would require an authorization source with the scalability of HBase and with more performance than HBase. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099469#comment-14099469 ] Alejandro Abdelnur commented on HDFS-6826: -- [~sneethiraj], got it, it makes sense. Next week I would start working on the v2 patch to get it in proper shape. As a first cut I would prefer to make things pluggable without altering APIs. we can work on refining the APIs once we have the desired functionality. Also, something to keep in mind, this plugin API is meant to be used by somebody with very good understanding of the NameNode guts and expected behavior. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826v3.patch Jitendra, attaching v3 patch, in this patch i've moving the checkpermission logic to the plugin and the FsPemissionChecker does delegate to the plugin. Still the FSDirectory is exposed in the API. Between v2 and v3 I prefer v2. Still I would argue we shouldn't allow replacing the permission checker logic, to ensure consistent check behavior. I don't have a use case for having a different permission check logic, do you? If nothing in sight at the moment, then we can table that till is needed. Thoughts? > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFS-6826v3.patch, HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14098711#comment-14098711
]
Alejandro Abdelnur commented on HDFS-6776:
--
{{NullTokenMsgHeader}} constant name should be all capitals.
I'm not sure I like looking for a string occurrence in the IOException message
to detect the issue. I thought WebHdfs was recreating exceptions on the client
side but it doesn't seem the case for these DT calls.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Nati
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFSPluggableAuthorizationProposal-v2.pdf updated proposal removing the refresh() method and adding the createPermissionChecker() method to the plugin interface. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFSPluggableAuthorizationProposal-v2.pdf, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14097944#comment-14097944 ] Alejandro Abdelnur edited comment on HDFS-6826 at 8/15/14 5:06 AM: --- Attached is new POC where the FsPermissionChecker has been made an interface, the original one renamed to DefaultFsPermissionChecker and the plugin creates a permission checker instance. (the patch is bigger because of the rename, when committing an svn move would be done to preserve history of the permission checker) Then the plugin can do both, replace authz info and replace the permission check logic. I've also remove the refresh() call from the plugin. this means that the plugin does not provide for means to make external call during a filesystem operation. A proper plugin impl should fetch all authz info async from fs operations. was (Author: tucu00): attach is new POC where the FsPermissionChecker has been made an interface, the original one renamed to DefaultFsPermissionChecker and the plugin creates a permission checker instance. (the patch is bigger because of the rename, when committing an svn move would be done to preserve history of the permission checker) then the plugin can do both, replace authz info and replace the permission check logic. I've also remove the refresh() call from the plugin. this means that the plugin does not provide for means to make external call during a filesystem operation. A proper plugin impl should fetch all authz info async from fs operations. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826-idea2.patch attach is new POC where the FsPermissionChecker has been made an interface, the original one renamed to DefaultFsPermissionChecker and the plugin creates a permission checker instance. (the patch is bigger because of the rename, when committing an svn move would be done to preserve history of the permission checker) then the plugin can do both, replace authz info and replace the permission check logic. I've also remove the refresh() call from the plugin. this means that the plugin does not provide for means to make external call during a filesystem operation. A proper plugin impl should fetch all authz info async from fs operations. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14097896#comment-14097896 ] Alejandro Abdelnur commented on HDFS-6826: -- Maybe a way of doing that would be to make an interface for the PermissionChecker methods and the the plugin to return an impl of it. I'll update the patch with those changes to see what people thinks. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14097666#comment-14097666
]
Alejandro Abdelnur commented on HDFS-6826:
--
[~sneethiraj],
The current proposal (and POC code) only externalizes the source of truth for
authorization information (user/group/permission/ACLs), it does not allow
changing the behavior of checking permissions. IMO, doing this is safer than
allowing a plugin to externalize the authorization assertion logic (which is
not simple) and being exposed to unexpected behavior. In order words, with the
current approach, the plugin only allows to change the data used to assert
authorization, not how the authorization is asserted.
Regarding exposing the INode, good point, we should create an interface with
methods the plugin should see (INode woudl implement this new interface).
Something like:
{code}
public interface INodeAuthorizationInfo {
public String getFullPath();
public void setUser(String user);
public String getUser(int snapshot);
public void setGroup(String group);
public String getGroup(int snapshot);
public void setPermission(long permission);
public long getPermission(int snapshot);
public void setAcls(List acls);
public List getAcls(int snaphot);
}
Also, keep in mind that the plugin is not only used for authorization
assertions, it also has to be used for producing the right
authorization/ownership info back to the user via methods like getFileStatus()
and getAcls(). Finally, a plugin could chose to implement changing authz info
(user/group/permissions/acls) via the HDFS FS API (thing that is possible with
the attached POC).
> Plugin interface to enable delegation of HDFS authorization assertions
> --
>
> Key: HDFS-6826
> URL: https://issues.apache.org/jira/browse/HDFS-6826
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFS-6826-idea.patch,
> HDFSPluggableAuthorizationProposal.pdf
>
>
> When Hbase data, HiveMetaStore data or Search data is accessed via services
> (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
> permissions on corresponding entities (databases, tables, views, columns,
> search collections, documents). It is desirable, when the data is accessed
> directly by users accessing the underlying data files (i.e. from a MapReduce
> job), that the permission of the data files map to the permissions of the
> corresponding data entity (i.e. table, column family or search collection).
> To enable this we need to have the necessary hooks in place in the NameNode
> to delegate authorization to an external system that can map HDFS
> files/directories to data entities and resolve their permissions based on the
> data entities permissions.
> I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096629#comment-14096629 ] Alejandro Abdelnur commented on HDFS-6134: -- on create() the NN creates the INode for the file and sets the IV and the EDEK as xAttrs, then it returns them as part of the create response. A single client to NN RPC is done for the create(). If the client does is responsible for creating the new EDEK, then you need 2 RPCs on create(), the first one to 'create' the file, the second one is (now that you know that your file is to be encrypted because of the first RPC call response) to set the EDEK in the INode. On the current behavior, keep in mind the call to the KMS to get EDEK is not done during file create(), the NN has a warm cache of EDEKs that are being replenished asynchronously from file create() calls. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096590#comment-14096590 ] Alejandro Abdelnur commented on HDFS-6134: -- Sanjay, HttpFS is a service that requires to be configured as proxyuser in HDFS. Different from the 'hdfs' user, the 'httpfs' user do not have blanket access to all HDFS files, only to the files of users that can proxy-user as and with HDFS permissions being enforced. Also, the 'httpfs' user does not have access to all encrypted files, which the 'hdfs' user does. The same holds for Oozie, Templeton, HiveServer2, Knox and any other service that needs proxyuser config in HDFS. Regarding returning encrypted data back to the HTTP client. Well, that would mean that you cannot simply use tools/libraries like curl/libcurl to integrate via the WebHDFS protocol anymore. You'll need a client library that interacts with KMS to decrypt the encrypted key and use libopenssl to decrypt. And if you are accessing file ranges, you'll have to know how to manipulate the IV. IMO, going this path, completely defeats the motivation out of which WebHDFS came to be. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096214#comment-14096214 ] Alejandro Abdelnur commented on HDFS-6134: -- if httpfs and NN or DNs run in the same box, yes. however, in a prod environment that would not commonly be the case. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6849) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HDFS-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6849: - Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk and branch-2. > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HDFS-6849 > URL: https://issues.apache.org/jira/browse/HDFS-6849 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.6.0 > > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HDFS-6849) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HDFS-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur moved HADOOP-10836 to HDFS-6849: --- Component/s: (was: security) security Target Version/s: (was: 2.6.0) Affects Version/s: (was: 2.4.1) 2.4.1 Key: HDFS-6849 (was: HADOOP-10836) Project: Hadoop HDFS (was: Hadoop Common) > Replace HttpFS custom proxyuser handling with common implementation > --- > > Key: HDFS-6849 > URL: https://issues.apache.org/jira/browse/HDFS-6849 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, > HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch > > > Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095930#comment-14095930 ] Alejandro Abdelnur commented on HDFS-6826: -- [~daryn], bq. The group mapping authz is a bit different. It's not context sensitive, as in a user uniformly belongs to groups across the whole namesystem. Mmmhh, I’d argue that it is context sensitive, 'user' context, just a different context. bq. Path-based context sensitivity is adding hidden magic to a filesystem. How will the special magic be represented to the user confused by why the perms/ACLs aren't being honored? The authorization enforcement semantics does not change at all. The plugin cannot change the permission check logic. The plugin is responsible for providing user/group/permissions/ACLs information to the NN who enforces the permissions consistently regardless of the plugin in use. bq. How will permission apis and FsShell interact with the magic? The work as usual. Check the attached patch, the current HDFS user/group/permission/ACLs handling is done by a plugin implementation. Said that, a plugin implementation may decide to disable changes of user/group/permissions/ACLs. This can be done either silently or failing. bq. Instead of trying to hack special behavior for a specific use case into the NN, how about leveraging what's there. The proposal doc describes in detail 3 different usecases: HiveMetaStore tables, Hbase tables, Solr search collections. bq. A cleaner way may be for a custom group mapping to fabricate groups something like "hive:table" or "hive:table:column". No code changes in the NN. Everything is contained in the custom groups mapping. This does not solve the problem. When adding a directory as a HiveMetaStore table partition, unless you set those special groups explicitly, they would not be in the files being added to the table. It requires client side group manipulation and this is what breaks things. bq. I still think leveraging ACLs is the best way to go... Actually, we are. In the case of HiveMetaStore, the plugin would expose GRANT permissions as ACLs. Daryn, I'm happy to jump on the phone if you want have a synchronous discussion. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095730#comment-14095730 ] Alejandro Abdelnur commented on HDFS-6134: -- Larry, if the httpfs admin is a different person than the hdfs admin you don't have the problem. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095672#comment-14095672 ] Alejandro Abdelnur commented on HDFS-6826: -- [~clamb], that seems correct. We make sure this is not an issue under normal circumstances by implementing caching. The same would hold for any plugin implementation meant for production usage. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095668#comment-14095668 ] Alejandro Abdelnur commented on HDFS-6134: -- Let me try to explain things a different way. When setting up filesystem encryption in HDFS (forget about webhdfs and httpfs for now), things will be configured so the HDFS superuser cannot retrieve decrypted 'file encryption keys'. Because the HDFS superuser has access to the encrypted versions of the files, having access to the decrypted 'file encryption keys' would allow the HDFS superuser to get access to the decrypted file. One of the goals of HDFS encryption is to prevent that. This is achieved by blacklisting the HDFS superuser from retrieving decrypted 'file encryption keys' from the KMS. This blacklist is must be enforced on the real UGI hitting the KMS (regardless if it is doing a doAs or not). If you set up httpfs, it runs using the 'httpfs' user, a HDFS regular user configured as proxyuser to interact with HDFS and KMS doing doAs calls. If you set up webhdfs, it runs using the 'hdfs' user, the HDFS superuser, and this user will have to be configured as proxyuser in the KMS to work with doAs calls. Also the 'hdfs' user will have to be removed from the KMS decrypt-keys blacklist (*and this is the problem*). Even if you audit the webhdfs code running in the DNs to ensure things are always done using doAs and that there is no foul play in the DN code there is an issue. The issue is: * An HDFS admin logins to a DN in the cluster as 'hdfs' * Then he kinits as 'hdsf/HOST' * Then he curls the KMS asking to decrypted keys as user X doing a doAs * Because he has access to the encrypted file, and now has the decrypted key, gets access to the file in clear hope this clarifies. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094683#comment-14094683 ] Alejandro Abdelnur commented on HDFS-6134: -- httpfs accesses files using doAs() today, I would assume webhdfs does the same. Still in the case of webhdfs, is the hdfs user authenticating against the KMS. if the 'hdfs' user can be a KMS proxyuser, then an 'hdfs' admin can retrieve file encryption keys and gain access to decrypted content of encrypted files. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094587#comment-14094587 ] Alejandro Abdelnur commented on HDFS-6134: -- bq. Also I do not understand why httpfs works and webhdfs "breaks". Neither will be running as the end-user and hence neither will allow transparent encryption. Am I missing something? Both httpfs and webhdfs will work just fine. when reading/writing a file, webhdfs (DN) and httpfs (httpfs) will need to get the file encryption key in decrypted form. httpfs runs as 'httpfs' user, webhdfs runs as 'hdfs' user (embedded in the NN/DNs). Typically KMS would be configured not to decrypt keys for the 'hdfs' user (one of the goals is that the hdfs user should not have access to the keys so it cannot decrypt files). For webhdfs to work, the 'hdfs' user must not be blacklisted in the KMS, thus the 'hdfs' user has access to the decrypted keys for files. The point is, if webhdfs is enabled, then KMS has to be configured in a way that the 'hdfs' user can access all files in encrytped form. And this could fail some security audits users may have to do in their clusters. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6842) TestHttpFSFWithWebhdfsFileSystem fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094252#comment-14094252
]
Alejandro Abdelnur commented on HDFS-6842:
--
Ted, this is happening because of HADOOP-10835 refactoring changes,
HADOOP-10836 fixes it.
Mind checking with HADOOP-10836, I've tested locally and that fixes the
problem.
> TestHttpFSFWithWebhdfsFileSystem fails in trunk
> ---
>
> Key: HDFS-6842
> URL: https://issues.apache.org/jira/browse/HDFS-6842
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>
> This can be reproduced locally:
> {code}
> testOperationDoAs[21](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem)
> Time elapsed: 0.315 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: User: zy is not allowed to impersonate
> user1
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy24.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:512)
> at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy25.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2546)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2517)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:821)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094239#comment-14094239 ] Alejandro Abdelnur commented on HDFS-6826: -- [~daryn], A proxy NN was one of the alternatives considered before settling for the Authz plugin. I like it because of the very same reasons you point out. The concerns with that approach are: * It is a new service * It has to support HA * It will be error prone for users the use of 2 NN URIs: ** One for HDFS authz, another for Hive authz ** A file or directory being added to a Hive table would first be referenced via the NN and then via the proxy NN ** Users my add file URIs of the proxy NN in the HiveMetaStore Regarding the plugin approach. If the plugin does not interact with external systems during a filesystem RPC call and just uses inmemory data, that addresses your concerns, right? > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094054#comment-14094054 ] Alejandro Abdelnur commented on HDFS-6134: -- [~szetszwo], just a regular HttpFS setup. HttpFS runs a user 'httpfs' which is not an HDFS superuser, just a HDFS proxyuser. Because of this, the 'hfds' user does not have access to decrypted keys, which is the case with WebHDFS running out of NN/DNs. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093249#comment-14093249 ] Alejandro Abdelnur commented on HDFS-6826: -- [~daryn], looking forward to your alternate approach. BTW, the plugin could be implemented in a way that the whole 'external' authorization data is fetched in advance, or that the external call has a maxCallTime and timesout returning default fallback strict permissions. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6826: - Attachment: HDFS-6826-idea.patch [~daryn], A custom plugin, would have a list of region prefixes that are subject to 'external' permissions, any path not matching these prefixes would go straight to the default plugin. Only path’s matching the region prefixes would b subject to an 'external' permissions check. Attached is an initial prototype, with a basic testcase using a custom plugin showing the proposed solution. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-6826-idea.patch, > HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091433#comment-14091433 ] Alejandro Abdelnur commented on HDFS-6826: -- [~daryn], I believe the uploaded proposal will answer you questions. > Plugin interface to enable delegation of HDFS authorization assertions > -- > > Key: HDFS-6826 > URL: https://issues.apache.org/jira/browse/HDFS-6826 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.4.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSPluggableAuthorizationProposal.pdf > > > When Hbase data, HiveMetaStore data or Search data is accessed via services > (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce > permissions on corresponding entities (databases, tables, views, columns, > search collections, documents). It is desirable, when the data is accessed > directly by users accessing the underlying data files (i.e. from a MapReduce > job), that the permission of the data files map to the permissions of the > corresponding data entity (i.e. table, column family or search collection). > To enable this we need to have the necessary hooks in place in the NameNode > to delegate authorization to an external system that can map HDFS > files/directories to data entities and resolve their permissions based on the > data entities permissions. > I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
