[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241459#comment-14241459 ] mohamed Abo el haggag hassany commented on HADOOP-10433: sorry guys can any one give me a document describe the process of the authentication Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.6.0 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992794#comment-13992794 ] Hudson commented on HADOOP-10433: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1751 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/]) HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637) * /hadoop/common/trunk/.gitignore * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory * /hadoop/common/trunk/hadoop-common-project/hadoop-kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF *
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992830#comment-13992830 ] Hudson commented on HADOOP-10433: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/]) HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637) * /hadoop/common/trunk/.gitignore * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory * /hadoop/common/trunk/hadoop-common-project/hadoop-kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF *
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990032#comment-13990032 ] Larry McCay commented on HADOOP-10433: -- Thanks for all the heavy lifting here, [~tucu00]! Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 3.0.0 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990229#comment-13990229 ] Hudson commented on HADOOP-10433: - SUCCESS: Integrated in Hadoop-trunk-Commit #5593 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5593/]) HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637) * /hadoop/common/trunk/.gitignore * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory * /hadoop/common/trunk/hadoop-common-project/hadoop-kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT * /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF *
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989086#comment-13989086 ] Alejandro Abdelnur commented on HADOOP-10433: - planning to commit on monday Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987966#comment-13987966 ] Alejandro Abdelnur commented on HADOOP-10433: - Test failure seems unrelated. Owen, Andrew, are we good to go now? Vinay, a sample of the audit log follows. Regarding normalizing audit logs, that would be nice, mind opening a JIRA for it (it is out of scope from this JIRA). {code} 2014-05-02 10:14:12,326 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:12,994 Status:OK User:cli...@example.com Op:CREATE_KEY Name:ck0UserProvidedMaterial:false Description:null 2014-05-02 10:14:13,027 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,076 Status:OK User:client/h...@example.com Op:CREATE_KEY Name:ck1UserProvidedMaterial:false Description:null 2014-05-02 10:14:13,557 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,597 Status:UNAUTHORIZED User:cli...@example.com Op:CREATE_KEY Name:k 2014-05-02 10:14:13,597 Status:ERROR User:cli...@example.com Method:POST URL:http://localhost:65509/kms/v1/keys Exception:'User:cli...@example.com not allowed to do 'CREATE_KEY' on 'k'' 2014-05-02 10:14:13,606 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,645 Status:UNAUTHORIZED User:cli...@example.com Op:CREATE_KEY Name:k 2014-05-02 10:14:13,646 Status:ERROR User:cli...@example.com Method:POST URL:http://localhost:65509/kms/v1/keys Exception:'User:cli...@example.com not allowed to do 'CREATE_KEY' on 'k'' 2014-05-02 10:14:13,653 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,692 Status:UNAUTHORIZED User:cli...@example.com Op:ROLL_NEW_VERSION Name:k 2014-05-02 10:14:13,692 Status:ERROR User:cli...@example.com Method:POST URL:http://localhost:65509/kms/v1/key/k Exception:'User:cli...@example.com not allowed to do 'ROLL_NEW_VERSION' on 'k'' 2014-05-02 10:14:13,699 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,739 Status:UNAUTHORIZED User:cli...@example.com Op:ROLL_NEW_VERSION Name:k 2014-05-02 10:14:13,739 Status:ERROR User:cli...@example.com Method:POST URL:http://localhost:65509/kms/v1/key/k Exception:'User:cli...@example.com not allowed to do 'ROLL_NEW_VERSION' on 'k'' 2014-05-02 10:14:13,746 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/names?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,781 Status:UNAUTHORIZED User:cli...@example.com Op:GET_KEYS Name:* 2014-05-02 10:14:13,781 Status:ERROR User:cli...@example.com Method:GET URL:http://localhost:65509/kms/v1/keys/names Exception:'User:cli...@example.com not allowed to do 'GET_KEYS' on '*'' 2014-05-02 10:14:13,791 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/metadata?key=kuser.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,825 Status:UNAUTHORIZED User:cli...@example.com Op:GET_KEYS_METADATA Name:k 2014-05-02 10:14:13,826 Status:ERROR User:cli...@example.com Method:GET URL:http://localhost:65509/kms/v1/keys/metadata?key=k Exception:'User:cli...@example.com not allowed to do 'GET_KEYS_METADATA' on 'k'' 2014-05-02 10:14:13,836 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/keyversion/k%400?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,871 Status:UNAUTHORIZED User:cli...@example.com Op:GET_KEY_VERSION Name:k@0 2014-05-02 10:14:13,871 Status:ERROR User:cli...@example.com Method:GET URL:http://localhost:65509/kms/v1/keyversion/k%400 Exception:'User:cli...@example.com not allowed to do 'GET_KEY_VERSION' on 'k@0'' 2014-05-02 10:14:13,878 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k/_currentversion?user.name=tucu ErrorMsg:'Authentication required' 2014-05-02 10:14:13,912 Status:UNAUTHORIZED User:cli...@example.com Op:GET_CURRENT_KEY Name:k 2014-05-02 10:14:13,912 Status:ERROR User:cli...@example.com Method:GET URL:http://localhost:65509/kms/v1/key/k/_currentversion Exception:'User:cli...@example.com not allowed to do 'GET_CURRENT_KEY' on 'k'' 2014-05-02 10:14:13,919 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 Method:OPTIONS
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987985#comment-13987985 ] Vinay Shukla commented on HADOOP-10433: --- Thanks for the sample Audit record. I have created a JIRA for normalizing audit records. https://issues.apache.org/jira/browse/HADOOP-10569 Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987987#comment-13987987 ] Andrew Wang commented on HADOOP-10433: -- LGTM, +1 again Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988457#comment-13988457 ] Alejandro Abdelnur commented on HADOOP-10433: - Owen? are we good as the initial commit, and we follow up with other JIRAs improvements? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986936#comment-13986936 ] Owen O'Malley commented on HADOOP-10433: Alejandro, A couple more things on the interface: * You need to allow the user to specify the key-name when they create the key. * Instead of throwing if the url is too long, please break the user's request into parts that fit within the 2000 byte limit. * You never answered whether the body is xml or json (or both). * The underscores should be on the trailing part of the url. I'd propose it look like: {code} create key : PUThttp://HOST:PORT/kms/v1/key/key-name rollover key : POST http://HOST:PORT/kms/v1/key/key-name delete key : DELETE http://HOST:PORT/kms/v1/key/key-name get metadata : GEThttp://HOST:PORT/kms/v1/key/key-name/_metadata get current version: GET http://HOST:PORT/kms/v1/key/key-name/_currentversion get key versions : GEThttp://HOST:PORT/kms/v1/key/keyname-name/_versions get key version: GEThttp://HOST:PORT/kms/v1/keyversion/version-name get key names : GEThttp://HOST:PORT/kms/v1/keys/names get keys metadata : GET http://HOST:PORT/kms/v1/keys/metadata?key=key-namekey=keyname,... {code} Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987166#comment-13987166 ] Alejandro Abdelnur commented on HADOOP-10433: - Owen, bq. You need to allow the user to specify the key-name when they create the key. It is happening, it goes in the JSON payload of the create operation. Look in the {{createKeyInternal()}} method bq. You never answered whether the body is xml or json (or both). Request/response bodies are JSON. I'll integrate your other 2 suggestions, the underscore and the break into multiple requests if URL goes beyond 2K byte limit. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987185#comment-13987185 ] Vinay Shukla commented on HADOOP-10433: --- [~tucu00] What is the audit story about the Key? Do we record, who did various key operations? I couldn't find it in the attached PDFs. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987191#comment-13987191 ] Alejandro Abdelnur commented on HADOOP-10433: - Vinay, there is a kms-audit.log file that logs all requests, both successful and failed. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987339#comment-13987339 ] Vinay Shukla commented on HADOOP-10433: --- Is there a sample of the audit log record? What field are audited? Would it be useful to have a common audit format across Hadoop? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985782#comment-13985782 ] Andrew Wang commented on HADOOP-10433: -- +1 from me, thanks Tucu. Really nice work here. Let's wait a bit before committing to see if anyone else still has review comments. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986019#comment-13986019 ] Alejandro Abdelnur commented on HADOOP-10433: - sure, i'll wait till Friday. Thanks again. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984101#comment-13984101 ] Andrew Wang commented on HADOOP-10433: -- Thanks for the rev here Tucu, I can tell this wasn't a small effort. Overall it looks real nice, the new REST API is fine by me. I have some more review comments though, but expect to +1 after this round (pending agreement by reviewers on the REST API details). Running the KMS: * Overall a more pleasant experience, tarball builds, audit logging has newlines, I didn't see stacktraces when running commands. * I know I asked for less spew in kms.sh, but defaulting KMS_SILENT to true is perhaps too little. Without it, running just ./sbin/kms.sh shows nothing, not even help text, and if start/stop hit an error, we again see nothing (have to know where the logs are and hope it has the error there). This isn't a biggie, but maybe we can improve it a little more. JSON: * I see KMSMetadata and KMSKeyVersion classes, but they don't look like they do much. I'm guessing this is from an aborted attempt to do POJO-JSON? That'd would be awesome, but I won't get too excited. * If you stick with the JSON helper methods, could we pull them all out into a separate helper class and make them all static, along with the field constants? Feels weird to have the Server digging into ClientProvider for field names, when theoretically a different client implementation (REST!) could be interfacing with the KMS. KMSServer and KMSClientProvider: * Noticed that KMSServer#getKeyMetadata includes the name, but it's unused on the client side, nor part of KeyProvider#Metadata. * I think we still miss audit logging if an exception is thrown. e.g. in #rolloverKey, getPrincipal, checkNotEmpty, provider.rollNewVersion can all throw something. This is why a try/finally might be easiest. * Can make many methods static: validateResponse, call, all the JSON functions, getPrincipal, assertAccess, removeKeyMaterial, getKeyURI * On the NoSuchAlgorithmException, #validateResponse casts everything to an IOException, so I don't think anything else comes out of this function. Not sure this cast is kosher. I think the catch for Exception there also needs to wrap the created String into an exception and throw it. * Given the discussion about UUIDs as versions, should we be exposing KMSClientProvider#buildVersionName publicly? This seems like an implementation detail of the backing KeyProvider. Misc things: * I think a rogue regex changed GET to GET_OP across the entire hadoop repo, which is why we're picking up all these test changes, and quite possibly why we have some weird test failures. * KMSACLs, the new comment about in #run probably should go in #loadACLs, or be removed. * KeyNotFoundException serialVersionUID, this is something Eclipse is warning about, so it'd be nice to just add it. * In the cache provider, #getCurrentKey and getKeyVersion, could set the cause once in a temp var to avoid having to call getCause() repeatedly. * KMSServer semantically should still be renamed to KMServer or just KMS Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984138#comment-13984138 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642395/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984300#comment-13984300 ] Larry McCay commented on HADOOP-10433: -- Hey [~tucu00] - I was wondering about your It may barbaric, but that is how things are and we should make sure we can work with those. I think you may have misinterpreted my reference the the OpenStack Barbican project - which is a REST based KMS server. I don't think UUIDs are barbaric. :) Thank's for the response. I need to give the rest of your explanation a bit more thought. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984497#comment-13984497 ] Owen O'Malley commented on HADOOP-10433: Alejandro, I like the direction your api is going. I'm a little concerned about the length of the URL (2000 bytes) causing trouble for the keys/metadata method. I'd suggest adding a field in the header that provides the list of key names. I'd also suggest that you add a character to the fields (metadata, currentversion, versions) such as adding underscore (_metadata, _currentversion, _versions). Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984534#comment-13984534 ] Larry McCay commented on HADOOP-10433: -- I can see how having a .../keyversion/version-name is aligned with the KeyProvider API and that it allows the version-name to be more easily considered opaque. I would like to point out that I think we need to have the key-name for the create key API - you generally fully identify the resource that you are creating (PUTting). Are you assuming that it is part of the body of the request? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984651#comment-13984651 ] Larry McCay commented on HADOOP-10433: -- Yes, that aligns with collections based actions - although they are supposed to be a POST for collection actions that create a new element. A PUT is supposed to replace the entire collection with another collection - which is NOT what you want. Element based actions on the other hand are done with a PUT and replace the element or create it if it does not exist. So, it seems that create should either be made a POST or an element based action. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984660#comment-13984660 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642504/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984662#comment-13984662 ] Alejandro Abdelnur commented on HADOOP-10433: - Larry, good point, i'll update the patch making it a POST. and I'll take care of the findbugs Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984750#comment-13984750 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642512/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984908#comment-13984908 ] Vinay Shukla commented on HADOOP-10433: --- Does KMS support KMIP protocol ? (https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip) Will KMS integrate with Hardware Security Modules (HSM) devices such as SafeNet Luna RSA's Data Protection Manager (http://www.emc.com/security/rsa-data-protection-manager.htm)? If KMS does not speak KMIP, how can Hadoop encryption leverage enterprise grade Key management investment many enterprise level customers already have? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984917#comment-13984917 ] Larry McCay commented on HADOOP-10433: -- KMS leverages KeyProvider API a kmip provider would be the integration vehicle. KMS would pick it up for free. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984918#comment-13984918 ] Alejandro Abdelnur commented on HADOOP-10433: - Vinay, KMS uses a KeyProvider as the underlying keystore. A KeyProvider impl supporting RSA/SafeNet can be plugged in. AFAIK, both RSA and SafeNet have JCE providers with keystore support, thus getting the JavaKeyProvider to work with them shouldn't be that difficult (I would assume). Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985076#comment-13985076 ] Hadoop QA commented on HADOOP-10433: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642548/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983285#comment-13983285 ] Charles Lamb commented on HADOOP-10433: --- One little nit in the REST API: for the get operations, rather than use ?get=..., perhaps consider calling it op instead. For example, GET .../key/key-name?op=getCurrentVersion and GET .../keys?op=getKeyNames Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983299#comment-13983299 ] Alejandro Abdelnur commented on HADOOP-10433: - Charles, based on Andrew's feedback I've moved away from operations. get indicates what part of the resource we want to retrieve, either get= or show= I would say, op= is a bit confusing as it is not an operation. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983328#comment-13983328 ] Charles Lamb commented on HADOOP-10433: --- I guess what I was responding to was having keynames embedded in the URI rather than as a parameter after the ?. For instance, the get key version operation uses /key-name?get=version but the get keys metadata operation puts the keyname as a parameter value: /keys?get=keyMetaDatakey=key-name I like the second form where the key-name is a value rather than an operator so perhaps this can be unified? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983348#comment-13983348 ] Alejandro Abdelnur commented on HADOOP-10433: - well, typically REST resources are expressed as a PATH more than as QUERY_STRING, I'd prefer to keep it that way. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983355#comment-13983355 ] Larry McCay commented on HADOOP-10433: -- Personally, I feel that the keyname as part of the URI adheres to accepted REST principles. In fact, I'd like to understand why the parameters are needed at all. One should be able to represent the desired resource entirely by URI. There are times when non-hierarchical attributes need to be considered (as filters) that are easily added to a query string. But largely, the APIs we are looking at here are hierarchical. GET .../keys - would return all the keyNames GET .../keys/keyname - would return the metadata for that key (or the current key maybe) GET .../keys/keyname/version - would return the specific key (could have a well-known current version as well) Just my two cents. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983448#comment-13983448 ] Larry McCay commented on HADOOP-10433: -- I do have to say that I don't like the use of get as a query parameter. It is semantically redundant with the HTTP verb in a well designed REST API. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983478#comment-13983478 ] Alejandro Abdelnur commented on HADOOP-10433: - ok, let me work on a revised version using path to refer sub-resources metadata/version/versions Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983504#comment-13983504 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642256/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983702#comment-13983702 ] Charles Lamb commented on HADOOP-10433: --- I like this better. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983719#comment-13983719 ] Larry McCay commented on HADOOP-10433: -- I like this much better as well. To me, you could continue the hierarchy of the key/keyversion as well. Additionally, I'm not sure that you need to identify the path elements with key/keyname/version/version-name. Each resource's path is unique unto itself and I think it holds together to just use the values that way - as you follow the path the URI more specifically identifies the resource. With that in mind, I would propose adjustments along the following lines: {code} create key : PUThttp://HOST:PORT/kms/v1/keys rollover key : POST http://HOST:PORT/kms/v1/keys/key-name delete key : DELETE http://HOST:PORT/kms/v1/keys/key-name get metadata : GEThttp://HOST:PORT/kms/v1/keys/key-name/metadata get current version: GET http://HOST:PORT/kms/v1/keys/key-name/currentversion get key version: GET http://HOST:PORT/kms/v1/keys/key-name/version-name get key versions : GEThttp://HOST:PORT/kms/v1/keys/version-name/versions get key names : GEThttp://HOST:PORT/kms/v1/keys get keys metadata : GET http://HOST:PORT/kms/v1/keys/metadata?key=key-namekey=keyname,... {code} This flattens the hierarchy a good bit the only outlier being the get .../keys/metadata - I think that the rest of them identify unique resources within a hierarchy. [~tucu00] I'm not sure that I am grasping your concern with the 'get key version' URI and doubt that this minor adjustments changes your feelings there. Can you elaborate on what the issue is? Maybe provide an example? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983720#comment-13983720 ] Charles Lamb commented on HADOOP-10433: --- Nit: there are still some instances of chiper in the design doc. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983728#comment-13983728 ] Andrew Wang commented on HADOOP-10433: -- Larry, I think an issue with that scheme is that we can't have a key with the name currentversion or metadata then, or a key and a version with the same name. We need additional path elements to disambiguate. As to the get key version API, it's think it's an unfortunate consequence of the existing KeyProvider#getKeyVersion API, which takes a single string. Tucu's concern there is that if a KP decides to use a UUID instead of keyName@version like JavaKeyStoreProvider, we can't fit it into the hierarchical key-then-version scheme. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983753#comment-13983753 ] Larry McCay commented on HADOOP-10433: -- hmmm... Seems like this one won't work that way with the rest: {code} get key versions : GEThttp://HOST:PORT/kms/v1/keys/version-name/versions {code} I'm not even really sure I know what it is trying to depict. Perhaps it is supposed to be like?: {code} get key versions : GEThttp://HOST:PORT/kms/v1/keys/key-name/versions {code} I still don't quite like the last one with .../keys/metadata as it overlaps with a key named metadata in the hierarchy. I'm not sure what to do with that one. Maybe the following would work - obviously you aren't asking for the names since you are providing them: {code} get keys metadata : GET http://HOST:PORT/kms/v1/keys?key=key-namekey=keyname,... {code} Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983763#comment-13983763 ] Larry McCay commented on HADOOP-10433: -- Hi [~andrew.wang] - I also thought that those types of providers would need to be adapted by the KeyProvider implementation to always default the version - assuming that the provider doesn't actually support versions. So, in that case it would be be UUID@0. Current version would always be 0 or 1 - whatever makes sense. There are other systems that seem to have keyNames with UUIDs as the version. I think that barbican uses this scheme. It would have to do something similar within the KeyProvider implementation to adapt it to the API. Not sure whether the UUID version can be used as the actual version. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983783#comment-13983783 ] Larry McCay commented on HADOOP-10433: -- AFAICT - we don't have a problem for a key named currentversion or metadata. Currentversion would have special meaning within the versions context and metadata wouldn't make sense as a version name anyway. I don't see why a key and version name couldn't be the same either - though I think versions are generally numeric. So, key 1234 could have 1234 versions in which case 1234@1234 would be reasonable and not precluded by the URI's proposed. {code} get key version: GEThttp://HOST:PORT/kms/v1/keys/1234/1234 {code} Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983944#comment-13983944 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642355/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982273#comment-13982273 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.crypto.key.kms.server.TestKMSServer org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982282#comment-13982282 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.ha.TestZKFailoverControllerStress org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981935#comment-13981935 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642061/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982068#comment-13982068 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642080/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.lib.db.TestDBJob org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-openstack hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.pipes.TestPipeApplication {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981822#comment-13981822 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642042/HadoopKMSDocsv2.pdf against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3853//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978967#comment-13978967 ] Owen O'Malley commented on HADOOP-10433: What is the form of the urls for the REST service? What are the acceptable formats of the data in the REST service (JSON? XML?)? Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978969#comment-13978969 ] Owen O'Malley commented on HADOOP-10433: Please don't use the Hadoop ACLs. They are very error-prone with x meaning user x and x meaning group x. I'd suggest using : as the separator and making it ignore spaces. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977803#comment-13977803 ] Alejandro Abdelnur commented on HADOOP-10433: - Andrew, thanks for the detailed review. I'm working on a patch addressing most of your feedback, following some questions/clarifications: * KeyNotFoundException needing serialVersionUID, why do we need this? we are not serializing these exceptions. * The big log entries happen only on the first REST call only, a Jersey initialization thing (fixed in newer Jerseys) * Regarding using curl directly, the REST API is not meant to be public at the moment, that is why it is not documented. * Doing POJO-JSON automatically requires the beans to have a public default constructor and public getters/setters. Given the discussion in HADOOP-10431, this is not an option. * KMSClientProvider constants going private, all of them are used in the KMSServer as well. * KMSClientProvider.rollNewVersion() throwing a NoSuchAlgorithmException, yes it can, it may be come as part of the remote exception in the error message. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972199#comment-13972199 ] Andrew Wang commented on HADOOP-10433: -- Thanks for the huge amount of work here Tucu. I'm still fresh to this topic, so please forgive any silly review comments. * In the docs, Hadoop KMS Server is like saying ATM machine :) docs could use a pass, but we can do that after the base patch is committed. * For the same reason, you could rename KMSServer to just KMS or KMServer * More javadoc, at the minimum class javadoc for everything * KeyNotFoundException needs a serialVersionUID * pom.xml, not sure where these vars are being used, but I'm not aware of Maven's subtleties. {code} kms.source.repositoryREPO NOT AVAIL/kms.source.repository kms.source.revisionREVISION NOT AVAIL/kms.source.revision kms.build.timestamp${maven.build.timestamp}/kms.build.timestamp {code} * mvn package -Pdist -Dtar doesn't work because there's an extra newline in the pom.xml for dist-maketar.sh. Notes from running the KMS: * Running ./sbin/kms.sh has a lot of debugging output, not sure if we can quash it or not. * Is there some way of aborting tomcat if the KMSWebApp dies? I didn't put a core-site.xml in my conf dir initially, KMSWebApp FATAL'd, and I had a useless Tomcat process lying around. * Running ops against the KMS with hadoop key generates big log entries like this: {code} 2014-04-16 16:40:28,160 INFO WadlGeneratorJAXBGrammarGenerator - Couldn't find JAX-B element for class javax.ws.rs.core.Response 2014-04-16 16:40:28,160 ERROR WadlGeneratorJAXBGrammarGenerator - java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers protected stacktrace {code} * Audit logging doesn't have newlines * I didn't test anything outside of KeyShell operations. I couldn't figure out the right curl incantation to hit REST endpoints directly, whenever I tried I got an authentication exception about being an anonymous user. REST API design comments: * It's kind of ugly that we're doing manual serde of things into JSON, since it throws away types and makes the protocol harder to understand. Jackson has ways of automatically doing POJO-JSON for beans, which seems pretty reasonable to me. * If we use Jackson, we should really aspire to JSON in, JSON out instead of using so many query parameters (in particular, create). * I think we should bake the op into the path when possible, since it makes the API more self-descriptive. Maybe something like this: {code} /keys /keys?metadata # this is still ugly, dunno /keys/keyName # GET for metadata, PUT for create, POST for roll, DELETE /keys/keyName/versions # getKeyVersions /keys/keyName/versions/version # getKeyVersion /keys/keyName/versions/current {code} * Also, shouldn't the REST API be explained in the docs? I assume we chose REST for interop with non-Hadoop and non-Java components, so the protocol should be documented. KMSClientProvider: * Class javadoc refers to JksProviders * Typo: CHIPER_FIELD and chiper, I assume cipher * It'd be nice to put all the {{private static final}} before {{public static final}}. Some of the public ones also aren't used outside of this class, so we could close them up. * Constructor, can use equalsIgnoreCase rather than toLowerCase * Wherever we're building a URL, consider using URIBuilder and then toURL() instead. * Based on this page (http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html) we should try/catch when calling HttpURLConnection#getInputStream and I assume #getResponseCode too, and also close the input stream after we're done. The error stream logic in #validateResponse also doesn't quite adhere to the recommendations. * Is there a reason flush is a NOP? The JKSP doesn't do this, so a explanatory comment would help. * It doesn't look like rollNewVersionInternal needs to throw NoSuchAlgorithmException? KMSACLs: * You could write this with a ScheduledExecutorService and a ThreadFactoryBuilder, more idiomatic. * Probably should make start/stopReloader idempotent to future proof it, e.g. check a isRunning boolean. * In loadACLs, I assume you do the conf get to init the conf outside of the lock? Would help to clarify that in the comment. * The lastReload time should be set to the modtime of the file, not by calling System.currentTimeMillis again. Also, what's the point of waiting for it to be at least 100ms newer? * I'd bump the setACLs logs up to INFO, since this probably isn't a frequent operation and is pretty useful. KMSServer: * The hasAccess method is a bit gross. We could make an Access enum rather than passing the ACL as an argument. Could also get rid of the boolean for throwing an exception, and have two helper methods {{checkAccess}}
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966976#comment-13966976 ] Alejandro Abdelnur commented on HADOOP-10433: - [~lmccay] the KMS server is meant to be a proxy to any KeyProvider implementation with the following benefits: * Single entry point from Hadoop to the KeyProvider implementation * Supports Hadoop Kerberos Authentication, making secure integration easier * Provides caching, to support load from Hadoop services and jobs Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966988#comment-13966988 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639842/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings). {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967006#comment-13967006 ] Larry McCay commented on HADOOP-10433: -- [~tucu00] - that sounds like the right direction to me! Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967103#comment-13967103 ] Hadoop QA commented on HADOOP-10433: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639861/HADOOP-10433.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964751#comment-13964751 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639468/COMBO.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1289 javac compiler warnings (more than the trunk's current 1288 warnings). {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist: org.apache.hadoop.crypto.key.TestKeyShell {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958858#comment-13958858 ] Larry McCay commented on HADOOP-10433: -- I don't think that features are an issue at this point. My larger concern is whether Hadoop common needs a KMS server offering of its own or not. Given that the keystore provider is not likely a scalable provider for a KMS that will lead us down the path of requiring an appropriate DB to do it right. This was my thinking as I starting prototyping on an earlier version of the KeyProvider API as well. It became reasonable to me that the KeyProvider API within Hadoop common was sufficient to enable deployments to plugin any number of various KeyProvider implementations. This might be: 1. direct to KMIP 2. to an OpenStack Barbican server 3. to a Knox (or wherever) KMS This would allow Hadoop common to not have to take on the weight of another server. I'm not really arguing against putting it common here but recollecting my thought process on the matter. Here is a question that I have had trouble answering with regard to the server and API being in common together... Given the pluggable nature of the KeyProvider API what value does adding a full KMS server with additional (and maybe redundant) pluggability to common provide? I've had trouble reconciling that in my mind. Now, take that same server implementation and move it out of common and it easily makes sense to be able to have pluggability for the Hadoop KeyProvider API, KMIP and others. So, when I said as long as it make sense to do so, I was really talking about whether it made sense to have it colocated with the KeyProvider API in common. I think the feature set is less of an issue. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958368#comment-13958368 ] Larry McCay commented on HADOOP-10433: -- I have a lot of interest in this project. We have done some prototyping for a REST based key server within Knox. I'd like to hopefully consolidate these into this jira as long as it makes sense to do so. Obviously, there are HA requirements that aren't described here yet that will need some thought and discussion. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958505#comment-13958505 ] Uma Maheswara Rao G commented on HADOOP-10433: -- Hi Alejandro, Thanks a lot, for the doc. Seems like you plan to use Tomcat which will be bundled with Hadoop. Is there any special reason for not using Jetty which we are already using for simple web in Hadoop components? Yes, As pointed by Larry, I would like to see how the HA requirements would be handled. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958541#comment-13958541 ] Alejandro Abdelnur commented on HADOOP-10433: - [~lmccay], if you have additional features in mind we should incorporate them in the KMS, agree. Would make sense to do them as follow up improvements/new-features JIRAs? Regarding HA, the KMS server itself is stateless, you can put multiple behind a VIP and are done (from the KMS server perspective). And the KeyProvider impl you are using must be HA itself (for example, it could be a HA DB or a HA KMIP sever). [~umamaheswararao], regarding Tomcat vs Jetty. With embedded Jetty we will have to create configurations for all sorts of HTTP and server related things (connection threadpool size, http/socket timeout, ssl, port, etc, etc). With Tomcat all that come for free. The same approach is taken by HttpFS (and Oozie). Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955849#comment-13955849 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637934/KMS-ALL-PATCHES-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 7 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1487 javac compiler warnings (more than the trunk's current 1485 warnings). {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955929#comment-13955929 ] Hadoop QA commented on HADOOP-10433: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637950/KMS-ALL-PATCHES-v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 7 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API
[ https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950446#comment-13950446 ] Hadoop QA commented on HADOOP-10433: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637338/KMS-ALL-PATCHES.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 7 new or modified test files. {color:red}-1 javac{color}. The applied patch generated 1504 javac compiler warnings (more than the trunk's current 1491 warnings). {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-dist: org.apache.hadoop.crypto.key.kms.server.TestKMSACLs org.apache.hadoop.crypto.key.kms.server.TestKMSServer {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//console This message is automatically generated. Key Management Server based on KeyProvider API -- Key: HADOOP-10433 URL: https://issues.apache.org/jira/browse/HADOOP-10433 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10433.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf (from HDFS-6134 proposal) Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying KMS. It provides an interface that works with existing Hadoop security components (authenticatication, confidentiality). Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 and HADOOP-10177. Hadoop KMS will provide an additional implementation of the Hadoop KeyProvider class. This implementation will be a client-server implementation. The client-server protocol will be secure: * Kerberos HTTP SPNEGO (authentication) * HTTPS for transport (confidentiality and integrity) * Hadoop ACLs (authorization) The Hadoop KMS implementation will not provide additional ACL to access encrypted files. For sophisticated access control requirements, HDFS ACLs (HDFS-4685) should be used. Basic key administration will be supported by the Hadoop KMS via the, already available, Hadoop KeyShell command line tool There are minor changes that must be done in Hadoop KeyProvider functionality: The KeyProvider contract, and the existing implementations, must be thread-safe KeyProvider API should have an API to generate the key material internally JavaKeyStoreProvider should use, if present, a password provided via configuration KeyProvider Option and Metadata should include a label (for easier cross-referencing) To avoid overloading the underlying KeyProvider implementation, the Hadoop KMS will cache keys using a TTL policy. Scalability and High Availability of the Hadoop KMS can achieved by running multiple instances behind a VIP/Load-Balancer. For High Availability, the underlying KeyProvider implementation used by the Hadoop KMS must be High Available. -- This message was sent by Atlassian JIRA (v6.2#6252)