[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-12-10 Thread mohamed Abo el haggag hassany (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241459#comment-14241459
 ] 

mohamed Abo el haggag hassany commented on HADOOP-10433:


sorry guys can any one give me a document describe the process of the 
authentication


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.6.0

 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-10 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992794#comment-13992794
 ] 

Hudson commented on HADOOP-10433:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #1751 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/])
HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637)
* /hadoop/common/trunk/.gitignore
* 
/hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
* 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-10 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992830#comment-13992830
 ] 

Hudson commented on HADOOP-10433:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1777 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1777/])
HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637)
* /hadoop/common/trunk/.gitignore
* 
/hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
* 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-05 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990032#comment-13990032
 ] 

Larry McCay commented on HADOOP-10433:
--

Thanks for all the heavy lifting here, [~tucu00]!

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 3.0.0

 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-05 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990229#comment-13990229
 ] 

Hudson commented on HADOOP-10433:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #5593 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5593/])
HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) (tucu: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1592637)
* /hadoop/common/trunk/.gitignore
* 
/hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSRESTConstants.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/pom.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-acls.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSACLs.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAudit.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSCacheKeyProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSMDCFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSServerJSONUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat
* /hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF
* 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-04 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989086#comment-13989086
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

planning to commit on monday

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-02 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987966#comment-13987966
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Test failure seems unrelated.

Owen, Andrew, are we good to go now?

Vinay, a sample of the audit log follows. Regarding normalizing audit logs, 
that would be nice, mind opening a JIRA for it (it is out of scope from this 
JIRA).

{code}
2014-05-02 10:14:12,326 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:12,994 Status:OK User:cli...@example.com Op:CREATE_KEY 
Name:ck0UserProvidedMaterial:false Description:null
2014-05-02 10:14:13,027 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65499/kms/v1/keys?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,076 Status:OK User:client/h...@example.com Op:CREATE_KEY 
Name:ck1UserProvidedMaterial:false Description:null
2014-05-02 10:14:13,557 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,597 Status:UNAUTHORIZED User:cli...@example.com 
Op:CREATE_KEY Name:k
2014-05-02 10:14:13,597 Status:ERROR User:cli...@example.com Method:POST 
URL:http://localhost:65509/kms/v1/keys Exception:'User:cli...@example.com not 
allowed to do 'CREATE_KEY' on 'k''
2014-05-02 10:14:13,606 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65509/kms/v1/keys?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,645 Status:UNAUTHORIZED User:cli...@example.com 
Op:CREATE_KEY Name:k
2014-05-02 10:14:13,646 Status:ERROR User:cli...@example.com Method:POST 
URL:http://localhost:65509/kms/v1/keys Exception:'User:cli...@example.com not 
allowed to do 'CREATE_KEY' on 'k''
2014-05-02 10:14:13,653 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,692 Status:UNAUTHORIZED User:cli...@example.com 
Op:ROLL_NEW_VERSION Name:k
2014-05-02 10:14:13,692 Status:ERROR User:cli...@example.com Method:POST 
URL:http://localhost:65509/kms/v1/key/k Exception:'User:cli...@example.com not 
allowed to do 'ROLL_NEW_VERSION' on 'k''
2014-05-02 10:14:13,699 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65509/kms/v1/key/k?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,739 Status:UNAUTHORIZED User:cli...@example.com 
Op:ROLL_NEW_VERSION Name:k
2014-05-02 10:14:13,739 Status:ERROR User:cli...@example.com Method:POST 
URL:http://localhost:65509/kms/v1/key/k Exception:'User:cli...@example.com not 
allowed to do 'ROLL_NEW_VERSION' on 'k''
2014-05-02 10:14:13,746 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS URL:http://localhost:65509/kms/v1/keys/names?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,781 Status:UNAUTHORIZED User:cli...@example.com Op:GET_KEYS 
Name:*
2014-05-02 10:14:13,781 Status:ERROR User:cli...@example.com Method:GET 
URL:http://localhost:65509/kms/v1/keys/names Exception:'User:cli...@example.com 
not allowed to do 'GET_KEYS' on '*''
2014-05-02 10:14:13,791 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS 
URL:http://localhost:65509/kms/v1/keys/metadata?key=kuser.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,825 Status:UNAUTHORIZED User:cli...@example.com 
Op:GET_KEYS_METADATA Name:k
2014-05-02 10:14:13,826 Status:ERROR User:cli...@example.com Method:GET 
URL:http://localhost:65509/kms/v1/keys/metadata?key=k 
Exception:'User:cli...@example.com not allowed to do 'GET_KEYS_METADATA' on 'k''
2014-05-02 10:14:13,836 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS 
URL:http://localhost:65509/kms/v1/keyversion/k%400?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,871 Status:UNAUTHORIZED User:cli...@example.com 
Op:GET_KEY_VERSION Name:k@0
2014-05-02 10:14:13,871 Status:ERROR User:cli...@example.com Method:GET 
URL:http://localhost:65509/kms/v1/keyversion/k%400 
Exception:'User:cli...@example.com not allowed to do 'GET_KEY_VERSION' on 'k@0''
2014-05-02 10:14:13,878 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS 
URL:http://localhost:65509/kms/v1/key/k/_currentversion?user.name=tucu 
ErrorMsg:'Authentication required'
2014-05-02 10:14:13,912 Status:UNAUTHORIZED User:cli...@example.com 
Op:GET_CURRENT_KEY Name:k
2014-05-02 10:14:13,912 Status:ERROR User:cli...@example.com Method:GET 
URL:http://localhost:65509/kms/v1/key/k/_currentversion 
Exception:'User:cli...@example.com not allowed to do 'GET_CURRENT_KEY' on 'k''
2014-05-02 10:14:13,919 Status:UNAUTHENTICATED RemoteHost:127.0.0.1 
Method:OPTIONS 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-02 Thread Vinay Shukla (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987985#comment-13987985
 ] 

Vinay Shukla commented on HADOOP-10433:
---

Thanks for the sample Audit record. I have created a JIRA for normalizing audit 
records. https://issues.apache.org/jira/browse/HADOOP-10569

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-02 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987987#comment-13987987
 ] 

Andrew Wang commented on HADOOP-10433:
--

LGTM, +1 again

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-02 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988457#comment-13988457
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Owen? are we good as the initial commit, and we follow up with other JIRAs 
improvements?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-01 Thread Owen O'Malley (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986936#comment-13986936
 ] 

Owen O'Malley commented on HADOOP-10433:


Alejandro,
  A couple more things on the interface:
* You need to allow the user to specify the key-name when they create the key.
* Instead of throwing if the url is too long, please break the user's request 
into parts that fit within the 2000 byte limit.
* You never answered whether the body is xml or json (or both).
* The underscores should be on the trailing part of the url.

I'd propose it look like:
{code}
create key : PUThttp://HOST:PORT/kms/v1/key/key-name
rollover key   : POST   http://HOST:PORT/kms/v1/key/key-name
delete key : DELETE http://HOST:PORT/kms/v1/key/key-name
get metadata   : GEThttp://HOST:PORT/kms/v1/key/key-name/_metadata
get current version: GET
http://HOST:PORT/kms/v1/key/key-name/_currentversion
get key versions   : GEThttp://HOST:PORT/kms/v1/key/keyname-name/_versions
get key version: GEThttp://HOST:PORT/kms/v1/keyversion/version-name
get key names  : GEThttp://HOST:PORT/kms/v1/keys/names
get keys metadata  : GET
http://HOST:PORT/kms/v1/keys/metadata?key=key-namekey=keyname,...
{code}

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-01 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987166#comment-13987166
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Owen,

bq. You need to allow the user to specify the key-name when they create the key.

It is happening, it goes in the JSON payload of the create operation. Look in 
the {{createKeyInternal()}} method

bq. You never answered whether the body is xml or json (or both).

Request/response bodies are JSON.

I'll integrate your other 2 suggestions, the underscore and the break into 
multiple requests if URL goes beyond 2K byte limit.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-01 Thread Vinay Shukla (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987185#comment-13987185
 ] 

Vinay Shukla commented on HADOOP-10433:
---

[~tucu00] What is the audit story about the Key? Do we record, who did various 
key operations? I couldn't find it in the attached PDFs.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-01 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987191#comment-13987191
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Vinay, there is a kms-audit.log file that logs all requests, both successful 
and failed. 

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-05-01 Thread Vinay Shukla (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987339#comment-13987339
 ] 

Vinay Shukla commented on HADOOP-10433:
---

Is there a sample of the audit log record? What field are audited? Would it be 
useful to have a common audit format across Hadoop?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-30 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985782#comment-13985782
 ] 

Andrew Wang commented on HADOOP-10433:
--

+1 from me, thanks Tucu. Really nice work here. Let's wait a bit before 
committing to see if anyone else still has review comments.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-30 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986019#comment-13986019
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

sure, i'll wait till Friday. Thanks again.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984101#comment-13984101
 ] 

Andrew Wang commented on HADOOP-10433:
--

Thanks for the rev here Tucu, I can tell this wasn't a small effort. Overall it 
looks real nice, the new REST API is fine by me. I have some more review 
comments though, but expect to +1 after this round (pending agreement by 
reviewers on the REST API details).

Running the KMS:
* Overall a more pleasant experience, tarball builds, audit logging has 
newlines, I didn't see stacktraces when running commands.
* I know I asked for less spew in kms.sh, but defaulting KMS_SILENT to true is 
perhaps too little. Without it, running just ./sbin/kms.sh shows nothing, not 
even help text, and if start/stop hit an error, we again see nothing (have to 
know where the logs are and hope it has the error there). This isn't a biggie, 
but maybe we can improve it a little more.

JSON:
* I see KMSMetadata and KMSKeyVersion classes, but they don't look like they do 
much. I'm guessing this is from an aborted attempt to do POJO-JSON? That'd 
would be awesome, but I won't get too excited.
* If you stick with the JSON helper methods, could we pull them all out into a 
separate helper class and make them all static, along with the field constants? 
Feels weird to have the Server digging into ClientProvider for field names, 
when theoretically a different client implementation (REST!) could be 
interfacing with the KMS.

KMSServer and KMSClientProvider:
* Noticed that KMSServer#getKeyMetadata includes the name, but it's unused on 
the client side, nor part of KeyProvider#Metadata.
* I think we still miss audit logging if an exception is thrown. e.g. in 
#rolloverKey, getPrincipal, checkNotEmpty, provider.rollNewVersion can all 
throw something. This is why a try/finally might be easiest.
* Can make many methods static: validateResponse, call, all the JSON functions, 
getPrincipal, assertAccess, removeKeyMaterial, getKeyURI
* On the NoSuchAlgorithmException, #validateResponse casts everything to an 
IOException, so I don't think anything else comes out of this function. Not 
sure this cast is kosher. I think the catch for Exception there also needs to 
wrap the created String into an exception and throw it.
* Given the discussion about UUIDs as versions, should we be exposing 
KMSClientProvider#buildVersionName publicly? This seems like an implementation 
detail of the backing KeyProvider.

Misc things:
* I think a rogue regex changed GET to GET_OP across the entire hadoop repo, 
which is why we're picking up all these test changes, and quite possibly why we 
have some weird test failures.
* KMSACLs, the new comment about in #run probably should go in #loadACLs, or be 
removed.
* KeyNotFoundException serialVersionUID, this is something Eclipse is warning 
about, so it'd be nice to just add it.
* In the cache provider, #getCurrentKey and getKeyVersion, could set the cause 
once in a temp var to avoid having to call getCause() repeatedly.
* KMSServer semantically should still be renamed to KMServer or just KMS

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984138#comment-13984138
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642395/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3866//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984300#comment-13984300
 ] 

Larry McCay commented on HADOOP-10433:
--

Hey [~tucu00] - I was wondering about your  It may barbaric, but that is how 
things are and we should make sure we can work with those.

I think you may have misinterpreted my reference the the OpenStack Barbican 
project - which is a REST based KMS server. I don't think UUIDs are barbaric. :)

Thank's for the response.
I need to give the rest of your explanation a bit more thought.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Owen O'Malley (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984497#comment-13984497
 ] 

Owen O'Malley commented on HADOOP-10433:


Alejandro,
   I like the direction your api is going. I'm a little concerned about the 
length of the URL (2000 bytes) causing trouble for the keys/metadata method. 
I'd suggest adding a field in the header that provides the list of key names. 
I'd also suggest that you add a character to the fields (metadata, 
currentversion, versions) such as adding underscore (_metadata, 
_currentversion, _versions).

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984534#comment-13984534
 ] 

Larry McCay commented on HADOOP-10433:
--

I can see how having a .../keyversion/version-name is aligned with the 
KeyProvider API and that it allows the version-name to be more easily 
considered opaque.

I would like to point out that I think we need to have the key-name for the 
create key API - you generally fully identify the resource that you are 
creating (PUTting). Are you assuming that it is part of the body of the request?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984651#comment-13984651
 ] 

Larry McCay commented on HADOOP-10433:
--

Yes, that aligns with collections based actions - although they are supposed to 
be a POST for collection actions that create a new element. A PUT is supposed 
to replace the entire collection with another collection - which is NOT what 
you want.

Element based actions on the other hand are done with a PUT and replace the 
element or create it if it does not exist.

So, it seems that create should either be made a POST or an element based 
action.


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984660#comment-13984660
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642504/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3873//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984662#comment-13984662
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Larry, good point, i'll update the patch making it a POST. and I'll take care 
of the findbugs

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984750#comment-13984750
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642512/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3874//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Vinay Shukla (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984908#comment-13984908
 ] 

Vinay Shukla commented on HADOOP-10433:
---

Does KMS support KMIP protocol ? 
(https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip)

Will KMS integrate with Hardware Security Modules (HSM) devices such as SafeNet 
Luna   RSA's Data Protection Manager 
(http://www.emc.com/security/rsa-data-protection-manager.htm)?

If KMS does not speak KMIP, how can Hadoop encryption leverage enterprise grade 
Key management investment many enterprise level customers already have?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984917#comment-13984917
 ] 

Larry McCay commented on HADOOP-10433:
--

KMS leverages KeyProvider API a kmip provider would be the integration vehicle. 
KMS would pick it up for free.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984918#comment-13984918
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Vinay, KMS uses a KeyProvider as the underlying keystore. A KeyProvider impl 
supporting RSA/SafeNet can be plugged in. AFAIK, both RSA and SafeNet have JCE 
providers with keystore support, thus getting the JavaKeyProvider to work with 
them shouldn't be that difficult (I would assume).

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-29 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985076#comment-13985076
 ] 

Hadoop QA commented on HADOOP-10433:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642548/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3878//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Charles Lamb (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983285#comment-13983285
 ] 

Charles Lamb commented on HADOOP-10433:
---

One little nit in the REST API: for the get operations, rather than use 
?get=..., perhaps consider calling it op instead. For example, 

GET .../key/key-name?op=getCurrentVersion

and

GET .../keys?op=getKeyNames


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983299#comment-13983299
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Charles, based on Andrew's feedback I've moved away from operations. get 
indicates what part of the resource we want to retrieve, either get= or show= I 
would say, op= is a bit confusing as it is not an operation.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Charles Lamb (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983328#comment-13983328
 ] 

Charles Lamb commented on HADOOP-10433:
---

I guess what I was responding to was having keynames embedded in the URI rather 
than as a parameter after the ?. For instance, the get key version operation 
uses

/key-name?get=version

but the get keys metadata operation puts the keyname as a parameter value:

/keys?get=keyMetaDatakey=key-name

I like the second form where the key-name is a value rather than an operator so 
perhaps this can be unified?


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983348#comment-13983348
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

well, typically REST resources are expressed as a PATH more than as 
QUERY_STRING, I'd prefer to keep it that way.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983355#comment-13983355
 ] 

Larry McCay commented on HADOOP-10433:
--

Personally, I feel that the keyname as part of the URI adheres to accepted REST 
principles. In fact, I'd like to understand why the parameters are needed at 
all. One should be able to represent the desired resource entirely by URI. 
There are times when non-hierarchical attributes need to be considered (as 
filters) that are easily added to a query string. But largely, the APIs we are 
looking at here are hierarchical.

GET .../keys - would return all the keyNames
GET .../keys/keyname - would return the metadata for that key (or the current 
key maybe)
GET .../keys/keyname/version - would return the specific key (could have a 
well-known current version as well)

Just my two cents.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983448#comment-13983448
 ] 

Larry McCay commented on HADOOP-10433:
--

I do have to say that I don't like the use of get as a query parameter. It is 
semantically redundant with the HTTP verb in a well designed REST API.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983478#comment-13983478
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

ok, let me work on a revised version using path to refer sub-resources 
metadata/version/versions

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983504#comment-13983504
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642256/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3863//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Charles Lamb (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983702#comment-13983702
 ] 

Charles Lamb commented on HADOOP-10433:
---

I like this better.


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983719#comment-13983719
 ] 

Larry McCay commented on HADOOP-10433:
--

I like this much better as well.
To me, you could continue the hierarchy of the key/keyversion as well.
Additionally, I'm not sure that you need to identify the path elements with 
key/keyname/version/version-name.
Each resource's path is unique unto itself and I think it holds together to 
just use the values that way - as you follow the path the URI more specifically 
identifies the resource.

With that in mind, I would propose adjustments along the following lines:

{code}
create key : PUThttp://HOST:PORT/kms/v1/keys
rollover key   : POST   http://HOST:PORT/kms/v1/keys/key-name
delete key : DELETE http://HOST:PORT/kms/v1/keys/key-name
get metadata   : GEThttp://HOST:PORT/kms/v1/keys/key-name/metadata
get current version: GET
http://HOST:PORT/kms/v1/keys/key-name/currentversion
get key version: GET
http://HOST:PORT/kms/v1/keys/key-name/version-name
get key versions   : GEThttp://HOST:PORT/kms/v1/keys/version-name/versions
get key names  : GEThttp://HOST:PORT/kms/v1/keys
get keys metadata  : GET
http://HOST:PORT/kms/v1/keys/metadata?key=key-namekey=keyname,...
{code}

This flattens the hierarchy a good bit the only outlier being the get 
.../keys/metadata - I think that the rest of them identify unique resources 
within a hierarchy.

[~tucu00]
I'm not sure that I am grasping your concern with the 'get key version' URI and 
doubt that this minor adjustments changes your feelings there. Can you 
elaborate on what the issue is? Maybe provide an example?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Charles Lamb (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983720#comment-13983720
 ] 

Charles Lamb commented on HADOOP-10433:
---

Nit: there are still some instances of chiper in the design doc.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983728#comment-13983728
 ] 

Andrew Wang commented on HADOOP-10433:
--

Larry, I think an issue with that scheme is that we can't have a key with the 
name currentversion  or metadata then, or a key and a version with the same 
name. We need additional path elements to disambiguate.

As to the get key version API, it's think it's an unfortunate consequence of 
the existing KeyProvider#getKeyVersion API, which takes a single string. Tucu's 
concern there is that if a KP decides to use a UUID instead of 
keyName@version like JavaKeyStoreProvider, we can't fit it into the 
hierarchical key-then-version scheme.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983753#comment-13983753
 ] 

Larry McCay commented on HADOOP-10433:
--

hmmm...

Seems like this one won't work that way with the rest:
{code}
get key versions   : GEThttp://HOST:PORT/kms/v1/keys/version-name/versions
{code}

I'm not even really sure I know what it is trying to depict.
Perhaps it is supposed to be like?:

{code}
get key versions   : GEThttp://HOST:PORT/kms/v1/keys/key-name/versions
{code}

I still don't quite like the last one with .../keys/metadata as it overlaps 
with a key named metadata in the hierarchy.
I'm not sure what to do with that one.

Maybe the following would work - obviously you aren't asking for the names 
since you are providing them:

{code}
get keys metadata  : GET
http://HOST:PORT/kms/v1/keys?key=key-namekey=keyname,...
{code}

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983763#comment-13983763
 ] 

Larry McCay commented on HADOOP-10433:
--

Hi [~andrew.wang] - I also thought that those types of providers would need to 
be adapted by the KeyProvider implementation to always default the version - 
assuming that the provider doesn't actually support versions. So, in that case 
it would be be UUID@0. Current version would always be 0 or 1 - whatever makes 
sense.

There are other systems that seem to have keyNames with UUIDs as the version. I 
think that barbican uses this scheme. It would have to do something similar 
within the KeyProvider implementation to adapt it to the API. Not sure whether 
the UUID version can be used as the actual version.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983783#comment-13983783
 ] 

Larry McCay commented on HADOOP-10433:
--

AFAICT - we don't have a problem for a key named currentversion or metadata. 
Currentversion would have special meaning within the versions context and 
metadata wouldn't make sense as a version name anyway.

I don't see why a key and version name couldn't be the same either - though I 
think versions are generally numeric. So, key 1234 could have 1234 versions in 
which case 1234@1234 would be reasonable and not precluded by the URI's 
proposed.

{code}
get key version: GEThttp://HOST:PORT/kms/v1/keys/1234/1234
{code}

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-28 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13983944#comment-13983944
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642355/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
  
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3865//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982273#comment-13982273
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.crypto.key.kms.server.TestKMSServer
  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3859//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-27 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982282#comment-13982282
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642105/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.ha.TestZKFailoverControllerStress
  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3858//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981935#comment-13981935
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642061/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3856//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982068#comment-13982068
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642080/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

  org.apache.hadoop.mapreduce.lib.db.TestDBJob
  
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

  The following test timeouts occurred in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-mapreduce-project/hadoop-mapreduce-examples 
hadoop-tools/hadoop-openstack 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

org.apache.hadoop.mapred.pipes.TestPipeApplication

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3857//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-25 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981822#comment-13981822
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12642042/HadoopKMSDocsv2.pdf
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3853//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, HadoopKMSDocsv2.pdf, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-23 Thread Owen O'Malley (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978967#comment-13978967
 ] 

Owen O'Malley commented on HADOOP-10433:


What is the form of the urls for the REST service? What are the acceptable 
formats of the data in the REST service (JSON? XML?)?

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-23 Thread Owen O'Malley (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978969#comment-13978969
 ] 

Owen O'Malley commented on HADOOP-10433:


Please don't use the Hadoop ACLs. They are very error-prone with x meaning 
user x and  x meaning group x. I'd suggest using : as the separator and 
making it ignore spaces.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-22 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977803#comment-13977803
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

Andrew, thanks for the detailed review. 

I'm working on a patch addressing most of your feedback, following some 
questions/clarifications:

* KeyNotFoundException needing serialVersionUID, why do we need this? we are 
not serializing these exceptions.
* The big log entries happen only on the first REST call only, a Jersey 
initialization thing (fixed in newer Jerseys)
* Regarding using curl directly, the REST API is not meant to be public at the 
moment, that is why it is not documented.
* Doing POJO-JSON automatically requires the beans to have a public default 
constructor and public getters/setters. Given the discussion in HADOOP-10431, 
this is not an option.
* KMSClientProvider constants going private, all of them are used in the 
KMSServer as well.
* KMSClientProvider.rollNewVersion() throwing a NoSuchAlgorithmException, yes 
it can, it may be come as part of the remote exception in the error message.


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-16 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13972199#comment-13972199
 ] 

Andrew Wang commented on HADOOP-10433:
--

Thanks for the huge amount of work here Tucu. I'm still fresh to this topic, so 
please forgive any silly review comments.

* In the docs, Hadoop KMS Server is like saying ATM machine :) docs could 
use a pass, but we can do that after the base patch is committed.
* For the same reason, you could rename KMSServer to just KMS or KMServer
* More javadoc, at the minimum class javadoc for everything
* KeyNotFoundException needs a serialVersionUID
* pom.xml, not sure where these vars are being used, but I'm not aware of 
Maven's subtleties.
{code}
kms.source.repositoryREPO NOT AVAIL/kms.source.repository
kms.source.revisionREVISION NOT AVAIL/kms.source.revision
kms.build.timestamp${maven.build.timestamp}/kms.build.timestamp
{code}
* mvn package -Pdist -Dtar doesn't work because there's an extra newline in 
the pom.xml for dist-maketar.sh.

Notes from running the KMS:
* Running ./sbin/kms.sh has a lot of debugging output, not sure if we can 
quash it or not.
* Is there some way of aborting tomcat if the KMSWebApp dies? I didn't put a 
core-site.xml in my conf dir initially, KMSWebApp FATAL'd, and I had a useless 
Tomcat process lying around.
* Running ops against the KMS with hadoop key generates big log entries like 
this:
{code}
2014-04-16 16:40:28,160 INFO  WadlGeneratorJAXBGrammarGenerator - Couldn't find 
JAX-B element for class javax.ws.rs.core.Response
2014-04-16 16:40:28,160 ERROR WadlGeneratorJAXBGrammarGenerator - 
java.lang.IllegalAccessException: Class 
com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can 
not access a member of class javax.ws.rs.core.Response with modifiers 
protected
stacktrace
{code}
* Audit logging doesn't have newlines
* I didn't test anything outside of KeyShell operations. I couldn't figure out 
the right curl incantation to hit REST endpoints directly, whenever I tried I 
got an authentication exception about being an anonymous user.

REST API design comments:
* It's kind of ugly that we're doing manual serde of things into JSON, since it 
throws away types and makes the protocol harder to understand. Jackson has ways 
of automatically doing POJO-JSON for beans, which seems pretty reasonable to 
me.
* If we use Jackson, we should really aspire to JSON in, JSON out instead of 
using so many query parameters (in particular, create).
* I think we should bake the op into the path when possible, since it makes the 
API more self-descriptive. Maybe something like this:
{code}
/keys
/keys?metadata # this is still ugly, dunno
/keys/keyName # GET for metadata, PUT for create, POST for roll, DELETE
/keys/keyName/versions # getKeyVersions
/keys/keyName/versions/version # getKeyVersion
/keys/keyName/versions/current
{code}
* Also, shouldn't the REST API be explained in the docs? I assume we chose REST 
for interop with non-Hadoop and non-Java components, so the protocol should be 
documented.

KMSClientProvider:
* Class javadoc refers to JksProviders
* Typo: CHIPER_FIELD and chiper, I assume cipher
* It'd be nice to put all the {{private static final}} before {{public static 
final}}. Some of the public ones also aren't used outside of this class, so we 
could close them up.
* Constructor, can use equalsIgnoreCase rather than toLowerCase
* Wherever we're building a URL, consider using URIBuilder and then toURL() 
instead.
* Based on this page 
(http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html) 
we should try/catch when calling HttpURLConnection#getInputStream and I assume 
#getResponseCode too, and also close the input stream after we're done. The 
error stream logic in #validateResponse also doesn't quite adhere to the 
recommendations.
* Is there a reason flush is a NOP? The JKSP doesn't do this, so a explanatory 
comment would help.
* It doesn't look like rollNewVersionInternal needs to throw 
NoSuchAlgorithmException?

KMSACLs:
* You could write this with a ScheduledExecutorService and a 
ThreadFactoryBuilder, more idiomatic.
* Probably should make start/stopReloader idempotent to future proof it, e.g. 
check a isRunning boolean.
* In loadACLs, I assume you do the conf get to init the conf outside of the 
lock? Would help to clarify that in the comment.
* The lastReload time should be set to the modtime of the file, not by calling 
System.currentTimeMillis again. Also, what's the point of waiting for it to be 
at least 100ms newer?
* I'd bump the setACLs logs up to INFO, since this probably isn't a frequent 
operation and is pretty useful.

KMSServer:
* The hasAccess method is a bit gross. We could make an Access enum rather than 
passing the ACL as an argument. Could also get rid of the boolean for throwing 
an exception, and have two helper methods {{checkAccess}} 

[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-11 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966976#comment-13966976
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

[~lmccay] the KMS server is meant to be a proxy to any KeyProvider 
implementation with the following benefits:

* Single entry point from Hadoop to the KeyProvider implementation
* Supports Hadoop Kerberos Authentication, making secure integration easier
* Provides caching, to support load from Hadoop services and jobs


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: COMBO.patch, HADOOP-10433-v2.patch, 
 HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-11 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966988#comment-13966988
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12639842/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1289 javac 
compiler warnings (more than the trunk's current 1288 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3787//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: COMBO.patch, HADOOP-10433-v2.patch, 
 HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-11 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967006#comment-13967006
 ] 

Larry McCay commented on HADOOP-10433:
--

[~tucu00] - that sounds like the right direction to me!


 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: COMBO.patch, HADOOP-10433-v2.patch, 
 HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-11 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967103#comment-13967103
 ] 

Hadoop QA commented on HADOOP-10433:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12639861/HADOOP-10433.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3788//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, HADOOP-10433.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-09 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964751#comment-13964751
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12639468/COMBO.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1289 javac 
compiler warnings (more than the trunk's current 1288 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist:

  org.apache.hadoop.crypto.key.TestKeyShell

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3775//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: COMBO.patch, HADOOP-10433-v2.patch, 
 HADOOP-10433-v3.patch, HADOOP-10433.patch, HADOOP-10433.patch, 
 KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, KMS-ALL-PATCHES.patch, 
 KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-03 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958858#comment-13958858
 ] 

Larry McCay commented on HADOOP-10433:
--

I don't think that features are an issue at this point.
My larger concern is whether Hadoop common needs a KMS server offering of its 
own or not.
Given that the keystore provider is not likely a scalable provider for a KMS 
that will lead us down the path of requiring an appropriate DB to do it right. 
This was my thinking as I starting prototyping on an earlier version of the 
KeyProvider API as well.

It became reasonable to me that the KeyProvider API within Hadoop common was 
sufficient to enable deployments to plugin any number of various KeyProvider 
implementations. This might be:
1. direct to KMIP
2. to an OpenStack Barbican server
3. to a Knox (or wherever) KMS

This would allow Hadoop common to not have to take on the weight of another 
server.

I'm not really arguing against putting it common here but recollecting my 
thought process on the matter.

Here is a question that I have had trouble answering with regard to the server 
and API being in common together...

Given the pluggable nature of the KeyProvider API what value does adding a full 
KMS server with additional (and maybe redundant) pluggability to common provide?

I've had trouble reconciling that in my mind.

Now, take that same server implementation and move it out of common and it 
easily makes sense to be able to have pluggability for the Hadoop KeyProvider 
API, KMIP and others.

So, when I said as long as it make sense to do so, I was really talking about 
whether it made sense to have it colocated with the KeyProvider API in common. 
I think the feature set is less of an issue.



 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-02 Thread Larry McCay (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958368#comment-13958368
 ] 

Larry McCay commented on HADOOP-10433:
--

I have a lot of interest in this project.
We have done some prototyping for a REST based key server within Knox.
I'd like to hopefully consolidate these into this jira as long as it makes 
sense to do so.

Obviously, there are HA requirements that aren't described here yet that will 
need some thought and discussion.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-02 Thread Uma Maheswara Rao G (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958505#comment-13958505
 ] 

Uma Maheswara Rao G commented on HADOOP-10433:
--

Hi Alejandro, Thanks a lot, for the doc.
Seems like you plan to use Tomcat which will be bundled with Hadoop.  Is there 
any special reason for not using Jetty which we are already using for simple 
web in Hadoop components? 
Yes, As pointed by Larry, I would like to see how the HA requirements would be 
handled. 

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-04-02 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958541#comment-13958541
 ] 

Alejandro Abdelnur commented on HADOOP-10433:
-

[~lmccay], if you have additional features in mind we should incorporate them 
in the KMS, agree. Would make sense to do them as follow up 
improvements/new-features JIRAs?

Regarding HA, the KMS server itself is stateless, you can put multiple behind a 
VIP and are done (from the KMS server perspective).  And the KeyProvider impl 
you are using must be HA itself (for example, it could be a HA DB or a HA KMIP 
sever).

[~umamaheswararao], regarding Tomcat vs Jetty. With embedded Jetty we will have 
to create configurations for all sorts of HTTP and server related things 
(connection threadpool size, http/socket timeout, ssl, port, etc, etc). With 
Tomcat all that come for free. The same approach is taken by HttpFS (and Oozie).

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-03-31 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955849#comment-13955849
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12637934/KMS-ALL-PATCHES-v2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1487 javac 
compiler warnings (more than the trunk's current 1485 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3730//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433.patch, 
 KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-03-31 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955929#comment-13955929
 ] 

Hadoop QA commented on HADOOP-10433:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12637950/KMS-ALL-PATCHES-v3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3731//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433-v2.patch, HADOOP-10433-v3.patch, 
 HADOOP-10433.patch, KMS-ALL-PATCHES-v2.patch, KMS-ALL-PATCHES-v3.patch, 
 KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HADOOP-10433) Key Management Server based on KeyProvider API

2014-03-28 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950446#comment-13950446
 ] 

Hadoop QA commented on HADOOP-10433:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12637338/KMS-ALL-PATCHES.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 1504 javac 
compiler warnings (more than the trunk's current 1491 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 9 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-assemblies hadoop-common-project/hadoop-common 
hadoop-common-project/hadoop-kms hadoop-dist:

  org.apache.hadoop.crypto.key.kms.server.TestKMSACLs
  org.apache.hadoop.crypto.key.kms.server.TestKMSServer

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-kms.html
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/3726//console

This message is automatically generated.

 Key Management Server based on KeyProvider API
 --

 Key: HADOOP-10433
 URL: https://issues.apache.org/jira/browse/HADOOP-10433
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-10433.patch, KMS-ALL-PATCHES.patch, KMS-doc.pdf


 (from HDFS-6134 proposal)
 Hadoop KMS is the gateway, for Hadoop and Hadoop clients, to the underlying 
 KMS. It provides an interface that works with existing Hadoop security 
 components (authenticatication, confidentiality).
 Hadoop KMS will be implemented leveraging the work being done in HADOOP-10141 
 and HADOOP-10177.
 Hadoop KMS will provide an additional implementation of the Hadoop 
 KeyProvider class. This implementation will be a client-server implementation.
 The client-server protocol will be secure:
 * Kerberos HTTP SPNEGO (authentication)
 * HTTPS for transport (confidentiality and integrity)
 * Hadoop ACLs (authorization)
 The Hadoop KMS implementation will not provide additional ACL to access 
 encrypted files. For sophisticated access control requirements, HDFS ACLs 
 (HDFS-4685) should be used.
 Basic key administration will be supported by the Hadoop KMS via the, already 
 available, Hadoop KeyShell command line tool
 There are minor changes that must be done in Hadoop KeyProvider functionality:
 The KeyProvider contract, and the existing implementations, must be 
 thread-safe
 KeyProvider API should have an API to generate the key material internally
 JavaKeyStoreProvider should use, if present, a password provided via 
 configuration
 KeyProvider Option and Metadata should include a label (for easier 
 cross-referencing)
 To avoid overloading the underlying KeyProvider implementation, the Hadoop 
 KMS will cache keys using a TTL policy.
 Scalability and High Availability of the Hadoop KMS can achieved by running 
 multiple instances behind a VIP/Load-Balancer. For High Availability, the 
 underlying KeyProvider implementation used by the Hadoop KMS must be High 
 Available.



--
This message was sent by Atlassian JIRA
(v6.2#6252)