Hi all, I'd like to call a vote to merge the fs-encryption branch to trunk. Development of this feature has been ongoing since March on HDFS-6134 and HADOOP-10150, totally approximately 50 commits.
The fs-encryption branch introduces support for transparent, end-to-end encryption within an "encryption zone". Each file stored within an encryption zone is automatically encrypted and decrypted with a unique key. These per-file keys are encrypted with an encryption key only accessible by the client, ensuring that only the client is able to decrypt sensitive data. Furthermore, there is support for native, hardware-accelerated AES encryption. For further details, please see the design doc on HDFS-6134. In terms of merge readiness, we've posted some successful consolidated patches to the JIRA for Jenkins runs. distcp and fs -cp support has also recently been completed, allowing users to securely copy encrypted files without first decrypting them. There is ongoing work to add support for WebHDFS, HttpFS, and other alternative access methods. Stephen Chu has also posted a test plan, and has already identified a few issues that have been fixed. Design and development of this feature was also a cross-company effort with many different contributors. I'd like to thank Charles Lamb, Yi Liu, Uma Maheswara Rao G, Colin McCabe, and Juan Yu for their code contributions and reviews. Alejandro Abdelnur was also instrumental, doing a lot of the design work and as well as writing most of the Hadoop Key Mangement Server (KMS). Finally, I'd like to thank everyone who gave feedback on the JIRAs. This includes Owen, Sanjay, Larry, Mike Y, ATM, Todd, Nicholas, and Andy, among others. With that, here's my +1 to merge this to trunk. Thanks, Andrew
