Jenkins build is back to normal : Hadoop-Common-trunk #1526
See https://builds.apache.org/job/Hadoop-Common-trunk/1526/changes
Jenkins build is back to normal : Hadoop-common-trunk-Java8 #228
See https://builds.apache.org/job/Hadoop-common-trunk-Java8/228/changes
Re: Key Rotation in Data-at-Rest Encryption
Apologize if I wasn't clear Is the EZ key version same as an alias for the key? yup the EDEK along with the EZ key version is stored in the FIleInfo FileInfo contains both EDEK and EZ key version. The FileInfo (you can look at the *org.apache.hadoop.fs.FileEncryptionInfo* class for more info) object is stored as the value of the extended attribute of that file. How is the KeyMaterial derived from the KeyAlias and where is the mapping between the two stored? Is it in the KMS? Yup. KMS extends the *org.apache.hadoop.crypto.key.KeyProvider* class. You can take a look at it or a concrete implementation such as JavaKeyStoreProvider for more information. Also, you should probably direct questions related to HDFS encryption to hdfs-...@hadoop.apache.org Cheers -Arun On Sun, Jun 14, 2015 at 12:11 AM, Sitaraman Vilayannur vrsitaramanietfli...@gmail.com wrote: Hi Arun, Thanks for your response. Could you explain this a bit further for me Is the EZ key version same as an alias for the key? The EDEK is stored in the extended attributes of the file and the EZkey Version is stored in the FileInfo why is the EZKey Version not stored in the extended attributes too. Where is the FileInfo object persisted? Is it in the NameNode? How is the KeyMaterial derived from the KeyAlias and where is the mapping between the two stored? Is it in the KMS? Thanks much for your help in this. Sitaraman On Sun, Jun 14, 2015 at 12:14 PM, Arun Suresh asur...@cloudera.com wrote: Hello Sitaraman, It is the EZ key version that is used to generate the EDEK (and which is ultimately stored in the encrypted file's extended attributes '*raw.hdfs.crypto.encryption.info http://raw.hdfs.crypto.encryption.info*'), not really the the EZ key itself (which is stored in the directory's extended attribute ‘ *raw.hdfs.crypto.encryption.zone*’). Essentially, each file in a directory has a unique EDEK.. and an EDEK is is generated with the current version of the directory EZ key. The EDEK along with the EZ key version is stored in the FIleInfo. While decrypting, both these are passed on to the KMS which provides the client with the DEK which can be used to decrypt the file. Hope this clarifies things. Cheers -Arun On Sat, Jun 13, 2015 at 9:51 PM, Sitaraman Vilayannur vrsitaramanietfli...@gmail.com wrote: HDFSDataatRestEncryption.pdf says the following about key rotation..(please see appended below at the end of the mail) If the existing files do not have their EDEKs reencrypted using the new ezkeyid, how would the existing files be decrypted? That is where is the mapping between files and its EZKey (for after key rotation different files have different EZKeys)ids stored and how is it retrieved? Thanks Sitaraman Key Rotation When the administrator causes a key rotation of the EZkey in the KMS, the encryption zone’s EZkey (stored in the encryption zone directory’s raw.hdfs.crypto.encryption.zone extended attribute) gets the new keyid and version (only the version changes). Any new files created in the encryption zone have their DEKs encrypted using the new key version. Existing files do not have their EDEKs reencrypted using the new ezkeyid/ version, but this will be considered as a future enhancement. Note that a key rotation only needs to causes a reencryption of the DEK, not a reencryption of the underlying file.
Re: Key Rotation in Data-at-Rest Encryption
Hello Sitaraman, It is the EZ key version that is used to generate the EDEK (and which is ultimately stored in the encrypted file's extended attributes '*raw.hdfs.crypto.encryption.info http://raw.hdfs.crypto.encryption.info*'), not really the the EZ key itself (which is stored in the directory's extended attribute ‘ *raw.hdfs.crypto.encryption.zone*’). Essentially, each file in a directory has a unique EDEK.. and an EDEK is is generated with the current version of the directory EZ key. The EDEK along with the EZ key version is stored in the FIleInfo. While decrypting, both these are passed on to the KMS which provides the client with the DEK which can be used to decrypt the file. Hope this clarifies things. Cheers -Arun On Sat, Jun 13, 2015 at 9:51 PM, Sitaraman Vilayannur vrsitaramanietfli...@gmail.com wrote: HDFSDataatRestEncryption.pdf says the following about key rotation..(please see appended below at the end of the mail) If the existing files do not have their EDEKs reencrypted using the new ezkeyid, how would the existing files be decrypted? That is where is the mapping between files and its EZKey (for after key rotation different files have different EZKeys)ids stored and how is it retrieved? Thanks Sitaraman Key Rotation When the administrator causes a key rotation of the EZkey in the KMS, the encryption zone’s EZkey (stored in the encryption zone directory’s raw.hdfs.crypto.encryption.zone extended attribute) gets the new keyid and version (only the version changes). Any new files created in the encryption zone have their DEKs encrypted using the new key version. Existing files do not have their EDEKs reencrypted using the new ezkeyid/ version, but this will be considered as a future enhancement. Note that a key rotation only needs to causes a reencryption of the DEK, not a reencryption of the underlying file.
set up jenkins test for branch-2
Hi, We touched this topic before but it was put on hold. I'd like to bring it to our attention again. From time to time we saw changes that work fine in trunk but not branch-2, and we don't catch the issue in a timely manner. The difference between trunk and branch-2 is sufficient to justify periodic jenkins test and even pre-commit test for branch-2. I created https://issues.apache.org/jira/browse/INFRA-9226 earlier but I'm not sure who are the right folks to take care of it. Any one could help follow-up? Thanks a lot and best regards, --Yongjun
Re: set up jenkins test for branch-2
pre-commit will already test on branch-2 provided you follow the patch naming guidelines. there is also a branch-2 specific jenkins job: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/ I'd suggest starting by looking at that job and filing jiras to address whatever the failures are. May 14th was the last time is wasn't marked as failed and that build was unstable, so there's probably a good deal of work. On Sun, Jun 14, 2015 at 3:00 PM, Yongjun Zhang yzh...@cloudera.com wrote: Hi, We touched this topic before but it was put on hold. I'd like to bring it to our attention again. From time to time we saw changes that work fine in trunk but not branch-2, and we don't catch the issue in a timely manner. The difference between trunk and branch-2 is sufficient to justify periodic jenkins test and even pre-commit test for branch-2. I created https://issues.apache.org/jira/browse/INFRA-9226 earlier but I'm not sure who are the right folks to take care of it. Any one could help follow-up? Thanks a lot and best regards, --Yongjun -- Sean
