Re: Encryption At Rest Question
I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add files to the zone. How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view the raw file, a *hdfs fs -cat *shows me the actual contents of the files since the datanode decrypts it before sending it. Thanks, Rajesh On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee ranadi...@gmail.com wrote: In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdisco http://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WAND http://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
Re: Encryption At Rest Question
You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell. Olivier From: Rajesh Kartha karth...@gmail.commailto:karth...@gmail.com Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Date: Tuesday, 24 February 2015 19:47 To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Cc: hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org Subject: Re: Encryption At Rest Question I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add files to the zone. How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it before sending it. Thanks, Rajesh On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee ranadi...@gmail.commailto:ranadi...@gmail.com wrote: In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.commailto:plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdiscohttp://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WANDhttp://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
RE: Encryption At Rest Question
The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN. Right, currently DEK is better to be protected by https on the wire. If you want to confirm the file is encrypted, one way is to see the content of file blocks. Regards, Yi Liu From: Rajesh Kartha [mailto:karth...@gmail.com] Sent: Wednesday, February 25, 2015 3:48 AM To: user@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org Subject: Re: Encryption At Rest Question I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add files to the zone. How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it before sending it. Thanks, Rajesh On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee ranadi...@gmail.commailto:ranadi...@gmail.com wrote: In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.commailto:plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdiscohttp://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WANDhttp://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
Re: Encryption At Rest Question
Thank you Olivier, I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would kick in and prevent an unwarranted user to read them. However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block. Regards, Rajesh On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault orena...@hortonworks.com wrote: You can try looking at it with a user who doesn’t have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell. Olivier From: Rajesh Kartha karth...@gmail.com Reply-To: user@hadoop.apache.org user@hadoop.apache.org Date: Tuesday, 24 February 2015 19:47 To: user@hadoop.apache.org user@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org Subject: Re: Encryption At Rest Question I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add files to the zone. How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view the raw file, a *hdfs fs -cat *shows me the actual contents of the files since the datanode decrypts it before sending it. Thanks, Rajesh On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee ranadi...@gmail.com wrote: In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdisco http://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WAND http://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
Re: Encryption At Rest Question
You can set some ACL on the KMS so you could have an open access from an HDFS ACL perspective but a restriction from KSM side. Olivier From: Rajesh Kartha karth...@gmail.commailto:karth...@gmail.com Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Date: Tuesday, 24 February 2015 22:16 To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Subject: Re: Encryption At Rest Question Thank you Olivier, I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would kick in and prevent an unwarranted user to read them. However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block. Regards, Rajesh On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault orena...@hortonworks.commailto:orena...@hortonworks.com wrote: You can try looking at it with a user who doesn't have permission to the folder. An alternative is to check which block it is on Linux and looking at the block using cat from a linux shell. Olivier From: Rajesh Kartha karth...@gmail.commailto:karth...@gmail.com Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Date: Tuesday, 24 February 2015 19:47 To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Cc: hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org Subject: Re: Encryption At Rest Question I was trying out the Transparent data at rest encryption and was able to setup the KMS, zones etc. and add files to the zone. How do I confirm if the files I added to the encryption zone are encrypted ? Is there a way to view the raw file, a hdfs fs -cat shows me the actual contents of the files since the datanode decrypts it before sending it. Thanks, Rajesh On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee ranadi...@gmail.commailto:ranadi...@gmail.com wrote: In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.commailto:plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdiscohttp://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WANDhttp://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
Re: Encryption At Rest Question
On 2/24/2015 8:56 PM, Liu, Yi A wrote: The data is decrypted on client side after obtaining DEK from KMS, *not* decrypted by DN. My colleague Yi is correct that data is not decrypted by the DN with one exception: WebHDFS uses the DN as the proxy and therefore the DN does the decryption in that case. HttpFs is recommended instead. Right, currently DEK is better to be protected by https on the wire. If you want to confirm the file is encrypted, one way is to see the content of file blocks. Another way is to use the /.reserved/raw prefix on a file. This special prefix is only accessible by the hdfs admin. It gives the encrypted (raw) bits of a file rather than the decrypted bits. For example, if you have a file /ez/myfile, then /.reserved/raw/ez/myfile will yield the encrypted bits of the file. Charles
Re: Encryption At Rest Question
In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted data block and associate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a different DEK, the intercepter has to gain quite a bit of access before the data is compromised. On 18 February 2015 at 00:04, Plamen Jeliazkov plamen.jeliaz...@wandisco.com wrote: Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen 5 reasons your Hadoop needs WANdisco http://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WAND http://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. -- Regards, Ranadip Chatterjee
Encryption At Rest Question
Hey guys, I had a question about how the new file encryption work done primarily in HDFS-6134. I was just curious, how is the DEK protected on the wire? Particularly after the KMS decrypts the EDEK and returns it to the client. Thanks, -Plamen -- 5 reasons your Hadoop needs WANdisco http://www.wandisco.com/system/files/documentation/5-Reasons.pdf Listed on the London Stock Exchange: WAND http://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
