[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Attachment: HDFS-3637.patch Updated patch addressing Eli's feedback. Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-3637:
-
Attachment: HDFS-3637.patch
Thanks a lot for the updated review, Eli. I'm attaching an updated full patch
including this feedback.
For reference, here's the diff which incorporates your latest feedback:
{code}
diff --git
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
index 8190b37..7a95626 100644
---
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
+++
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
@@ -489,7 +489,8 @@ public class RemoteBlockReader extends FSInputChecker
implements BlockReader {
@Override
public IOStreamPair getStreams() {
-// TODO: Make RemoteBlockReader support encryption.
+// This class doesn't support encryption, which is the only thing this
+// method is used for. See HDFS-3637.
return null;
}
diff --git
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java
index 6726663..0d21a37 100644
---
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java
+++
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java
@@ -330,8 +330,10 @@ public class TestEncryptedTransfer {
assertEquals(PLAIN_TEXT, DFSTestUtil.readFile(fs, TEST_PATH));
assertEquals(checksum, fs.getFileChecksum(TEST_PATH));
- // Sleep for 15 seconds, after which the encryption key will no
- // longer be valid.
+ // Sleep for 15 seconds, after which the encryption key will no longer be
+ // valid. It needs to be a few multiples of the block token lifetime,
+ // since several block tokens are valid at any given time (the current
+ // and the last two, by default.)
LOG.info(Sleeping so that encryption keys expire...);
Thread.sleep(15 * 1000);
LOG.info(Done sleeping.);
{code}
I'm going to go ahead and commit this momentarily.
Add support for encrypting the DataTransferProtocol
---
Key: HDFS-3637
URL: https://issues.apache.org/jira/browse/HDFS-3637
Project: Hadoop HDFS
Issue Type: New Feature
Components: data-node, hdfs client, security
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch,
HDFS-3637.patch, HDFS-3637.patch
Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally
encrypted. However, actual data read or written between DNs and clients (or
DNs to DNs) is sent in the clear. When processing sensitive data on a shared
cluster, confidentiality of the data read/written from/to HDFS may be desired.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Resolution: Fixed Fix Version/s: 2.2.0-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a ton for the very thorough reviews, Eli. Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 2.2.0-alpha Attachments: HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Attachment: HDFS-3637.patch Identical to the last patch, but fixes the findbugs warning. Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3637.patch, HDFS-3637.patch, HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Attachment: HDFS-3637.patch Here's an updated patch which should fix the findbugs warnings, fixes the HA test failures, and adds support for using the client-side socket cache even when encryption is enabled. (The last patch disabled the socket cache when encryption was enabled.) Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3637.patch, HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Attachment: HDFS-3637.patch Here's a patch which implements this feature. This implementation works by adding SASL support to the DataTransferProtocol. When this feature is enabled, all uses of the DataTransferProtocol are wrapped by encrypted Input/Output streams, which are created based on an MD5-DIGEST SASL handshake. The shared key for this handshake is created by reusing Hadoop's existing security infrastructure for BlockTokens, which relies on a secret key shared between the NN and DNs. The BlockTokenSecretManager is extended to be able to issue/validate EncryptionKeys, which consist of a randomly-generated nonce signed with one of the block token secret keys. DataNodes are capable of creating/validating EncryptionKeys on their own, since they have access to the block token secret keys. Clients get an encryption key by requesting one from the NN, and using that subsequently when communicating with DNs. It's assumed that one must enable RPC confidentiality in order for the fetch of the encryption key to be protected. Regarding configuration, the NN and all DNs must enable this feature in their configurations. Clients need not update their configurations at all, but rather determine whether or not encryption is enabled when first communicating with the NN. Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3637) Add support for encrypting the DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3637: - Target Version/s: 2.2.0-alpha (was: 2.1.0-alpha) Status: Patch Available (was: Open) Add support for encrypting the DataTransferProtocol --- Key: HDFS-3637 URL: https://issues.apache.org/jira/browse/HDFS-3637 Project: Hadoop HDFS Issue Type: New Feature Components: data-node, hdfs client, security Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3637.patch Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally encrypted. However, actual data read or written between DNs and clients (or DNs to DNs) is sent in the clear. When processing sensitive data on a shared cluster, confidentiality of the data read/written from/to HDFS may be desired. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
