[ https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123767#comment-14123767 ]
Chris Nauroth commented on HDFS-6606: ------------------------------------- Hi, [~hitliuyi]. Nice work! This looks like it's fully compatible too with the recent work in HDFS-2856 to remove the requirement to run DataNode as root. If I understand correctly, the {{DFSClient}} is still going to contact the NameNode to obtain an encryption key via {{ClientProtocol#getDataEncryptionKey}} when {{dfs.encrypt.data.transfer}} is true, but then the result wouldn't actually be used if a cipher is negotiated. It's a shame to keep around that extraneous RPC, but it's very small, and I don't see an easy way to change the code to avoid it. Maybe we could queue this up for future consideration. I'd just like to suggest a few more tests: # {{TestSaslDataTransfer}}: A new test here would validate that it works with the HDFS-2856 style, setting {{dfs.data.transfer.protection}} instead of {{dfs.encrypt.data.transfer}}. # {{TestBalancerWithEncryptedTransfer}}: A new test here would validate that everything works correctly end-to-end with the balancer. # {{TestBalancerWithSaslDataTransfer}}: Same as #2, using the HDFS-2856 style with {{dfs.data.transfer.protection}} configured instead of {{dfs.encrypt.data.transfer}}. > Optimize HDFS Encrypted Transport performance > --------------------------------------------- > > Key: HDFS-6606 > URL: https://issues.apache.org/jira/browse/HDFS-6606 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs-client, security > Reporter: Yi Liu > Assignee: Yi Liu > Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch, > HDFS-6606.003.patch, OptimizeHdfsEncryptedTransportperformance.pdf > > > In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol, > it was a great work. > It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports > three security strength: > * high 3des or rc4 (128bits) > * medium des or rc4(56bits) > * low rc4(40bits) > 3des and rc4 are slow, only *tens of MB/s*, > http://www.javamex.com/tutorials/cryptography/ciphers.shtml > http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/ > I will give more detailed performance data in future. Absolutely it’s > bottleneck and will vastly affect the end to end performance. > AES(Advanced Encryption Standard) is recommended as a replacement of DES, > it’s more secure; with AES-NI support, the throughput can reach nearly > *2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is > supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add > a new mode support for AES). > This JIRA will use AES with AES-NI support as encryption algorithm for > DataTransferProtocol. -- This message was sent by Atlassian JIRA (v6.3.4#6332)