[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472743#comment-16472743
]
Wei-Chiu Chuang edited comment on HADOOP-10768 at 5/12/18 1:17 AM:
-------------------------------------------------------------------
I am reviewing this patch now, and trying to push this feature as far as
possible, as RPC encryption performance problem is blocking some clusters that
need to meet more stringent security compliance.
There are already excellent reviews and comments made by [~daryn], [~atm],
[~dapengsun] so I am just trying to clear roadblocks.
rev008 still applies against trunk but does not compile due to changes in
HDFS-13087, HDFS-12594, .. and etc.
To expedite the review process, here's rev 009 that compiles against trunk.
We are testing rev008 on a live cluster now (Hadoop 3.0.0 + HBase 2.0.0-beta1 +
other components). So far, I found HBase2 does not compile with it, so filed
HBASE-20572 to address that.
Protocol-wise, it looks backward compatible, which is good since we won't wait
for Hadoop4 to include this feature.
Ran some simple tests (reading/writing files) successfully that involve mixing
new clients with old cluster. So that verifies the ciphers&codecs are
compatible too.
After applying the patch, rolling upgrade performed successfully with Cloudera
Manager.
Full cluster restart performed successfully too.
More reviews to come ...
[Edit: upon further look, it looks like HBase Master failed in some really bad
way, and it couldn't start working. Will dig into this further.]
was (Author: jojochuang):
I am reviewing this patch now, and trying to push this feature as far as
possible, as RPC encryption performance problem is blocking some clusters that
need to meet more stringent security compliance.
There are already excellent reviews and comments made by [~daryn], [~atm],
[~dapengsun] so I am just trying to clear roadblocks.
rev008 still applies against trunk but does not compile due to changes in
HDFS-13087, HDFS-12594, .. and etc.
To expedite the review process, here's rev 009 that compiles against trunk.
We are testing rev008 on a live cluster now (Hadoop 3.0.0 + HBase 2.0.0-beta1 +
other components). So far, I found HBase2 does not compile with it, so filed
HBASE-20572 to address that.
Protocol-wise, it looks backward compatible, which is good since we won't wait
for Hadoop4 to include this feature.
Ran some simple tests (reading/writing files) successfully that involve mixing
new clients with old cluster. So that verifies the ciphers&codecs are
compatible too.
After applying the patch, rolling upgrade performed successfully with Cloudera
Manager.
Full cluster restart performed successfully too.
More reviews to come ...
> Optimize Hadoop RPC encryption performance
> ------------------------------------------
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
> Affects Versions: 3.0.0-alpha1
> Reporter: Yi Liu
> Assignee: Dapeng Sun
> Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
