Re: [DISCUSS] Hadoop RPC encryption performance improvements

Hi Jonathan, Thanks for filing the jira. I've been working on a scope doc the past few days. Initially I thought the KMS over Hadoop RPC depends on Daryn's RPC encryption improvement. But the past few weeks I realized I should work on this in parallel, so that we can have a proof of concept soon.

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Filed https://issues.apache.org/jira/browse/HADOOP-16119 to implemented KMS on Hadoop RPC. It seemed there were interested parties involved in the development of this feature. On Fri, Nov 2, 2018 at 4:35 PM Wei-Chiu Chuang wrote: > > Thanks all for the inputs, > > To offer additional information

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Thanks Daryn for your work. I saw you filed an upstream jira HADOOP-15977 and uploaded some patches for review. I'm watching the jira and will review shortly as fast as I can. Best On Wed, Oct 31, 2018 at 7:39 AM Daryn Sharp wrote: > Various

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Thanks all for the inputs, To offer additional information (while Daryn is working on his stuff), optimizing RPC encryption opens up another possibility: migrating KMS service to use Hadoop RPC. Today's KMS uses HTTPS + REST API, much like webhdfs. It has very undesirable performance (a few thous

Re: [DISCUSS] Hadoop RPC encryption performance improvements

One possibility (which we use in Kudu) is to use SSL for encryption but with a self-signed certificate, maintaining the existing SASL/GSSAPI handshake for authentication. The one important bit here, security wise, is to implement channel binding (RFC 5056 and RFC 5929) to prevent against MITMs. The

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Hi Wei-Chiu, Thanks for starting the thread and summarizing the problem. Sorry for slow response. We've been looking at the encrypted performance as well and are interested in this effort. We ran some benchmarks locally. Our benchmarks also showed substantial penalty for turning on wire encryption

Re: [DISCUSS] Hadoop RPC encryption performance improvements

r 31, 2018 6:43 AM To: Hadoop Common; Hdfs-dev Subject: Re: [DISCUSS] Hadoop RPC encryption performance improvements Ping. Any one? Cloudera is interested in moving forward with the RPC encryption improvements, but I just like to get a consensus which approach to go with. Otherwise I'll pi

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Various KMS tasks have been delaying my RPC encryption work – which is 2nd on TODO list. It's becoming a top priority for us so I'll try my best to get a preliminary netty server patch (sans TLS) up this week if that helps. The two cited jiras had some critical flaws. Skimming my comments, both

Re: [DISCUSS] Hadoop RPC encryption performance improvements

Ping. Any one? Cloudera is interested in moving forward with the RPC encryption improvements, but I just like to get a consensus which approach to go with. Otherwise I'll pick HADOOP-10768 since it's ready for commit, and I've spent time on testing it. On Thu, Oct 25, 2018 at 11:04 AM Wei-Chiu Ch

[DISCUSS] Hadoop RPC encryption performance improvements

Folks, I would like to invite all to discuss the various Hadoop RPC encryption performance improvements. As you probably know, Hadoop RPC encryption currently relies on Java SASL, and have _really_ bad performance (in terms of number of RPCs per second, around 15~20% of the one without SASL) Ther