Have we evaluated GRPC? A robust RPC requires significant effort. Migrating to GRPC can save ourselves a lot of headache.
Haohui On Sat, Feb 27, 2016 at 1:35 AM Andrew Purtell <andrew.purt...@gmail.com> wrote: > I get a excited thinking about the prospect of better performance with > auth-conf QoP. HBase RPC is an increasingly distant fork but still close > enough to Hadoop in that respect. Our bulk data transfer protocol isn't a > separate thing like in HDFS, which avoids a SASL wrapped implementation, so > we really suffer when auth-conf is negotiated. You'll see the same impact > where there might be a high frequency of NameNode RPC calls or similar > still. Throughput drops 3-4x, or worse. > > > On Feb 22, 2016, at 4:56 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > > > > Thanks for the confirm and further inputs, Steve. > > > >>> the latter would dramatically reduce the cost of wire-encrypting IPC. > > Yes to optimize Hadoop IPC/RPC encryption is another opportunity Kerby > can help with, it's possible because we may hook Chimera or AES-NI thing > into the Kerberos layer by leveraging the Kerberos library. As it may be > noted, HADOOP-12725 is on the going for this aspect. There may be good > result and further update on this recently. > > > >>> For now, I'd like to see basic steps -upgrading minkdc to krypto, see > how it works. > > Yes, starting with this initial steps upgrading MiniKDC to use Kerby is > the right thing we could do. After some interactions with Kerby project, we > may have more ideas how to proceed on the followings. > > > >>> Long term, I'd like Hadoop 3 to be Kerby-ized > > This sounds great! With necessary support from the community like > feedback and patch reviewing, we can speed up the related work. > > > > Regards, > > Kai > > > > -----Original Message----- > > From: Steve Loughran [mailto:ste...@hortonworks.com] > > Sent: Monday, February 22, 2016 6:51 PM > > To: common-dev@hadoop.apache.org > > Subject: Re: Introduce Apache Kerby to Hadoop > > > > > > > > I've discussed this offline with Kai, as part of the "let's fix > kerberos" project. Not only is it a better Kerberos engine, we can do more > diagnostics, get better algorithms and ultimately get better APIs for doing > Kerberos and SASL —the latter would dramatically reduce the cost of > wire-encrypting IPC. > > > > For now, I'd like to see basic steps -upgrading minkdc to krypto, see > how it works. > > > > Long term, I'd like Hadoop 3 to be Kerby-ized > > > > > >> On 22 Feb 2016, at 06:41, Zheng, Kai <kai.zh...@intel.com> wrote: > >> > >> Hi folks, > >> > >> I'd like to mention Apache Kerby [1] here to the community and propose > to introduce the project to Hadoop, a sub project of Apache Directory > project. > >> > >> Apache Kerby is a Kerberos centric project and aims to provide a first > Java Kerberos library that contains both client and server supports. The > relevant features include: > >> It supports full Kerberos encryption types aligned with both MIT KDC > >> and MS AD; Client APIs to allow to login via password, credential > >> cache, keytab file and etc.; Utilities for generate, operate and > >> inspect keytab and credential cache files; A simple KDC server that > >> borrows some ideas from Hadoop-MiniKDC and can be used in tests but > >> with minimal overhead in external dependencies; A brand new token > mechanism is provided, can be experimentally used, using it a JWT token can > be used to exchange a TGT or service ticket; Anonymous PKINIT support, can > be experientially used, as the first Java library that supports the > Kerberos major extension. > >> > >> The project stands alone and is ensured to only depend on JRE for > easier usage. It has made the first release (1.0.0-RC1) and 2nd release > (RC2) is upcoming. > >> > >> > >> As an initial step, this proposal suggests using Apache Kerby to > upgrade the existing codes related to ApacheDS for the Kerberos support. > The advantageous: > >> > >> 1. The kerby-kerb library is all the need, which is purely in Java, > >> SLF4J is the only dependency, the whole is rather small; > >> > >> 2. There is a SimpleKDC in the library for test usage, which borrowed > >> the MiniKDC idea and implemented all the support existing in MiniKDC. > >> We had a POC that rewrote MiniKDC using Kerby SimpleKDC and it works > >> fine; > >> > >> 3. Full Kerberos encryption types (many of them are not available in > >> JRE but supported by major Kerberos vendors) and more functionalities > >> like credential cache support; > >> > >> 4. Perhaps the most concerned, Hadoop MiniKDC and etc. depend on the > >> old Kerberos implementation in Directory Server project, but the > >> implementation is stopped being maintained. Directory project has a > >> plan to replace the implementation using Kerby. MiniKDC can use Kerby > >> directly to simplify the deps; > >> > >> 5. Extensively tested with all kinds of unit tests, already being used > >> for some time (like PSU), even in production environment; > >> > >> 6. Actively developed, and can be fixed and released in time if > necessary, separately and independently from other components in Apache > Directory project. By actively developing Apache Kerby and now applying it > to Hadoop, our side wish to make the Kerberos deploying, troubleshooting > and further enhancement can be much easier and thereafter possible. > >> > >> > >> > >> Wish this is a good beginning, and eventually Apache Kerby can benefit > other projects in the ecosystem as well. > >> > >> > >> > >> This Kerberos related work is actually a long time effort led by Weihua > Jiang in Intel, and had been kindly encouraged by Andrew Purtell, Steve > Loughran, Gangumalla Uma, Andrew Wang and etc., thanks a lot for their > great discussions and inputs in the past. > >> > >> > >> > >> Your feedback is very welcome. Thanks in advance. > >> > >> > >> > >> [1] https://github.com/apache/directory-kerby > >> > >> > >> > >> Regards, > >> > >> Kai > > >
