> On Dec. 30, 2014, 10:46 p.m., Christopher Tubbs wrote:
> > server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java,
> >  line 41
> > <https://reviews.apache.org/r/29386/diff/4/?file=803164#file803164line41>
> >
> >     I don't think this should be so tightly coupled with the 
> > ZKAuthenticator. It doesn't seem that this coupling is necessary to provide 
> > authentication functionality.
> >     
> >     Have a different implementation of an Authenticator that hijacks the 
> > data storage of another implementation, could be quite confusing to 
> > administrators, especially if they need to swap out the implementation.
> >     
> >     See ACCUMULO-1300 and related issues for other options.
> 
> Josh Elser wrote:
>     I understand the intentions for ACCUMULO-1300; however, that's not 
> implemented. Being able to leverage the existing ZKAuthorizor was wonderful 
> so I really don't want to move away from how this fundamentally works.
>     
>     Shouldn't the Authenticator and Authorizor be fixed per instance 
> (allowing reset via `accumulo init --reset-security` or w/e)? What do you 
> actually want changed aside from documentation to make it clear to 
> administrators (I haven't written any documentation yet).
> 
> Christopher Tubbs wrote:
>     So, I pointed out in the conversation around ACCUMULO-259, that I would 
> prefer a pluggable security module that integrated all the security-related 
> functions, because they are so dependent on each other. However, they were 
> implemented as 3 separately configurable pluggable components. I would still 
> like to see them merged (which doesn't preclude making one from modular 
> components themselves).
>     
>     For now, I'm not really sure what it means for somebody to "create a 
> user" when the authenticator already validates them. That's why it's 
> confusing. Are we essentially making ZKAuthenticator Kerberos-aware, or are 
> we providing a separate Kerberos Authenticator? That's really what it boils 
> down to for me. It seems like we're doing a hybrid... but I think we should 
> solidify on one or the other.
> 
> Josh Elser wrote:
>     I view it as a Kerberos Authenticator. The implementation details of that 
> authenticator is that is happens to persist information into ZK to handle 
> things like permissions and authorizations, but a user doesn't have to know 
> that to use it. We could, hypothetically, swap out ZK for any other 
> persistent store for that information and it would continue to work.
>     
>     What would help alleviate your confusion? More javadocs? A different 
> class name?
> 
> Christopher Tubbs wrote:
>     Well, part is documentation. Part is behavior.
>     
>     If it's a separate authenticator, we should make sure that it does not 
> use the same storage area in such a tightly coupled way. It should not, for 
> instance, be setting things that look like passwords, and "create local 
> user", "list local users", and "delete local users" should be unsupported 
> client side operations, just as you've done with "change password" (since 
> users are managed within Kerberos). We already share storage between the 
> ZKAuthenticator and the corresponding permissions handler and authorizer, so 
> it's probably okay to share the directory identified with the principal. 
> However, it can seamlessly create this directory as needed.
>     
>     Mainly, I'm thinking the more we can separate the "local user" management 
> functions from the KerberosAuthenticator, the better, since users are managed 
> in Kerberos. Some of my concerns are with the current state of things, but 
> I'm hoping not to exacerbate the problems I see with the current 
> implementations. I don't want to make it harder to make improvements later. 
> I'd rather move towards a better design, then simply add features to the 
> existing ad-hoc design.
> 
> Josh Elser wrote:
>     bq. It should not, for instance, be setting things that look like 
> passwords, and "create local user", "list local users", and "delete local 
> users" should be unsupported client side operations, just as you've done with 
> "change password" (since users are managed within Kerberos)
>     
>     Thanks for pointing that out. That helps me understand where your 
> concerns are coming from. Perhaps it would be cleaner to just keep a 
> reference to the ZkAuthenticator as a delegate (instead of extending it). 
> This would give us a bit more obvious control over what's happening.
>     
>     bq. I'm thinking the more we can separate the "local user" management 
> functions from the KerberosAuthenticator, the better, since users are managed 
> in Kerberos. Some of my concerns are with the current state of things, but 
> I'm hoping not to exacerbate the problems I see with the current 
> implementations
>     
>     Agreed, those are very.. awkward, presently. Is making sure that we throw 
> exceptions from KerberosAuthenticator when we call these non-sensical methods 
> a sufficient change? Then, we can move to a better design later on like you 
> said?
> 
> Christopher Tubbs wrote:
>     What do you actually need the ZKAuthenticator for? Just to create the 
> directory, so the ZKPermHandler and ZKAuthorizor have a place to put stuff? 
> It might be better to sever completely (vs. delegate or subclass), and just 
> use a common constant for the ZK Path to create the user directories to make 
> those other two components happy (assuming that's all that ZKAuthenticator 
> was needed for).
>     
>     Throwing an exception from KerberosAuthenticator might work... I'm more 
> concerned about what happens on the client side, and these may not map 
> one-to-one. So, a closer inspection/testing will be need to ensure that's 
> sufficient.
>     
>     The important bit about implementing additional Authenticators, is that 
> when a user is deleted in the external system, somebody needs to call the 
> cleanup/dropuser methods on the PermHandler/Authorizor. That will need to be 
> documented. And, it's a bit weird, because the actual interfaces aren't 
> public API (it's one of the reasons why I prefer a single, comprehensive 
> pluggable security module, because it helps consolidate this kind of 
> integration between components).
> 
> Josh Elser wrote:
>     > What do you actually need the ZKAuthenticator for? Just to create the 
> directory, so the ZKPermHandler and ZKAuthorizor have a place to put stuff?
>     
>     Yes. It also lets us know what users currently exist (even though any new 
> user *could* come in at any point). I'm not sure what else the 
> ZKAuthenticator is magically handling for me behind the scenes. I'd need to 
> re-audit it.
>     
>     > Throwing an exception from KerberosAuthenticator might work... I'm more 
> concerned about what happens on the client side, and these may not map 
> one-to-one. So, a closer inspection/testing will be need to ensure that's 
> sufficient.
>     
>     Will do.
>     
>     > The important bit about implementing additional Authenticators, is that 
> when a user is deleted in the external system, somebody needs to call the 
> cleanup/dropuser methods on the PermHandler/Authorizor. That will need to be 
> documented. And, it's a bit weird, because the actual interfaces aren't 
> public API (it's one of the reasons why I prefer a single, comprehensive 
> pluggable security module, because it helps consolidate this kind of 
> integration between components).
>     
>     Noted and understand/agree about the security module. Will be clear as 
> clear as possible in the docs to outline the situation.
> 
> Josh Elser wrote:
>     I cleaned up the logic in KerberosAuthenticator to remove the random 
> password bits that were required from directly using the ZKAuthenticator 
> which did clean things up a little bit. I think leaving in dropUser, 
> listUsers and createUser do each have merits as long as they are properly 
> explained in documentation (as I plan to do today).
>     
>     dropUser allows a simple way for all perms/auths to be reset on a user 
> (even if they might come back because of their Kerberos auths), listUsers can 
> show the users who have actually connected to Accumulo (known by us, not just 
> the KDC) and createUser can allow an admin to seed permissions/authorizations 
> without the user authenticating with the system once (this also prevents us 
> from having to alter how SecurityOperations works in the kerberos case). I 
> still agree that all three of these can be better handled with a more 
> centralized approach and a less password-centric client API to implement.
> 
> Christopher Tubbs wrote:
>     As long as we understand that the client side names for these operations 
> are `(create|drop|list)LocalUser(s?)`, the `Local` part being relevant. As 
> long as the documentation clarifies that these operations are used to manage 
> a "local" view of the users in the remote system, and do not manage users in 
> the KDC, I think it's fine (though a bit awkward).

Addressed this through explicit documentation in the user manual.


> On Dec. 30, 2014, 10:46 p.m., Christopher Tubbs wrote:
> > server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java,
> >  line 46
> > <https://reviews.apache.org/r/29386/diff/4/?file=803164#file803164line46>
> >
> >     Why is this seed fixed?
> 
> Josh Elser wrote:
>     To keep using ZKAuthenticator as a "delegate", I need to put some 
> password into ZK. I don't think it's required that these random strings 
> actually be secure, it's mostly intended as obfuscation. Thoughts?
> 
> Christopher Tubbs wrote:
>     If it's not intended to be secure, why use SecureRandom in the first 
> place? And, because ZK is being used as a delegate, and because these are 
> being used in place of a password, I think they should be secure. If somebody 
> changed from the KerberosAuthenticator back to the default (which is easy to 
> do, if one accidentally clobbers the configuration file when upgrading a 
> system), non-secure passwords could actually be used to authenticate. If you 
> really want to store something which cannot be used to authenticate, you need 
> to replace it with something that cannot parse as a hashed password. This is 
> what happens in /etc/shadow when you disable an account.
>     
>     In any case, I still don't see the need to do anything with the seed. 
> SecureRandom does special stuff with the seed, and I'm not sure it is needed 
> to set it. setSeed doesn't even behave the same way as regular Random does 
> with a fixed seed. It doesn't seem "wrong". It just seems like unnecessary 
> code.
> 
> Josh Elser wrote:
>     Ok, I can just remove the seed, then.

This was ultimately removed when I pulled more implementation into 
KerberosAuthenticator directly, instead of delegating to ZKAuthenticator.


> On Dec. 30, 2014, 10:46 p.m., Christopher Tubbs wrote:
> > shell/src/main/java/org/apache/accumulo/shell/ShellOptionsJC.java, line 210
> > <https://reviews.apache.org/r/29386/diff/4/?file=803175#file803175line210>
> >
> >     Why is username the "short" user name? Is that unique in Kerberos? If 
> > not, the long version should be used everywhere instead. Otherwise, one 
> > user can appear to be another in logs, etc.
> >     
> >     If "getShortUserName" is not unique, it should avoided everywhere.
> 
> Josh Elser wrote:
>     Check out: 
> http://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-user/What-is-a-Kerberos-Principal_003f.html
>     
>     Kerberos principals are of the form: primary/instance@realm. Kerberos 
> principals are typically categorized as users and services. A user is not 
> qualified to a single instance (a host) and represent authentication across 
> the realm. For example, [email protected] means that I can "roam". 
> Conversely, a service is typically "fixed" to a specific host. For example, 
> accumulo/[email protected] means that there is a process, logged 
> in as 'accumulo' on the host 'node1.example.com'. That service can't be run 
> on any other host. Now, an important note if someone actually creates a 
> principal "[email protected]" this is unique with respect to any other 
> "accumulo/`host`@EXAMPLE.COM" principal. I'm not sure if we need to do 
> anything else other than convention of kerberos principals, or if we should 
> be including the instance in "our" username when present.
>     
>     This kind of ties back into the SystemCredentials discussion again.
> 
> Christopher Tubbs wrote:
>     Okay, so a smart configuration would make shortnames unique. However, 
> UserGroupInformation returns only the `primary` for the short name. This 
> means that user names will have to be unique across realms and instances. 
> Right now, you are storing permissions using the short name. So, any user 
> with the same primary, will be able to masquerade as any other user with the 
> same primary from a different instance and/or realm, and be able to user 
> their permissions and authorizations. That's the problem with the shortname 
> here. That's very unexpected.
> 
> Josh Elser wrote:
>     Bingo. If you look at how HDFS does their configuration, this is the same 
> convention. The lack of documentation from me leaves something to be desired 
> here, and I apologize for that.
>     
>     To save you looking at HDFS (if you care not to look), you'll see that an 
> HDFS process uses a given principal with a special replacement string 
> `_HOST`. The common convention is to use something like 
> `dn/[email protected]` (the realm is unimportant for this example). This 
> ensures that the same configuration files can be used across all hosts in the 
> HDFS instance, and Hadoop dynamically replaces `_HOST` with the FQDN of the 
> host. Thus, there's an implicit link that all `dn/*@EXAMPLE.COM` can act as 
> datanodes and this is protected by the fact that access to the KDC is 
> restricted (you can't make your own user). The circle of trust is two-fold: 
> having a keytab with the correct principal and that Hadoop is requires that 
> specific configuration (which restricts the principal).
> 
> Christopher Tubbs wrote:
>     My concerns here are more about the impact on users, than for the system 
> credentials. I don't know what HDFS is doing, but if they aren't (minimally) 
> checking the realm when checking permissions/access on an authenticated 
> principal, then they are less secure than I think we should be. Referencing 
> HDFS also seems to imply that we're not so much doing Kerberos, as we are 
> implementing HDFS-specific Kerberos conventions (which are less secure, with 
> respect to data authorizations/permissions within Accumulo, than I'm 
> comfortable with).
> 
> Josh Elser wrote:
>     bq. if they aren't (minimally) checking the realm when checking 
> permissions/access on an authenticated principal
>     
>     Do you mean the instance instead of the realm? In the case of a single 
> realm, the KDC is going to verify the correct realm. Assuming you meant the 
> instance though (the optional "/hostname"), it's typical that a user has the 
> ability to use their credentials anywhere. Thus, you typically see principals 
> without instances for actual users. As far as I understand it, that's what 
> HDFS tends to follow and what I tried to as well. Accumulo doesn't care where 
> you come from, just what your name is and that you have valid credentials. I 
> don't think we're being substantially less secure by not including the 
> instance in the Accumulo principal.
> 
> Christopher Tubbs wrote:
>     No, I mean the realm, to make it only necessary to guarantee uniqueness 
> within a realm, vs. across all known realms (more reasonable of a guarantee 
> to make for a KDC user admin). We could also include the instance (when 
> specified), if we want to really be careful that users aren't sharing 
> permissions.
>     
>     In my concerns, I'm assuming we authenticate users in any realm. If we 
> are somehow restricted to a single realm (either by a "permittedRealm" 
> configuration item or by the nature of Kerberos itself), then realm isn't 
> that important, but we should discuss more about the instance. My 
> understanding is that Kerberos authenticates the user by the fully qualified 
> Kerberos principal (`primary/instance@realm`) in whatever realm they are, but 
> it doesn't have to be a specific realm (like the same one as the server), and 
> then we are truncating their identity, essentially binning people from 
> different realms into the same bucket. It's like authenticating me as 
> `Christopher Tubbs`, and then assigning me to a bucket called `Christopher` 
> where I share permissions/authorizations with all other `Christopher`s.
> 
> Josh Elser wrote:
>     Oh, I apologize, I follow you now. Your concern wasn't clicking for me.
>     
>     > My understanding is that Kerberos authenticates the user by the fully 
> qualified Kerberos principal (primary/instance@realm) in whatever realm they 
> are, but it doesn't have to be a specific realm (like the same one as the 
> server), and then we are truncating their identity, essentially binning 
> people from different realms into the same bucket
>     
>     Well, the KDC you're communicating with has to be set up for the realm 
> being requested (and if one isn't provided, it will delegate to another KDC 
> or drop you into a default realm, depending on krb5.conf). As I understand 
> it, if you haven't defined a `default_realm` in `libdefaults` in krb5.conf, 
> and a user comes in with an incorrect hostname (instance) or realm 
> specification, the KDC won't authenticate you which keeps them out of 
> Accumulo completely. I use `default_realm` locally, since I just use a dummy 
> realm instead of actually matching my laptop.
>     
>     In all honesty thought, I haven't thought past single-realm KDC setups. 
> Is enforcing that clients are a member of the same realm the Accumulo server 
> principals reside in sufficient? I'm worried about scope-creep of trying to 
> do multi-realm configuration correct before single realm is adequately 
> polished.
> 
> Christopher Tubbs wrote:
>     bq. Is enforcing that clients are a member of the same realm the Accumulo 
> server principals reside in sufficient?
>     
>     Perhaps. Where would we do this? In the site configuration?
>     
>     bq. I'm worried about scope-creep of trying to do multi-realm 
> configuration correct before single realm is adequately polished.
>     
>     Understood, but I'm thinking about it from the other side. I don't want 
> to make assumptions which are valid in a narrow case, but which leave 
> security holes in a more general case. I'm also coming at this from the 
> perspective of dealing with X.509 certificates, and understanding the 
> differences between a CN and a DN.
>     
>     If we lock things down to a single realm (so we can safely omit it in our 
> internal structures), we'd still need to address the `instance` portion. For 
> that, it sounded like you were saying that `myPrimary/myInstance@myRealm` is 
> distinct from `myPrimary@myRealm` and could both be valid users according to 
> the KDC. If that's the case, I think it makes sense for the permissions 
> handler/authorizer to use the `primary/instance` for the principal and not 
> just the `primary` (which is what shortname does), because it could have 
> different permissions. If the user administrator wishes to allow 
> `myPrimary@myRealm`, then they should create such a user in the KDC (I hope 
> I'm understanding this correctly.), so we would just use `myPrimary` as the 
> user principal in Accumulo, but we shouldn't strip the instance off if it is 
> present.
> 
> Josh Elser wrote:
>     > > Is enforcing that clients are a member of the same realm the Accumulo 
> server principals reside in sufficient?
>     > Perhaps. Where would we do this? In the site configuration?
>     
>     Yeah. My thought was to just piggy-back on top of the realm provided in 
> the kerberos principal. That keeps us from having to introduce a new property 
> for something we know that might not be entirely sufficient.
>     
>     > we'd still need to address the instance portion. For that, it sounded 
> like you were saying that myPrimary/myInstance@myRealm is distinct from 
> myPrimary@myRealm and could both be valid users according to the KDC
>     
>     Yes, principals are valid (and distinct!) both with and without an 
> instance. In our case, I believe the instance being distinct is undesirable 
> (and where I was going with the reference to how Hadoop does things). Any 
> server with a given principal (or matching a certain principal) is considered 
> the Accumulo "system" user (along with the `instance.*` check we mentioned 
> earlier). A simple way to do this (without getting into complicated regex's 
> defining who is actually considered the system user) is to just treat any 
> instance also as that user. It brings a bit of coordination required in how 
> KRB principals are created, but it's the "common" configuration/deployment at 
> the cost of flexibility. I would envision leveraging something similar to the 
> `auth_to_local` RULEs 
> (http://web.mit.edu/kerberos/krb5-devel/doc/admin/conf_files/krb5_conf.html) 
> like Hadoop does, but I don't *really* want to do that right now (mapping 
> some set of principal regexs to a "user"). This would let us say th
 ings like "accumulo/node1.example.com" is "accumulo" as is 
"old_server/node2.example.com".
>     
>     For normal users, convention is that they aren't attached to an instance 
> (and are valid within the realm), and this implementation would be a 
> limitation on us for edge cases in KDC configurations.
>     
>     > If the user administrator wishes to allow myPrimary@myRealm, then they 
> should create such a user in the KDC (I hope I'm understanding this 
> correctly.), so we would just use myPrimary as the user principal in 
> Accumulo, but we shouldn't strip the instance off if it is present.
>     
>     Yes, you are correct. One thing I'm confused about is if there is ever a 
> case that a user would have an instance in their principal. Not understanding 
> why this might actually happen pushes me in the direction that truncating 
> things is ok. That covers "human" users, but "application" users would still 
> be likely tied to a specific hostname, in which case perhaps I can't punt on 
> this for now. I really just want to avoid having N `accumulo/hostname` users 
> in our "database" which would the sum of all Accumulo server processes. The 
> regex matching would be needed to avoid that.
>     
>     Maybe this is experimental until I do that as well? Maybe I shouldn't 
> commit any of this without that? I'm not completely decided yet, but I'm 
> erring on the former presently.
> 
> Christopher Tubbs wrote:
>     bq. My thought was to just piggy-back on top of the realm provided in the 
> kerberos principal.
>     
>     You mean the server's own realm? That makes sense to me. We can document 
> that they should match, but we'd need to make sure we explicitly check that.
>     
>     
>     bq. For normal users, convention is that they aren't attached to an 
> instance (and are valid within the realm), and this implementation would be a 
> limitation on us for edge cases in KDC configurations.
>     
>     My concerns here are for normal users. The !SYSTEM user doesn't even have 
> permissions or authorizations stored in ZK (it shouldn't anyway). I had 
> assumed the !SYSTEM user would be treated specially after authentication at 
> the transport layer. I don't think it should rely on the Kerberos principal. 
> This relates to our other discussion about the SystemToken.
>     
>     bq. One thing I'm confused about is if there is ever a case that a user 
> would have an instance in their principal.
>     
>     I can imagine use cases where a user has permission to access a table, 
> but only from a specific, vetted system. This is analogous to OpenStack and 
> EC2 security group / firewall rules which allow access only from specific 
> sources. MySQL also has this concept in its permissions model.
>     
>     bq. That covers "human" users, but "application" users would still be 
> likely tied to a specific hostname, in which case perhaps I can't punt on 
> this for now.
>     
>     Agreed.
>     
>     bq. I really just want to avoid having N accumulo/hostname users in our 
> "database" which would the sum of all Accumulo server processes. The regex 
> matching would be needed to avoid that.
>     
>     I don't think that's the case. The system user doesn't (shouldn't) write 
> to the ZK user database. Its permissions are evaluated separately, and it 
> should never have any authorizations. Rather than regex matching, our 
> discussion around the SystemToken might help resolve this. If the system 
> credentials (!SYSTEM, SystemToken) are left as-is, then you can keep using 
> those internally after the transport layer is finished. I wouldn't use the 
> server's Kerberos principal for the server components. I'd keep using the 
> existing !SYSTEM principal, but only after the server component is verified 
> at the transport layer to actually reflect a server component.
> 
> Josh Elser wrote:
>     > You mean the server's own realm? That makes sense to me. We can 
> document that they should match, but we'd need to make sure we explicitly 
> check that.
>     
>     Precisely. I plan to add that check.
>     
>     > If the system credentials (!SYSTEM, SystemToken) are left as-is, then 
> you can keep using those internally after the transport layer is finished.
>     
>     That's a good point. I was thinking around this, instead of tackling it 
> directly. If we address the SYSTEM case specifically, is there anything else 
> we have to do other than switch the shortUserName to the full name 
> (primary+instance)? I think that would address it.
> 
> Christopher Tubbs wrote:
>     If we're okay with the restriction that it must be in the same realm, 
> then that seems all. Just to be clear, are we really sure we want to have 
> that restriction? It seems like the only reason to restrict it is to avoid 
> including the realm internally (like in the ZK storage). And, it'll be 
> problematic if we decide to permit multi-realm authentication in the future.
>     
>     We could serialize the whole thing, to future proof, but keep the 
> restriction to one realm (for now, until we think through the implications of 
> multi-realm), and conveniently only display without the realm (for example, 
> in the shell, in whoami, etc.). That way, if we do add multi-realm support 
> later (by releasing the restriction), we can keep the shorter names for those 
> in the same realm, and only include the realm when it is different than the 
> server.
> 
> Josh Elser wrote:
>     Hiding the realm is a possibility, I believe I need to think on this 
> and/or look at what other projects have done in this regard. Perhaps I'm just 
> being obstinant in not wanting to include the realm at all, and we should 
> always have it. I'm not sure.
>     
>     I believe that the above decision is also going to impact what we do 
> about multiple realms as well. If we store the full principal, the 
> multi-realm problem goes away. Perhaps that's my sign?
> 
> Christopher Tubbs wrote:
>     I don't know about signs, but it does seem to be the most future proof. 
> I'm just not sure how important multi-realm support is, or whether it would 
> be nice to assume a default realm, or whatever. I just haven't used Kerberos 
> that much to know what makes sense.

Resolved this by moving fully to krb princ == accumulo princ. Introduced more 
problems in that we use ZK for the user DB, but side-stepped it by using base64 
on the principal names.


- Josh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29386/#review66382
-----------------------------------------------------------


On Jan. 8, 2015, 11:52 p.m., Josh Elser wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29386/
> -----------------------------------------------------------
> 
> (Updated Jan. 8, 2015, 11:52 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-2815
>     https://issues.apache.org/jira/browse/ACCUMULO-2815
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> ACCUMULO-2815 Initial support for Kerberos client authentication.
> 
> Leverage SASL transport provided by Thrift which can speak GSSAPI, which 
> Kerberos implements. Introduced...
> 
> * An Accumulo KerberosToken which is an AuthenticationToken to validate users.
> * Custom thrift processor and invocation handler to ensure server RPCs have a 
> valid KRB identity and Accumulo authentication.
> * A KerberosAuthenticator which extends ZKAuthenticator to support Kerberos 
> identities seamlessly.
> * New ClientConf variables to use SASL transport and pass Kerberos server 
> principal
> * Updated ClientOpts and Shell opts to transparently use a KerberosToken when 
> SASL is enabled (no extra client work).
> 
> I believe this is the "bare minimum" for Kerberos support. They are also 
> grossly lacking in unit and integration tests. I believe that I might have 
> somehow broken the client address string in the server (I saw log messages 
> with client: null, but I'm not sure if it's due to these changes or not). A 
> necessary limitation in the Thrift server used is that, like the SSL 
> transport, the SASL transport cannot presently be used with the 
> TFramedTransport, which means none of the [half]async thrift servers will 
> function with this -- we're stuck with the TThreadPoolServer.
> 
> Performed some contrived benchmarks on my laptop (while still using it 
> myself) to get at big-picture view of the performance impact against "normal" 
> operation and Kerberos alone. Each "run" was the duration to ingest 100M 
> records using continuous-ingest, timed with `time`, using 'real'.
> 
> THsHaServer (our default), 6 runs:
> 
> Avg: 10m7.273s (607.273s)
> Min: 9m43.395s
> Max: 10m52.715s
> 
> TThreadPoolServer (no SASL), 5 runs:
> 
> Avg: 11m16.254s (676.254s)
> Min: 10m30.987s
> Max: 12m24.192s
> 
> TThreadPoolServer+SASL/GSSAPI (these changes), 6 runs:
> 
> Avg: 13m17.187s (797.187s)
> Min: 10m52.997s
> Max: 16m0.975s
> 
> The general takeway is that there's about 15% performance degredation in its 
> initial state which is in the realm of what I expected (~10%).
> 
> 
> Diffs
> -----
> 
>   README ad6f2bf 
>   core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java f6ea934 
>   core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java 
> 6fe61a5 
>   core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java 
> e75bec6 
>   core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java 
> f481cc3 
>   core/src/main/java/org/apache/accumulo/core/client/impl/MasterClient.java 
> a9ad8a1 
>   
> core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java
>  6dc846f 
>   
> core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportPool.java
>  5da803b 
>   
> core/src/main/java/org/apache/accumulo/core/client/security/tokens/KerberosToken.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/accumulo/core/conf/Property.java e054a5f 
>   core/src/main/java/org/apache/accumulo/core/rpc/FilterTransport.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/accumulo/core/rpc/TTimeoutTransport.java 
> 6eace77 
>   core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java 09bd6c4 
>   core/src/main/java/org/apache/accumulo/core/rpc/UGIAssumingTransport.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/accumulo/core/rpc/UGIAssumingTransportFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/accumulo/core/security/Credentials.java 
> 525a958 
>   core/src/test/java/org/apache/accumulo/core/cli/TestClientOpts.java ff49bc0 
>   
> core/src/test/java/org/apache/accumulo/core/client/ClientConfigurationTest.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/accumulo/core/conf/ClientConfigurationTest.java 
> 40be70f 
>   
> core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java 
> PRE-CREATION 
>   docs/src/main/asciidoc/accumulo_user_manual.asciidoc ec8e538 
>   docs/src/main/asciidoc/chapters/clients.txt 64f0e55 
>   docs/src/main/asciidoc/chapters/kerberos.txt PRE-CREATION 
>   
> minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
>  27d6b19 
>   
> minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java
>  26c23ed 
>   pom.xml ae188a0 
>   proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java 4b048eb 
>   
> server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java
>  09ae4f4 
>   server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 
> 046cfb5 
>   
> server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java
>  PRE-CREATION 
>   
> server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingWrapper.java
>  PRE-CREATION 
>   server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java 
> 641c0bf 
>   
> server/base/src/main/java/org/apache/accumulo/server/rpc/ThriftServerType.java
>  PRE-CREATION 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java
>  5e81018 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/SecurityUtil.java
>  29e4939 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java
>  a59d57c 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java
>  PRE-CREATION 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthorizor.java
>  PRE-CREATION 
>   
> server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosPermissionHandler.java
>  PRE-CREATION 
>   
> server/base/src/main/java/org/apache/accumulo/server/thrift/UGIAssumingProcessor.java
>  PRE-CREATION 
>   server/base/src/main/java/org/apache/accumulo/server/util/Admin.java 
> ae36f1f 
>   server/base/src/main/java/org/apache/accumulo/server/util/ZooZap.java 
> 7fdbf13 
>   
> server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java
>  PRE-CREATION 
>   
> server/base/src/test/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandlerTest.java
>  PRE-CREATION 
>   
> server/base/src/test/java/org/apache/accumulo/server/rpc/ThriftServerTypeTest.java
>  PRE-CREATION 
>   
> server/base/src/test/java/org/apache/accumulo/server/security/SystemCredentialsTest.java
>  4202a7e 
>   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 
> 93a9a49 
>   
> server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
>  f98721f 
>   
> server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java
>  99558b8 
>   
> server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
>  cad1e01 
>   server/master/src/main/java/org/apache/accumulo/master/Master.java 12195fa 
>   
> server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/trace/Basic.java
>  2d98fed 
>   server/tracer/src/main/java/org/apache/accumulo/tracer/TraceServer.java 
> 7e33300 
>   server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 
> d5c1d2f 
>   
> server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationWorker.java
>  1d20e2b 
>   shell/src/main/java/org/apache/accumulo/shell/Shell.java 58308ff 
>   shell/src/main/java/org/apache/accumulo/shell/ShellOptionsJC.java 8167ef8 
>   shell/src/test/java/org/apache/accumulo/shell/ShellConfigTest.java 0e72c8c 
>   shell/src/test/java/org/apache/accumulo/shell/ShellOptionsJCTest.java 
> PRE-CREATION 
>   test/pom.xml b0a926f 
>   test/src/main/java/org/apache/accumulo/test/functional/ZombieTServer.java 
> eb84533 
>   
> test/src/main/java/org/apache/accumulo/test/performance/thrift/NullTserver.java
>  2ebc2e3 
>   test/src/test/java/org/apache/accumulo/harness/AccumuloClusterIT.java 
> 8f7e1b7 
>   test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java 
> abdb627 
>   test/src/test/java/org/apache/accumulo/harness/SharedMiniClusterIT.java 
> 2380f66 
>   test/src/test/java/org/apache/accumulo/harness/TestingKdc.java PRE-CREATION 
>   
> test/src/test/java/org/apache/accumulo/harness/conf/AccumuloMiniClusterConfiguration.java
>  11b7530 
>   
> test/src/test/java/org/apache/accumulo/server/security/SystemCredentialsIT.java
>  fb71f5f 
>   test/src/test/java/org/apache/accumulo/test/ArbitraryTablePropertiesIT.java 
> aa5c164 
>   test/src/test/java/org/apache/accumulo/test/CleanWalIT.java 1fcd5a4 
>   
> test/src/test/java/org/apache/accumulo/test/functional/BatchScanSplitIT.java 
> 221889b 
>   test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java 
> PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/test/security/KerberosTokenTest.java 
> PRE-CREATION 
>   test/src/test/resources/log4j.properties cb35840 
> 
> Diff: https://reviews.apache.org/r/29386/diff/
> 
> 
> Testing
> -------
> 
> Ensure existing unit tests still function. Accumulo is functional and ran 
> continuous ingest multiple times using a client with only a Kerberos identity 
> (no user/password provided). Used MIT Kerberos with Apache Hadoop 2.6.0 and 
> Apache ZooKeeper 3.4.5.
> 
> 
> Thanks,
> 
> Josh Elser
> 
>

Reply via email to