Re: hadoop.job.ugi backwards compatibility

[Owen O'Malley]

Moving the discussion over to the more appropriate mapreduce-dev.

On Mon, Sep 13, 2010 at 9:08 AM, Todd Lipcon  wrote:

> 1) Groups resolution happens on the server side, where it used to happen on
> the client. Thus, all Hadoop users must exist on the NN/JT machines in order
> for group mapping to succeed (or the user must write a custom group mapper).

There is a plugin that performs the group lookup. See HADOOP-4656.
There is no requirement for having the user accounts on the NN/JT
although that is the easiest approach. It is not recommended that the
users be allowed to login.

I think it is important that turning security on and off doesn't
drastically change the semantics or protocols. That will become much
much harder to support downstream.

> 2) The hadoop.job.ugi parameter is ignored - instead the user has to use the
> new UGI.createRemoteUser("foo").doAs() API, even in simple security.

User code that counts on hadoop.job.ugi working will be horribly
broken once you turn on security. Turning on and off security should
not involve testing all of your applications. It is unfortunate that
we ever used the configuration value as the user, but continuing to
support it will make our user's code much much more brittle.

-- Owen

hadoop.job.ugi backwards compatibility

[Todd Lipcon]

Hi all,

I wanted to start a (hopefully short) discussion around the treatment of the
hadoop.job.ugi configuration in Hadoop 0.22 and beyond (as well as the
secure 0.20 branch). In the current security implementation, the following
incompatible changes have been made even for users who are sticking with
"simple" security.

1) Groups resolution happens on the server side, where it used to happen on
the client. Thus, all Hadoop users must exist on the NN/JT machines in order
for group mapping to succeed (or the user must write a custom group mapper).
2) The hadoop.job.ugi parameter is ignored - instead the user has to use the
new UGI.createRemoteUser("foo").doAs() API, even in simple security.

I'm curious whether the general user community feels these are acceptable
breaking changes. The potential solutions I can see are:

For 1) Add a configuration like hadoop.security.simple.groupmappinglocation
-> "client" or "server". If it's set to "client", the group mapping would
continue to happen as it does in prior versions on the client side.
For 2) If security is "simple", we can have the FileSystem and JobClient
constructors check for this parameter. If it's set, and there is no Subject
object associated with the current AccessControlContext, wrap the creation
of the RPC proxy with the correct doAs() call.

Although security is obviously an absolute necessity for many organizations,
I know of a lot of people who have small clusters and small teams who don't
have any plans to deploy it. For these people, I imagine the above
backward-compatibility layer may be very helpful as they adopt the next
releases of Hadoop. If we don't want to support these options going forward,
we can of course emit deprecation warnings when they are in effect and
remove the compatibility layer in the next major release.

Any thoughts here? Do people often make use of the hadoop.job.ugi variable
to such an extent that this breaking change would block your organization
from upgrading?

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera