Re: [RFE] Support MIT Kerberos localauth plugin API
The big question is whether or not Java’s implementation of Kerberos supports it. If so, which JDK release. Java’s implementation tends to run a bit behind MIT. Additionally, there is a general reluctance to move Hadoop’s baseline Java version to something even supported until user outcry demands it. So I’d expect support to be a long way off. It’s worth noting that trunk exposes the hadoop kerbname command to help out with auth_to_local mapping, BTW. On Feb 23, 2015, at 2:12 AM, Sunny Cheung wrote: > Hi Hadoop Common developers, > > I am writing to seek your opinion about a feature request: support MIT > Kerberos localauth plugin API [1]. > > Hadoop currently provides the hadoop.security.auth_to_local setting to map > Kerberos principal to OS user account [2][3]. However, the regex-based > mappings (which mimics krb5.conf auth_to_local) could be difficult to use in > complex scenarios. Therefore, MIT Kerberos 1.12 added a plugin interface to > control krb5_aname_to_localname and krb5_kuserok behavior. And system daemon > SSSD (RHEL/Fedora) has already implemented a plugin to leverage this feature > [4]. > > Is that possible for Hadoop to support a plugin API similar to localauth > (when Kerberos security is enabled)? Thanks. > > References: > [1] Local authorization interface (localauth) > http://web.mit.edu/kerberos/krb5-1.12/doc/plugindev/localauth.html > [2] Hadoop in Secure Mode - Mapping from Kerberos principal to OS user account > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html#Mapping_from_Kerberos_principal_to_OS_user_account > [3] Need mapping from long principal names to local OS user names > https://issues.apache.org/jira/browse/HADOOP-6526 > [4] Allow Kerberos Principals in getpwnam() calls > https://fedorahosted.org/sssd/wiki/DesignDocs/NSSWithKerberosPrincipal
RE: [RFE] Support MIT Kerberos localauth plugin API
Sorry I was not clear enough about the problem. Let me explain more here.
Our problem is that normal user principal names can be very different from
their Unix login. Some customers simply have arbitrary mapping between their
Kerberos principals and Unix user accounts. For example, one customer has over
200K users on AD with Kerberos principals in format ".@REALM" (e.g. john@example.com). But their Unix names are in format
"user" or just "" (e.g. user123456, 123456).
So, when Kerberos security is enabled on Hadoop clusters, how should we
configure to authenticate these users from Hadoop clients?
The current way is to use the hadoop.security.auth_to_local setting, e.g. from
core-site.xml:
hadoop.security.auth_to_local
RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](hm@.*EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](rs@.*EXAMPLE.COM)s/.*/hbase/
DEFAULT
The mapping from kerberos principal names
to local OS user names.
These name translation rules can handle cases like mapping service accounts'
principals (e.g. nn/@REALM or dn/@REALM to hdfs). But that is not
scalable for normal users. There are just too many users to handle (as compared
to the finite amount of service accounts).
Therefore, we would like to ask if alternative name resolution plugin interface
can be supported by Hadoop. It could be similar to the way alternative
authentication plugin is supported for HTTP web-consoles [1]:
hadoop.http.authentication.type
org.my.subclass.of.AltKerberosAuthenticationHandler
And the plugin interface can be as simple as this function (error handling
ignored here):
String auth_to_local (String krb5Principal)
{
...
return unixName;
}
If this plugin interface is supported by Hadoop, then everyone can provide a
plugin to support arbitrary mapping. This will be extremely useful when
administrators need to tighten security on Hadoop with existing Kerberos
infrastructure.
References:
[1] Authentication for Hadoop HTTP web-consoles
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/HttpAuthentication.html
-Original Message-
From: Allen Wittenauer [mailto:a...@altiscale.com]
Sent: Tuesday, February 24, 2015 12:47 AM
To: common-dev@hadoop.apache.org
Subject: Re: [RFE] Support MIT Kerberos localauth plugin API
The big question is whether or not Java's implementation of Kerberos
supports it. If so, which JDK release. Java's implementation tends to run a
bit behind MIT. Additionally, there is a general reluctance to move Hadoop's
baseline Java version to something even supported until user outcry demands it.
So I'd expect support to be a long way off.
It's worth noting that trunk exposes the hadoop kerbname command to
help out with auth_to_local mapping, BTW.
On Feb 23, 2015, at 2:12 AM, Sunny Cheung wrote:
> Hi Hadoop Common developers,
>
> I am writing to seek your opinion about a feature request: support MIT
> Kerberos localauth plugin API [1].
>
> Hadoop currently provides the hadoop.security.auth_to_local setting to map
> Kerberos principal to OS user account [2][3]. However, the regex-based
> mappings (which mimics krb5.conf auth_to_local) could be difficult to use in
> complex scenarios. Therefore, MIT Kerberos 1.12 added a plugin interface to
> control krb5_aname_to_localname and krb5_kuserok behavior. And system daemon
> SSSD (RHEL/Fedora) has already implemented a plugin to leverage this feature
> [4].
>
> Is that possible for Hadoop to support a plugin API similar to localauth
> (when Kerberos security is enabled)? Thanks.
>
> References:
> [1] Local authorization interface (localauth)
> http://web.mit.edu/kerberos/krb5-1.12/doc/plugindev/localauth.html
> [2] Hadoop in Secure Mode - Mapping from Kerberos principal to OS user
> account
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-commo
> n/SecureMode.html#Mapping_from_Kerberos_principal_to_OS_user_account
> [3] Need mapping from long principal names to local OS user names
> https://issues.apache.org/jira/browse/HADOOP-6526
> [4] Allow Kerberos Principals in getpwnam() calls
> https://fedorahosted.org/sssd/wiki/DesignDocs/NSSWithKerberosPrincipal
RE: [RFE] Support MIT Kerberos localauth plugin API
Hello Leo/Liou
>> And the plugin interface can be as simple as this function (error handling
>> ignored here) ...
I thought it's good to have the pluggable allowing to customize the method how
to perform the mapping.
You could open a JIRA for this. If you'd like to work on it and need help,
please feel free to ask (me or the community), or discuss in the JIRA, as the
community do.
With the pluggable interface, you could provide a native implementation
leveraging the MIT localauth plugin via JNI, just as it's done for user groups
mapping provider.
If you're looking for something pure in Java, as Allen said, the localauth
plugin support isn't available in JRE as Java would not be so quick to catch up
with latest Kerberos features.
One possibility would be to leverage Apache Kerby, you can fire an issue
request there and let's see how it works out then.
https://issues.apache.org/jira/browse/DIRKRB-102
Regards,
Kai
-Original Message-
From: Sunny Cheung [mailto:sunny.che...@centrify.com]
Sent: Thursday, March 05, 2015 3:42 PM
To: common-dev@hadoop.apache.org
Cc: Leo Liou
Subject: RE: [RFE] Support MIT Kerberos localauth plugin API
Sorry I was not clear enough about the problem. Let me explain more here.
Our problem is that normal user principal names can be very different from
their Unix login. Some customers simply have arbitrary mapping between their
Kerberos principals and Unix user accounts. For example, one customer has over
200K users on AD with Kerberos principals in format ".@REALM" (e.g. john@example.com). But their Unix names are in format
"user" or just "" (e.g. user123456, 123456).
So, when Kerberos security is enabled on Hadoop clusters, how should we
configure to authenticate these users from Hadoop clients?
The current way is to use the hadoop.security.auth_to_local setting, e.g. from
core-site.xml:
hadoop.security.auth_to_local
RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](hm@.*EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](rs@.*EXAMPLE.COM)s/.*/hbase/
DEFAULT
The mapping from kerberos principal names
to local OS user names.
These name translation rules can handle cases like mapping service accounts'
principals (e.g. nn/@REALM or dn/@REALM to hdfs). But that is not
scalable for normal users. There are just too many users to handle (as compared
to the finite amount of service accounts).
Therefore, we would like to ask if alternative name resolution plugin interface
can be supported by Hadoop. It could be similar to the way alternative
authentication plugin is supported for HTTP web-consoles [1]:
hadoop.http.authentication.type
org.my.subclass.of.AltKerberosAuthenticationHandler
And the plugin interface can be as simple as this function (error handling
ignored here):
String auth_to_local (String krb5Principal) {
...
return unixName;
}
If this plugin interface is supported by Hadoop, then everyone can provide a
plugin to support arbitrary mapping. This will be extremely useful when
administrators need to tighten security on Hadoop with existing Kerberos
infrastructure.
References:
[1] Authentication for Hadoop HTTP web-consoles
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/HttpAuthentication.html
-Original Message-
From: Allen Wittenauer [mailto:a...@altiscale.com]
Sent: Tuesday, February 24, 2015 12:47 AM
To: common-dev@hadoop.apache.org
Subject: Re: [RFE] Support MIT Kerberos localauth plugin API
The big question is whether or not Java's implementation of Kerberos
supports it. If so, which JDK release. Java's implementation tends to run a
bit behind MIT. Additionally, there is a general reluctance to move Hadoop's
baseline Java version to something even supported until user outcry demands it.
So I'd expect support to be a long way off.
It's worth noting that trunk exposes the hadoop kerbname command to
help out with auth_to_local mapping, BTW.
On Feb 23, 2015, at 2:12 AM, Sunny Cheung wrote:
> Hi Hadoop Common developers,
>
> I am writing to seek your opinion about a feature request: support MIT
> Kerberos localauth plugin API [1].
>
> Hadoop currently provides the hadoop.security.auth_to_local setting to map
> Kerberos principal to OS user account [2][3]. However, the regex-based
> mappings (which mimics krb5.conf auth_to_local) could be difficult to use in
> complex scenarios. Therefore, MIT Kerberos 1.12 added a plugin interface to
> control krb5_aname_to_localname and krb5_kuserok behavior. And system daemon
> SSSD (RHEL/Fedora) has already implemented a plugin to leverage this feature
> [4].
>
> Is th
RE: [RFE] Support MIT Kerberos localauth plugin API
Thanks. Issue HADOOP-11683 created. Let's further discuss there.
Yes we are looking for a generic way to customize name translation in Hadoop.
With the pluggable interface, organizations can implement their own plugins.
Your idea about "leveraging the MIT localauth plugin via JNI" is good too, in a
sense that organizations can use their existing MIT localauth plugins for both
Hadoop and system authentication.
-Original Message-
From: Zheng, Kai [mailto:kai.zh...@intel.com]
Sent: Thursday, March 05, 2015 4:16 PM
To: common-dev@hadoop.apache.org
Cc: Leo Liou
Subject: RE: [RFE] Support MIT Kerberos localauth plugin API
Hello Leo/Liou
>> And the plugin interface can be as simple as this function (error handling
>> ignored here) ...
I thought it's good to have the pluggable allowing to customize the method how
to perform the mapping.
You could open a JIRA for this. If you'd like to work on it and need help,
please feel free to ask (me or the community), or discuss in the JIRA, as the
community do.
With the pluggable interface, you could provide a native implementation
leveraging the MIT localauth plugin via JNI, just as it's done for user groups
mapping provider.
If you're looking for something pure in Java, as Allen said, the localauth
plugin support isn't available in JRE as Java would not be so quick to catch up
with latest Kerberos features.
One possibility would be to leverage Apache Kerby, you can fire an issue
request there and let's see how it works out then.
https://issues.apache.org/jira/browse/DIRKRB-102
Regards,
Kai
-Original Message-
From: Sunny Cheung [mailto:sunny.che...@centrify.com]
Sent: Thursday, March 05, 2015 3:42 PM
To: common-dev@hadoop.apache.org
Cc: Leo Liou
Subject: RE: [RFE] Support MIT Kerberos localauth plugin API
Sorry I was not clear enough about the problem. Let me explain more here.
Our problem is that normal user principal names can be very different from
their Unix login. Some customers simply have arbitrary mapping between their
Kerberos principals and Unix user accounts. For example, one customer has over
200K users on AD with Kerberos principals in format ".@REALM" (e.g. john@example.com). But their Unix names are in format
"user" or just "" (e.g. user123456, 123456).
So, when Kerberos security is enabled on Hadoop clusters, how should we
configure to authenticate these users from Hadoop clients?
The current way is to use the hadoop.security.auth_to_local setting, e.g. from
core-site.xml:
hadoop.security.auth_to_local
RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/mapred/
RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/hdfs/
RULE:[2:$1@$0](hm@.*EXAMPLE.COM)s/.*/hbase/
RULE:[2:$1@$0](rs@.*EXAMPLE.COM)s/.*/hbase/
DEFAULT
The mapping from kerberos principal names
to local OS user names.
These name translation rules can handle cases like mapping service accounts'
principals (e.g. nn/@REALM or dn/@REALM to hdfs). But that is not
scalable for normal users. There are just too many users to handle (as compared
to the finite amount of service accounts).
Therefore, we would like to ask if alternative name resolution plugin interface
can be supported by Hadoop. It could be similar to the way alternative
authentication plugin is supported for HTTP web-consoles [1]:
hadoop.http.authentication.type
org.my.subclass.of.AltKerberosAuthenticationHandler
And the plugin interface can be as simple as this function (error handling
ignored here):
String auth_to_local (String krb5Principal) {
...
return unixName;
}
If this plugin interface is supported by Hadoop, then everyone can provide a
plugin to support arbitrary mapping. This will be extremely useful when
administrators need to tighten security on Hadoop with existing Kerberos
infrastructure.
References:
[1] Authentication for Hadoop HTTP web-consoles
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/HttpAuthentication.html
-Original Message-
From: Allen Wittenauer [mailto:a...@altiscale.com]
Sent: Tuesday, February 24, 2015 12:47 AM
To: common-dev@hadoop.apache.org
Subject: Re: [RFE] Support MIT Kerberos localauth plugin API
The big question is whether or not Java's implementation of Kerberos
supports it. If so, which JDK release. Java's implementation tends to run a
bit behind MIT. Additionally, there is a general reluctance to move Hadoop's
baseline Java version to something even supported until user outcry demands it.
So I'd expect support to be a long way off.
It's worth noting that trunk exposes the hadoop kerbname command to
help out with auth_to_local mapping, BTW.
On Feb 23, 2015, at 2:12 AM, Sunny Cheung wrote:
> Hi Hadoop Common dev
