[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981303#comment-14981303 ] Jun Rao commented on KAFKA-2675: [~sriharsha], could you explain a bit how cross realm connection works? In Sasl.createSaslClient(), it seems that it only takes the service name (protocol), but not the realm of the service. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978878#comment-14978878 ] ASF GitHub Bot commented on KAFKA-2675: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/376 > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978585#comment-14978585 ] Ismael Juma commented on KAFKA-2675: The PR fixes an issue with existing tests, but I didn't add any new ones. Due to the fact that we are relying on a system property to set the JAAS file with the principals, it seems tricky to set-up the test in a way where authentication fails after the broker and clients are initialised (it's easy to make it fail during initialisation as the `login` call fails if the principals are not set-up correctly). For now, it seems like system tests will be easier for these cases. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978531#comment-14978531 ] ASF GitHub Bot commented on KAFKA-2675: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/376 KAFKA-2675; SASL/Kerberos follow up You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka KAFKA-2675-sasl-kerberos-follow-up Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/376.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #376 commit 2e928f4705d731aa2754686651a65b0713d840b8 Author: Ismael Juma Date: 2015-10-23T08:30:46Z Fix handling of `kafka_jaas.conf` not found in `SaslTestHarness` commit 6aed268b3ff772bebb38c8f242ad63cca6ba83b6 Author: Ismael Juma Date: 2015-10-23T08:35:29Z Remove unnecessary code in `BaseProducerSendTest` In most cases, `producer` can never be null. In two cases, there are multiple producers and the `var producer` doesn't make sense. commit f0cc13190c0374cf72f76040a52a66bace950ef7 Author: Ismael Juma Date: 2015-10-23T09:09:05Z Move some tests from `BaseConsumerTest` to `PlaintextConsumerTest` in order to reduce build times commit 8f1aa28fda00820b14f519fd7df457ef7804c634 Author: Ismael Juma Date: 2015-10-23T09:10:39Z Make `Login` thread a daemon thread This way, it won't prevent shutdown if `close` is not called on `Consumer` or `Producer`. commit f9a3e4e1c918c92771403cabca089092c36c1638 Author: Ismael Juma Date: 2015-10-23T16:59:43Z Rename `kafka.security.auth.to.local` to `sasl.kerberos.principal.to.local.rules` Also improve wording for `SaslConfigs` docs. commit ac58906f4c446941c43f193aaee45366dfd50950 Author: Ismael Juma Date: 2015-10-26T09:52:20Z Remove unused `SASL_KAFKA_SERVER_REALM` property commit c68554f4001979ca9283f007e20fe599c1eb85fa Author: Ismael Juma Date: 2015-10-26T12:59:38Z Remove forced reload of `Configuration` from `Login` and set JAAS property before starting `MiniKdc` commit 503e2662a63bd39a1602ed73cba9b2c8fe4af55f Author: Ismael Juma Date: 2015-10-27T21:22:59Z Fix `IntegrationTestHarness` to set security configs correctly commit 133076603671c50c4ab820f754c6ebaaedc58f15 Author: Ismael Juma Date: 2015-10-27T23:27:49Z Improve logging in `ControllerChannelManager` by using `brokerNode` instead of `toBroker` commit 7dd7eeff4748b28f31010196c8fbb2cb65d0da0e Author: Ismael Juma Date: 2015-10-28T14:36:30Z Introduce `LoginManager.closeAll()` and use it in `SaslTestHarness` This is necessary to avoid authentication failures when consumers, producers or brokers are leaked during tests. commit 0f31db82a07b4be77cd2d95cf9d2f9eecd1343ee Author: Ismael Juma Date: 2015-10-28T14:37:42Z Improve exception handling in Sasl authenticators: avoid excessive exception chaining > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to principal in JAAS file > not being present in MiniKDC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976393#comment-14976393 ] Ismael Juma commented on KAFKA-2675: I asked [~fpj] to validate my reasoning above and he agreed. So, I'm going to proceed and remove the unused SASL_KAFKA_SERVER_REALM config in the PR for this JIRA. We can add it back when we have a concrete proposal on how to use it. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to invalid user > 6. Write test that shows authentication failure due to wrong password > 7. Write test that shows authentication failure due ticket expiring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[
https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973366#comment-14973366
]
Ismael Juma commented on KAFKA-2675:
[~harsha_ch], I had a look at implementing SASL_KAFA_SERVER_REALM and it's
unclear how this is meant to be used. Let's use the ZooKeeper code (which is
quite similar to ours) as an example since it has the equivalent
`zookeeper.server.realm` property:
{code}
final KerberosName clientKerberosName = new
KerberosName(clientPrincipal.getName());
// assume that server and client are in the same realm (by
default; unless the system property
// "zookeeper.server.realm" is set).
String serverRealm =
System.getProperty("zookeeper.server.realm",clientKerberosName.getRealm());
KerberosName serviceKerberosName = new
KerberosName(servicePrincipal+"@"+serverRealm);
final String serviceName = serviceKerberosName.getServiceName();
final String serviceHostname =
serviceKerberosName.getHostName();
final String clientPrincipalName =
clientKerberosName.toString();
try {
saslClient = Subject.doAs(subject,new
PrivilegedExceptionAction() {
public SaslClient run() throws SaslException {
LOG.info("Client will use GSSAPI as SASL
mechanism.");
String[] mechs = {"GSSAPI"};
LOG.debug("creating sasl client:
client="+clientPrincipalName+";service="+serviceName+";serviceHostname="+serviceHostname);
SaslClient saslClient =
Sasl.createSaslClient(mechs,clientPrincipalName,serviceName,serviceHostname,null,new
ClientCallbackHandler(null));
return saslClient;
}
});
return saslClient;
}
{code}
So, the server realm is used to create the `KerberosName`, but then it seems to
be ignored and only `getServiceName()` and `getHostName()` are used. It seems
to me that one could use any value for `serverRealm` and it would have no
effect on the parameters passed to `Sasl.createSaslClient`.
Am I missing something?
> SASL/Kerberos follow-up
> ---
>
> Key: KAFKA-2675
> URL: https://issues.apache.org/jira/browse/KAFKA-2675
> Project: Kafka
> Issue Type: Sub-task
> Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> This is a follow-up to KAFKA-1686.
> 1. Decide on `serviceName` configuration: do we want to keep it in two places?
> 2. auth.to.local config name is a bit opaque, is there a better one?
> 3. Implement or remove SASL_KAFKA_SERVER_REALM config
> 4. Consider making Login's thread a daemon thread
> 5. Write test that shows authentication failure due to invalid user
> 6. Write test that shows authentication failure due to wrong password
> 7. Write test that shows authentication failure due ticket expiring
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971271#comment-14971271 ] Ismael Juma commented on KAFKA-2675: Thanks [~harsha_ch]. You mentioned that many projects use `serviceName` in the JAAS file (even though it's a non-standard JAAS config and causes the IBM JVM to fail). It would be good if you could elaborate on this some more for our benefit. ZooKeeper uses a system property (zookeeper.sasl.client.username) and there is no mention of serviceName in the Hadoop Security book (which makes sense since it doesn't use JAAS for configuration). I did see serviceName mentioned in the HDP documentation for a few projects (including Kafka), but it wasn't clear to me if this was a HDP enhancement or an upstream feature (it is an enhancement for Kafka, of course). Regarding SASL_KAFKA_SERVER_REALM, that makes sense. I think I know where this needs to be changed and will include the change in the PR for this JIRA. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to invalid user > 6. Write test that shows authentication failure due to wrong password > 7. Write test that shows authentication failure due ticket expiring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971107#comment-14971107 ] Sriharsha Chintalapani commented on KAFKA-2675: --- 1. Decide on `serviceName` configuration: do we want to keep it in two places? We should keep this in two places. Configuring serviceName in jaas file as been the way to go in all other projects. We only kept in two places because of IBM jdk. 3. Implement or remove SASL_KAFKA_SERVER_REALM config This is required on the client side. Its very common scenario where server/broker in one relam and clients are in another . In this case clients needs to configure the server realm. By default we use clients realm to connect to server. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to invalid user > 6. Write test that shows authentication failure due to wrong password > 7. Write test that shows authentication failure due ticket expiring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970115#comment-14970115 ] Jun Rao commented on KAFKA-2675: For #4, making the login thread a daemon thread is also fine. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to invalid user > 6. Write test that shows authentication failure due to wrong password > 7. Write test that shows authentication failure due ticket expiring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2675) SASL/Kerberos follow-up
[ https://issues.apache.org/jira/browse/KAFKA-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969096#comment-14969096 ] Ismael Juma commented on KAFKA-2675: Some input would be appreciated on the following: 1. Decide on `serviceName` configuration: do we want to keep it in two places? This is the only config we are storing in 2 places (all the other ones are either in the Kafka config or the JAAS file). We need to have it in the Kafka config because the IBM JDK fails if it sees this property in the JAAS file. Given this, I think we need strong justification to support setting this in the JAAS file. [~harsha_ch], you said that the reason to have it in the JAAS file is that many other projects do that. Do you have a reference for the projects that do it? 2. auth.to.local config name is a bit opaque, is there a better one? The full config name is kafka.security.auth.to.local, and it's similar to the Hadoop system property: http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-3-1/CDH4-Security-Guide/cdh4sg_topic_19.html Other options that I can think of could be: principal.to.local principal.to.local.rules auth.to.local.rules A reason to have `rules` in the name is that we could support the definition of a class to do the mapping in the future. Hortonworks supports this via a `principal.to.local.class` property: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_secure-kafka-ambari/content/ch_secure-kafka-config-options.html 3. Implement or remove SASL_KAFKA_SERVER_REALM config This config is not used at the moment. The documentation says "Default will be from kafka jaas config". Maybe we can remove it then? [~harsha_ch], your input is appreciated. 4. Consider making Login's thread a daemon thread Rajini asked if this should be a daemon thread. It was a daemon thread to start with, but [~junrao] suggested that it perhaps should be non-daemon because we wait for it during shutdown. [~rsivaram]'s concern is that we may prevent clients from exiting normally. This won't happen if `KafkaConsumer.close()` is called, but it can happen if `close()` is not called. It seems like making it a daemon thread is the safest option. What do you think [~junrao]? It would be very helpful to get your feedback so that we can wrap the SASL/Kerberos feature. > SASL/Kerberos follow-up > --- > > Key: KAFKA-2675 > URL: https://issues.apache.org/jira/browse/KAFKA-2675 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.9.0.0 > > > This is a follow-up to KAFKA-1686. > 1. Decide on `serviceName` configuration: do we want to keep it in two places? > 2. auth.to.local config name is a bit opaque, is there a better one? > 3. Implement or remove SASL_KAFKA_SERVER_REALM config > 4. Consider making Login's thread a daemon thread > 5. Write test that shows authentication failure due to invalid user > 6. Write test that shows authentication failure due to wrong password > 7. Write test that shows authentication failure due ticket expiring -- This message was sent by Atlassian JIRA (v6.3.4#6332)
