[
https://issues.apache.org/jira/browse/KAFKA-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex closed KAFKA-3668.
-----------------------
> Unable to authenticate Kafka broker to secured Zookeeper
> --------------------------------------------------------
>
> Key: KAFKA-3668
> URL: https://issues.apache.org/jira/browse/KAFKA-3668
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0, 0.9.0.1
> Environment: Red Hat Enterprise Linux Server release 7.0 (Maipo)
> Java 1.8.0_66-b17
> Kafka 0.9.0.0 and 0.9.0.1
> Reporter: Alex
> Fix For: 0.9.0.0, 0.9.0.1
>
>
> Hello,
> we are running into trouble when trying to connect Kafka broker to secured
> Zookeeper, Kerberos protected.
> Configuration is as simple as possible: 1 Zookeeper, 1 Kafka broker and
> Kerberos. All running on local machine.
> Zookeeper successfully starts and receives TGT from Kerberos AS_REQ. Then
> Kafka broker obtains TGT from AS_REQ, but it is unable to get TGS from
> TGS_REQ because <unknown server> as krb5kdc.log shows:
> krb5kdc.log
> ...
> May 06 17:41:42 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4
> etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545702, etypes
> {rep=18 tkt=18 ses=18}, zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for
> krbtgt/ca.sbrf...@ca.sbrf.ru
> May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4
> etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545864, etypes
> {rep=18 tkt=18 ses=18}, kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for
> krbtgt/ca.sbrf...@ca.sbrf.ru
> May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): TGS_REQ (4
> etypes {18 17 16 23}) 10.116.93.88: LOOKING_UP_SERVER: authtime 0,
> kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for <unknown server>, Server not
> found in Kerberos database
> What is the possible reason of this problem?
> KAFKA CONFIG:
>
> zookeeper.properties
> dataDir=/tmp/zookeeper
> clientPort=2181
>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
> jaasLoginRenew=3600000
>
> server.properties
> broker.id=0
> log.dirs=/tmp/kafka-logs
> listeners=SASL_PLAINTEXT://10.116.93.88:9092
> security.inter.broker.protocol=SASL_PLAINTEXT
> zookeeper.connect=10.116.93.88:2181
> sasl.kerberos.service.name=kafka
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> zookeeper.set.acl=true
> #allow.everyone.if.no.acl.found=true
> #sasl.enabled.mechanisms=GSSAPI
> #sasl.mechanism.inter.broker.protocol=GSSAPI
> JVM params:
> Kafka:
> -Djava.security.krb5.conf=/etc/krb5.conf
>
> -Djava.security.auth.login.config=config/kafka-broker-jaas.conf
> Zookeeper:
> -Djava.security.krb5.conf=/etc/krb5.conf
> -Djava.security.auth.login.config=config/zookeeper.conf
>
> JAAS files:
> kafka-broker-jaas.conf:
> KafkaServer {
> com.sun.security.auth.module.Krb5LoginModule
> required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka.keytab"
> debug=true
> useTicketCache=false
>
> principal="kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
> };
> Client {
> com.sun.security.auth.module.Krb5LoginModule
> required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka.keytab"
> debug=true
> useTicketCache=false
>
> principal="kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
> };
> zookeeper-jaas.conf
> Server {
> com.sun.security.auth.module.Krb5LoginModule
> required
> useKeyTab=true
> keyTab="/etc/security/keytabs/zookeeper.keytab"
> storeKey=true
> useTicketCache=false
> debug=true
>
> principal="zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
> };
> KERBEROS 5 CONFIG:
> krb5.conf
> [logging]
> default = FILE:/var/log/krb5libs.log
> kdc = FILE:/var/log/krb5kdc.log
> admin_server = FILE:/var/log/kadmind.log
> [libdefaults]
> dns_lookup_realm = false
> ticket_lifetime = 24h
> renew_lifetime = 7d
> forwardable = true
> rdns = false
> default_realm = CA.SBRF.RU
> default_ccache_name = KEYRING:persistent:%{uid}
> [realms]
> CA.SBRF.RU = {
> kdc = SBT-IPO-204.ca.sbrf.ru
> admin_server = SBT-IPO-204.ca.sbrf.ru
> }
> [domain_realm]
> .ca.sbrf.ru = CA.SBRF.RU
> ca.sbrf.ru = CA.SBRF.RU
> kdc.conf
> [kdcdefaults]
> kdc_ports = 88
> kdc_tcp_ports = 88
> [realms]
> CA.SBRF.RU = {
> #master_key_type = aes256-cts
> acl_file = /var/kerberos/krb5kdc/kadm5.acl
> dict_file = /usr/share/dict/words
> admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
> supported_enctypes = aes256-cts:normal aes128-cts:normal
> des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal
> camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal
> des-cbc-crc:normal
> }
> kadm.conf
> */ad...@ca.sbrf.ru *
> LOGS:
> Zookeeper: bin/zookeeper-server-start.sh -daemon
> config/zookeeper.properties
> ...
> [2016-05-06 17:41:42,750] INFO minSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:41:42,750] INFO maxSessionTimeout set to -1
> (org.apache.zookeeper.server.ZooKeeperServer)
> Debug is true storeKey true useTicketCache false useKeyTab
> true doNotPrompt false ticketCache is null isInitiator true KeyTab is
> /etc/security/keytabs/zookeeper.keytab refreshKrb5Config is false principal
> is zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru tryFirstPass is false
> useFirstPass is false storePass is false clearPass is false
> principal is zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru
> Will use keytab
> Commit Succeeded
> [2016-05-06 17:41:43,137] INFO successfully logged in.
> (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,143] INFO TGT refresh thread started.
> (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,150] INFO binding to port
> 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-05-06 17:41:43,169] INFO TGT valid starting at:
> Fri May 06 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,170] INFO TGT expires:
> Sat May 07 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
> [2016-05-06 17:41:43,170] INFO TGT refresh sleeping until: Sat
> May 07 14:04:31 MSK 2016 (org.apache.zookeeper.Login)
>
> ...Here Kafka starts...
> [2016-05-06 17:44:24,933] INFO Accepted socket connection from
> /10.116.93.88:58825 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-05-06 17:44:24,952] ERROR Zookeeper Server failed to
> create a SaslServer to interact with a client during session initiation:
> javax.security.sasl.SaslException: Failure to initialize security context
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)]
> (org.apache.zookeeper.server.ZooKeeperSaslServer)
> javax.security.sasl.SaslException: Failure to initialize
> security context [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos credentails)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(Unknown Source)
> at
> com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(Unknown Source)
> at javax.security.sasl.Sasl.createSaslServer(Unknown
> Source)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
> at
> org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
> at
> org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
> at java.lang.Thread.run(Unknown Source)
> Caused by: GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos credentails)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
> at
> sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)
> at sun.security.jgss.GSSCredentialImpl.add(Unknown
> Source)
> at sun.security.jgss.GSSCredentialImpl.<init>(Unknown
> Source)
> at
> sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)
> ... 13 more
> [2016-05-06 17:44:24,961] INFO Client attempting to establish
> new session at /10.116.93.88:58825
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:24,963] INFO Creating new log file: log.53
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2016-05-06 17:44:24,972] INFO Established session
> 0x154868461350000 with negotiated timeout 6000 for client /10.116.93.88:58825
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:28,997] WARN caught end of stream exception
> (org.apache.zookeeper.server.NIOServerCnxn)
> EndOfStreamException: Unable to read additional data from
> client sessionid 0x154868461350000, likely client has closed socket
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Unknown Source)
> [2016-05-06 17:44:29,001] INFO Closed socket connection for
> client /10.116.93.88:58825 which had sessionid 0x154868461350000
> (org.apache.zookeeper.server.NIOServerCnxn)
> [2016-05-06 17:44:33,001] INFO Expiring session
> 0x154868461350000, timeout of 6000ms exceeded
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-05-06 17:44:33,002] INFO Processed session termination
> for sessionid: 0x154868461350000
> (org.apache.zookeeper.server.PrepRequestProcessor)
> Kafka: bin/kafka-server-start.sh -daemon config/server.properties
> ...
> [2016-05-06 17:44:24,353] INFO starting
> (kafka.server.KafkaServer)
> [2016-05-06 17:44:24,360] INFO Connecting to zookeeper on
> 10.116.93.88:2181 (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,428] FATAL Fatal error during KafkaServer
> startup. Prepare to shutdown (kafka.server.KafkaServer)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
> connect to zookeeper server within timeout: 6000
> at
> org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
> at
> org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
> at
> org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
> at
> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
> at
> kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
> at
> kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2016-05-06 17:44:30,431] INFO shutting down
> (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,438] INFO shut down completed
> (kafka.server.KafkaServer)
> [2016-05-06 17:44:30,439] FATAL Fatal error during
> KafkaServerStartable startup. Prepare to shutdown
> (kafka.server.KafkaServerStartable)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
> connect to zookeeper server within timeout: 6000
> at
> org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
> at
> org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
> at
> org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
> at
> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
> at
> kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
> at
> kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2016-05-06 17:44:30,442] INFO shutting down
> (kafka.server.KafkaServer)
>
>
> UPDATE:
> This is not actually a Kafka issue.
> The problem was at specifying the wrong FQDN (Fully Qualified Domain Name) at
> DNS.
> Kafka box has two DNS records:
> - with uppercase
> - with lowercase
> Kafka requests user with lowercase FQDN.
> Example:
> SBT-IPO-204.ca.sbrf.ru
> should be
> sbt-ipo-204.ca.sbrf.ru in JAAS file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
