Guys, thank you so much for helping, but the error was all on my end. This
morning I’ve looked at my krb5.conf and noticed this:
[domain_realm]
.ubuntu.oleg.com = OLEG.COM
ubuntu.oleg.com = OLEG.COM
instead of
[domain_realm]
.oleg.com = OLEG.COM
oleg.com = OLEG.COM
Once I changed it all went fine!
Cheers
Oleg
> On Feb 23, 2016, at 6:09 PM, Oleg Zhurakousky <[email protected]>
> wrote:
>
> Well, I am running on the same machine, so I say yes
>
> Sent from my iPhone
>
>> On Feb 23, 2016, at 18:05, Martin Gainty <[email protected]> wrote:
>>
>> one more thing to check:
>>
>> specifically are the /etc/krb5.conf credentials the same you use to
>> authenticate to ubuntu.oleg.com
>>
>> ?
>> Martin
>> __________________
>>
>>
>>
>>
>>> Subject: Re: Kerberized Kafka setup issues
>>> From: [email protected]
>>> To: [email protected]
>>> Date: Tue, 23 Feb 2016 21:58:48 +0000
>>>
>>> Harsh
>>>
>>> I followed this blog
>>> (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption)
>>> and got an environment via vagrant setup, no issues. I’ll poke around what
>>> the differences are and if find the issue will post.
>>> Thanks for your help anyway.
>>>
>>> Cheers
>>> Oleg
>>> On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky
>>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>> Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN
>>> and it is still the same including 'sname is
>>> zookeeper/[email protected]<mailto:zookeeper/[email protected]>’
>>>
>>> Oleg
>>>
>>> On Feb 23, 2016, at 4:00 PM, Harsha
>>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>> whats your zookeeper.connect in server.properties looks like. Did you
>>> use the hostname or localhost
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
>>> Still digging, but here is more info that may help
>>>
>>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
>>> (org.I0Itec.zkclient.ZkClient)
>>> Found ticket for
>>> kafka/[email protected]<mailto:kafka/[email protected]> to go
>>> to
>>> krbtgt/[email protected]<mailto:krbtgt/[email protected]> expiring on Wed
>>> Feb 24 00:59:24 EST 2016
>>> Entered Krb5Context.initSecContext with state=STATE_NEW
>>> Found ticket for
>>> kafka/[email protected]<mailto:kafka/[email protected]> to go
>>> to
>>> krbtgt/[email protected]<mailto:krbtgt/[email protected]> expiring on Wed
>>> Feb 24 00:59:24 EST 2016
>>> Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
>>> Using builtin default etypes for default_tgs_enctypes
>>> default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88,
>>> timeout=30000, number of retries =3, #bytes=660
>>> KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88,
>>> timeout=30000,Attempt =1, #bytes=660
>>> KrbKdcReq send: #bytes read=183
>>> KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com>
>>> KDCRep: init() encoding tag is 126 req type is 13
>>> KRBError:
>>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>>> suSec is 248635
>>> error code is 7
>>> error Message is Server not found in Kerberos database
>>> cname is
>>> kafka/[email protected]<mailto:kafka/[email protected]>
>>> sname is zookeeper/[email protected]<mailto:zookeeper/[email protected]>
>>> msgType is 30
>>>
>>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky
>>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>> No joy. the same error
>>>
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> serviceName=zookeeper
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>> On Feb 23, 2016, at 2:41 PM, Harsha <[email protected]<mailto:[email protected]>>
>>> wrote:
>>>
>>> My bad it should be under Client section
>>>
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> serviceName=zookeeper
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>>
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> serviceName=zookeeper
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>>
>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>> More info
>>>
>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>> and Kafka as ‘kafka’
>>>
>>> Oleg
>>>
>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky
>>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>> Harsha
>>>
>>> Thanks for following up. Here is is:
>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/[email protected]<mailto:kafka/[email protected]>";
>>> };
>>>
>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>> Server {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>> storeKey=true
>>> useTicketCache=false
>>> principal="zookeeper/[email protected]<mailto:zookeeper/[email protected]>";
>>> };
>>>
>>> Cheers
>>> Oleg
>>>
>>> On Feb 23, 2016, at 2:17 PM, Harsha
>>> <[email protected]<mailto:[email protected]>> wrote:
>>>
>>> Oleg,
>>> Can you post your jaas configs. Its important that serviceName
>>> must match the principal name with which zookeeper is running.
>>> Whats the principal name zookeeper service is running with.
>>> -Harsha
>>>
>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>> Hey guys, first post here so bare with me
>>>
>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to be
>>> very close, but not quite there yet.
>>>
>>> ZOOKEEPER
>>> Starting Zookeeper seems to be OK (below is the relevant part of the log)
>>> . . .
>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>> refreshKrb5Config is false principal is
>>> zookeeper/[email protected]<mailto:zookeeper/[email protected]><mailto:zookeeper/[email protected]>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> zookeeper/[email protected]<mailto:zookeeper/[email protected]><mailto:zookeeper/[email protected]>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23
>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23
>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new session
>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>> . . .
>>>
>>>
>>> KAFKA
>>> Starting Kafka server is not going well yet although I see that
>>> interaction with Kerberos is successful (see relevant log below. the
>>> error is at the bottom)
>>> . . .
>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>> (kafka.server.KafkaServer)
>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>> GMT (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,527] INFO Client
>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>> . . . . .
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,531] INFO Client
>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>> connectString=localhost:2181 sessionTimeout=6000
>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>> (org.apache.zookeeper.ZooKeeper)
>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated
>>> (org.I0Itec.zkclient.ZkClient)
>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>> refreshKrb5Config is false principal is
>>> kafka/[email protected]<mailto:kafka/[email protected]><mailto:kafka/[email protected]>
>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>> is false
>>> principal is
>>> kafka/[email protected]<mailto:kafka/[email protected]><mailto:kafka/[email protected]>
>>> Will use keytab
>>> Commit Succeeded
>>>
>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>> (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>> localhost/127.0.0.1:2181, initiating session
>>> (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23
>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23
>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:11,773] ERROR An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state.
>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum
>>> member failed: javax.security.sasl.SaslException: An error:
>>> (java.security.PrivilegedActionException:
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>> (org.I0Itec.zkclient.ZkClient)
>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>> (org.I0Itec.zkclient.ZkEventThread)
>>> . . .
>>>
>>> Any pointers?
>>>
>>> Cheers
>>> Oleg
>>
>
