Hi,
I am trying to migrate an unauthenticated zookeeper cluster to a kerberos
authenticated one. For the time being SSL is disabled. I have configured
the server and client as described below but when SASL is enabled I am
unable to retreive data using zookeeper shell client from the zookeeper
server. Could I get some help in understanding why this is failing?
server.log snippet
2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:44994
2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:44994
2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed
socket connection for client /127.0.0.1:44994 (no session established for
client)
2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client
environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT
2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client environment:
host.name=stage-kdc-zk-ivy
2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client
environment:java.version=1.8.0_172
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.vendor=Oracle Corporation
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.io.tmpdir=/tmp
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client
environment:java.compiler=<NA>
2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:
os.name=Linux
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:os.arch=amd64
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:os.version=4.9.0-9-amd64
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:
user.name=root
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:user.home=/root
2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client
environment:user.dir=/home/aparajita.singh
2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating client
connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
2020-06-10 17:09:26,752 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged in.
2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh thread
started.
2020-06-10 17:09:26,757 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - Client
will use GSSAPI as SASL mechanism.
2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid starting
at: Wed Jun 10 15:17:21 IST 2020
2020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires:
Thu Jun 11 15:17:21 IST 2020
2020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh
sleeping until: Thu Jun 11 11:17:04 IST 2020
2020-06-10 17:09:26,799 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] -
Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181.
Will attempt to SASL-authenticate using Login Context section 'Client'
2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /10.33.203.225:45018
2020-06-10 17:09:26,854 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - Socket
connection established to stage-kdc-zk-ivy/10.33.203.225:2181, initiating
session
2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish
new session at /10.33.203.225:45018
2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617] -
Established session 0x58729e0540980002 with negotiated timeout 30000 for
client /10.33.203.225:45018
2020-06-10 17:09:26,861 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] -
Session establishment complete on server stage-kdc-zk-ivy/10.33.203.225:2181,
sessionid = 0x58729e0540980002, negotiated timeout = 30000
2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969] - Client failed to SASL
authenticate: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism
level: Invalid argument (400) - Cannot find key of appropriate type to
decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]
2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975] - Closing client connection due
to SASL authentication failure.
2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
client /10.33.203.225:45018 which had sessionid 0x58729e0540980002
2020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
at
org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)
at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
at
org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:748)
2020-06-10 17:09:27,008 - INFO
[main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] -
Unable to read additional data from server sessionid 0x58729e0540980002,
likely server has closed socket, closing socket connection and attempting
reconnect
2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346] - Exception causing close of
session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException
2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /127.0.0.1:45004
2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing mntr command from /
127.0.0.1:45004
zookeeper shell client output
aparajita.singh@stage-kdc-zk-ivy:~$ sudo
/usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server
stage-kdc-zk-ivy get /test2
log4j:WARN Large window sizes are not allowed.
log4j:WARN MaxIndex reduced to 13.
Connecting to stage-kdc-zk-ivy
Debug is true storeKey false useTicketCache true useKeyTab true
doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is
/etc/krb5.keytab refreshKrb5Config is false principal is
zookeeper/stage-kdc-zk-...@stage.fdp.kafka tryFirstPass is false
useFirstPass is false storePass is false clearPass is false
Acquire TGT from Cache
Principal is zookeeper/stage-kdc-zk-...@stage.fdp.kafka
null credentials from Ticket Cache
principal is zookeeper/stage-kdc-zk-...@stage.fdp.kafka
Will use keytab
Commit Succeeded
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:Disconnected type:None path:null
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /test2
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
zoo.cfg
#setACL=False
autopurge.snapRetainCount=30
tickTime=2000
dataDir=/grid/1/var/lib/zookeeper
zookeeper_jmx_port=9009
initLimit=100
syncLimit=5
autopurge.purgeInterval=24
clientPort=2181
globalOutstandingLimit=5000
maxClientCnxns=2000
server.99=stage-kdc-zk-harley:2888:3888
server.88=stage-kdc-zk-ivy:2888:3888
server.77=stage-kdc-zk-2face:2888:3888
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.kerberos.servicePrincipal=host/stage-kdc-zk-...@stage.fdp.kafka
quorum.cnxn.threads.size=20
java.env
SERVER_JVMFLAGS="${SERVER_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"
CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS}
-Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf
-Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-Dsun.security.krb5.debug=true"
/home/aparajita.singh/jaas/jaas.conf
// Zookeeper server authentication
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-...@stage.fdp.kafka";
};
// Zookeeper quorum server authentication
QuorumServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-...@stage.fdp.kafka";
};
// Zookeeper learner authentication
QuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
//ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="host"
principal="host/stage-kdc-zk-...@stage.fdp.kafka";
};
/home/aparajita.singh/jaas/client.conf
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=true
ticketCache="/tmp/krb5cc_0"
renewTicket=true
doNotPrompt=true
debug=true
keyTab="/etc/krb5.keytab"
serviceName="zookeeper"
principal="zookeeper/stage-kdc-zk-...@stage.fdp.kafka";
};
Using kinit command I am able to generate the TGT for both principals. As
per the zookeeper server log, the TGT can be generated as expected. The
keytab file is accessible to all system users for now.
aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
zookeeper/stage-kdc-zk-...@stage.fdp.kafka -k -t /etc/krb5.keytab
aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit
host/stage-kdc-zk-...@stage.fdp.kafka -k -t /etc/krb5.keytab
--
Thanks,
Aparajita
