gentle reminder (unquoting the previous email) --
Hi, I am trying to migrate an unauthenticated zookeeper cluster to a kerberos authenticated one. For the time being SSL is disabled. I have configured the server and client as described below but when SASL is enabled I am unable to retreive data using zookeeper shell client from the zookeeper server. Could I get some help in understanding why this is failing? *server.log snippet* *2020-06-10 17:09:01,263 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted socket connection from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10 17:09:01,264 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827>] - Processing mntr command from /127.0.0.1:44994 <http://127.0.0.1:44994>2020-06-10 17:09:01,265 - INFO [Thread-5:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:44994 <http://127.0.0.1:44994> (no session established for client)2020-06-10 17:09:26,647 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-169--1, built on 02/10/2016 05:49 GMT2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client environment:host.name <http://host.name>=stage-kdc-zk-ivy2020-06-10 17:09:26,649 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_1722020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/oracle-java8-jdk-amd64/jre2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/classes:/usr/hdp/2.4.0.0-169/zookeeper/bin/../build/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/bin/../src/java/lib/*.jar:/usr/hdp/2.4.0.0-169/zookeeper/conf::/usr/hdp/2.4.0.0-169/zookeeper/conf:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/apache-log4j-extras-1.2.17.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/share/zookeeper/*2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>2020-06-10 17:09:26,651 - INFO [main:Environment@100] - Client environment:os.name <http://os.name>=Linux2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:os.arch=amd642020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:os.version=4.9.0-9-amd642020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:user.name <http://user.name>=root2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:user.home=/root2020-06-10 17:09:26,652 - INFO [main:Environment@100] - Client environment:user.dir=/home/aparajita.singh2020-06-10 17:09:26,653 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=stage-kdc-zk-ivy sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa2020-06-10 17:09:26,752 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):Login@293] - successfully logged in.2020-06-10 17:09:26,753 - INFO [Thread-0:Login$1@127] - TGT refresh thread started.2020-06-10 17:09:26,757 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):ZooKeeperSaslClient$1@285] - Client will use GSSAPI as SASL mechanism.2020-06-10 17:09:26,758 - INFO [Thread-0:Login@301] - TGT valid starting at: Wed Jun 10 15:17:21 IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login@302] - TGT expires: Thu Jun 11 15:17:21 IST 20202020-06-10 17:09:26,758 - INFO [Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Jun 11 11:17:04 IST 20202020-06-10 17:09:26,799 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>. Will attempt to SASL-authenticate using Login Context section 'Client'2020-06-10 17:09:26,854 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted socket connection from /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10 17:09:26,854 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@864] - Socket connection established to stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>, initiating session2020-06-10 17:09:26,856 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868 <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868>] - Client attempting to establish new session at /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10 17:09:26,859 - INFO [CommitProcessor:88:ZooKeeperServer@617] - Established session 0x58729e0540980002 with negotiated timeout 30000 for client /10.33.203.225:45018 <http://10.33.203.225:45018>2020-06-10 17:09:26,861 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1279] - Session establishment complete on server stage-kdc-zk-ivy/10.33.203.225:2181 <http://10.33.203.225:2181>, sessionid = 0x58729e0540980002, negotiated timeout = 300002020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969 <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@969>] - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)]2020-06-10 17:09:27,007 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975 <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@975>] - Closing client connection due to SASL authentication failure.2020-06-10 17:09:27,007 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket connection for client /10.33.203.225:45018 <http://10.33.203.225:45018> which had sessionid 0x58729e05409800022020-06-10 17:09:27,008 - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@178>] - Unexpected Exception: java.nio.channels.CancelledKeyExceptionat sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)at org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:936)at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)at java.lang.Thread.run(Thread.java:748)2020-06-10 17:09:27,008 - INFO [main-SendThread(stage-kdc-zk-ivy:2181):ClientCnxn$SendThread@1142] - Unable to read additional data from server sessionid 0x58729e0540980002, likely server has closed socket, closing socket connection and attempting reconnect2020-06-10 17:09:27,008 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@346>] - Exception causing close of session 0x58729e0540980002 due to java.nio.channels.CancelledKeyException2020-06-10 17:10:01,317 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted socket connection from /127.0.0.1:45004 <http://127.0.0.1:45004>2020-06-10 17:10:01,318 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827 <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827>] - Processing mntr command from /127.0.0.1:45004 <http://127.0.0.1:45004>* *zookeeper shell client outputaparajita.singh@stage-kdc-zk-ivy:~$ sudo /usr/hdp/2.4.0.0-169/zookeeper/bin/zookeeper-client -server stage-kdc-zk-ivy get /test2log4j:WARN Large window sizes are not allowed.log4j:WARN MaxIndex reduced to 13.Connecting to stage-kdc-zk-ivyDebug is true storeKey false useTicketCache true useKeyTab true doNotPrompt true ticketCache is /tmp/krb5cc_0 isInitiator true KeyTab is /etc/krb5.keytab refreshKrb5Config is false principal is zookeeper/[email protected] tryFirstPass is false useFirstPass is false storePass is false clearPass is falseAcquire TGT from CachePrincipal is zookeeper/[email protected] credentials from Ticket Cacheprincipal is zookeeper/[email protected] use keytabCommit Succeeded WATCHER::WatchedEvent state:SyncConnected type:None path:nullWATCHER::WatchedEvent state:Disconnected type:None path:nullException in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test2at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184)at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:717)at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:354)at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)zoo.cfg#setACL=Falseautopurge.snapRetainCount=30tickTime=2000dataDir=/grid/1/var/lib/zookeeperzookeeper_jmx_port=9009initLimit=100syncLimit=5autopurge.purgeInterval=24clientPort=2181globalOutstandingLimit=5000maxClientCnxns=2000server.99=stage-kdc-zk-harley:2888:3888server.88=stage-kdc-zk-ivy:2888:3888server.77=stage-kdc-zk-2face:2888:3888authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProviderrequireClientAuthScheme=saslquorum.auth.enableSasl=truequorum.auth.learnerRequireSasl=truequorum.auth.serverRequireSasl=truequorum.auth.kerberos.servicePrincipal=host/[email protected]=20* *java.envSERVER_JVMFLAGS="${SERVER_JVMFLAGS} -Djava.security.auth.login.config=/home/aparajita.singh/jaas/jaas.conf -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dsun.security.krb5.debug=true"CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Djava.security.auth.login.config=/home/aparajita.singh/jaas/client.conf -Dzookeeper.authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dsun.security.krb5.debug=true"/home/aparajita.singh/jaas/jaas.conf// Zookeeper server authenticationServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true debug=true keyTab="/etc/krb5.keytab" serviceName="host" principal="host/[email protected]"; }; // Zookeeper quorum server authenticationQuorumServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true debug=true keyTab="/etc/krb5.keytab" serviceName="host" principal="host/[email protected]"; }; // Zookeeper learner authenticationQuorumLearner { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false //ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true debug=true keyTab="/etc/krb5.keytab" serviceName="host" principal="host/[email protected]"; }; /home/aparajita.singh/jaas/client.conf// Zookeeper client authenticationClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=true ticketCache="/tmp/krb5cc_0" renewTicket=true doNotPrompt=true debug=true keyTab="/etc/krb5.keytab" serviceName="zookeeper" principal="zookeeper/[email protected]"; }; * Using kinit command I am able to generate the TGT for both principals. As per the zookeeper server log, the TGT can be generated as expected. The keytab file is accessible to all system users for now. The below commands don't give any output and the lack of error indicates that the ticket was generated successfully. klist command also shows the latest ticket generated as expected. *aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit zookeeper/[email protected] -k -t /etc/krb5.keytab aparajita.singh@stage-kdc-zk-ivy:~$ sudo /krb5/bin/kinit host/[email protected] -k -t /etc/krb5.keytab * Thanks, Aparajita
