Hello all, I sent this in a couple of months ago, but I'm not sure if I was subscribed to the mailing list correctly and didn't see any replies - so I thought I'd try again.
I'm having trouble getting Kerberos authentication to work between Zookeeper and Accumulo. I am not using any supporting platforms (e.g. Cloudera, Hortonworks) - this is all being done using Docker with single Accumulo (1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their own Docker network. My KDC is running on a separate CentOS machine, but can be reached by all of them. I have already managed to integrate Kerberos authentication with Hadoop and Accumulo, but cannot add Zookeeper client into the mix. * The Zookeeper container has this configuration: /conf/zoo.cfg # Kerberos Configuration authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider kerberos.removeHostFromPrincipal=true kerberos.removeRealmFromPrincipal=true requireClientAuthScheme=sasl jaasLoginRenew=3600000 /conf/zookeeper_jaas_server.conf Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/zookeeper-server.keytab" principal="zookeeper/zk-kerberos.accumulo-krb-netw...@example.com<mailto:zookeeper/zk-kerberos.accumulo-krb-netw...@example.com>"; }; export SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_server.conf /conf/zookeeper_jaas_client.conf Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/zookeeper-client.keytab" principal="zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com<mailto:zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com>"; }; export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_client.conf * And I've added this configuration to Accumulo: /usr/local/zookeeper/conf/zookeeper_jaas_client.conf Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/accumulo.keytab" principal="zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com<mailto:zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com>"; }; export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf" This means that Zookeeper server starts by authenticating as principle zookeeper/zk-kerberos.accumulo-krb-netw...@example.com<mailto:zookeeper/zk-kerberos.accumulo-krb-netw...@example.com> and Accumulo (a Zookeeper client) authenticates with Zookeeper using the zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com<mailto:zookeeper-client/zk-kerberos.accumulo-krb-netw...@example.com> principle. When starting Accumulo for the first time and running through an initialisation, this authentication works fine and Accumulo starts as expected. But when you restart Accumulo, the master fails to start with this error: 2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor] ERROR: Saw more than one ACL on the node 2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor] ERROR: Expected /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} , 31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=} ] 2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception, exiting java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper is not protected. at org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDistributor.initialize(ZooAuthenticationKeyDistributor.java:86) at org.apache.accumulo.master.Master.run(Master.java:1223) at org.apache.accumulo.master.Master.main(Master.java:1434) at org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.java:33) at org.apache.accumulo.start.Main$1.run(Main.java:120) at java.lang.Thread.run(Thread.java:748) It looks like the information that Accumulo writes to Zookeeper during the initialisation has two ACLs associated to it - one for the zookeeper-client Kerberos principle (sasl) and one for the Accumulo secret (digest). These two ACLs seem to fail one of the master's start-up checks and causes it to exit. Is there anyway of disabling the Accumulo secret so that I only have the Kerberos ACL? Or is there something wrong with the way I've tried to implement this that would cause this problem? I'd be very appreciative of any assistance. Many thanks, Oliver Jones