Made a small mistake, in zk_server_jaas.conf
provided,
> Server{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="super"
> password="adminsecret";
};
Instead of,
> Server{
> org.apache.zookeeper.server.auth.DigestLoginModule required
> user_super="adminsecret";
};
Now after making this change, I can make the kafka nodes as world-readable
and modifiable only by brokers (as mentioned in kafka doc)
Thanks and regards
Arjun S V
On Thu, Nov 23, 2023 at 10:57 AM arjun s v <arjun.cs...@gmail.com> wrote:
> Hi Alex Brekken,
>
> Sorry for the delayed response, I tried your fix,
> At first I got
>
>> Fatal error during KafkaServer startup. Prepare to shutdown"
>> "org.apache.kafka.common.KafkaException: Exception while determining if
>> ZooKeeper is secure
>> [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
>> zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client]
>> at
>> org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75)
>> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> at kafka.Kafka$.main(Kafka.scala:109)
>> at kafka.Kafka.main(Kafka.scala)
>
>
> Then I set zookeeper.sasl.client=true
>
>> - 10.91.21.142 arjun-8481 - - - 23
>> org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE
>> "23-11-2023 10:49:21:770" - "SASL authentication failed using login context
>> 'Client' with exception: {}" "javax.security.sasl.SaslException: Error in
>> authenticating with a Zookeeper Quorum member: the quorum member's
>> saslToken is null.
>> at
>> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
>> at
>> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
>> - 10.91.21.142 arjun-8481 - - - 23
>> org.apache.zookeeper.ClientCnxn$SendThread run INFO "23-11-2023
>> 10:49:21:771" - "Unable to read additional data from server sessionid
>> 0x100147c3ccb0000, likely server has closed socket, closing socket
>> connection and attempting reconnect" - - - - - - 1700716761771 - - - - - -
>> - - logger_name=org.apache.zookeeper.ClientCnxn
>> - 10.91.21.142 arjun-8481 - - - 24 kafka.utils.Logging error SEVERE
>> "23-11-2023 10:49:21:771" - "[ZooKeeperClient Kafka server] Auth failed." -
>> - - - - - 1700716761771 - - - - - - - - logtype=application
>> logger_name=kafka.zookeeper.ZooKeeperClient
>> - 10.91.21.142 arjun-8481 - - - 24
>> org.apache.zookeeper.ClientCnxn$EventThread run INFO "23-11-2023
>> 10:49:21:773" - "EventThread shut down for session: 0x100147c3ccb0000" - -
>> - - - - 1700716761773 - - - - - - - - logtype=application
>> thread_name=main-EventThread logger_name=org.apache.zookeeper.ClientCnxn
>> - 10.91.21.142 arjun-8481 - - - 1 kafka.utils.Logging fatal SEVERE
>> "23-11-2023 10:49:21:887" - "Fatal error during KafkaServer startup.
>> Prepare to shutdown"
>> "org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode
>> = AuthFailed for /kafka/1/kafka/1
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
>> at
>> kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
>> at
>> kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
>> at
>> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
>> at
>> kafka.server.KafkaServer.$anonfun$initZkClient$2(KafkaServer.scala:451)
>> at
>> kafka.server.KafkaServer.$anonfun$initZkClient$2$adapted(KafkaServer.scala:448)
>> at scala.Option.foreach(Option.scala:437)
>> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:448)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> at kafka.Kafka$.main(Kafka.scala:109)
>> at kafka.Kafka.main(Kafka.scala)
>
>
> Please advice.
>
> On Mon, Nov 13, 2023 at 4:11 AM Alex Brekken <brek...@gmail.com> wrote:
>
>> I see a couple of things that look wrong. First, remove this line from
>> your
>> ZK config:
>>
>> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider.
>> And replace it with this:
>>
>> authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider.
>>
>> Additionally, I think you need to add these lines to the ZK config if you
>> want ZK to ZK authentication:
>> quorum.auth.enableSasl=true
>> quorum.auth.learnerRequireSasl=true
>> quorum.auth.serverRequireSasl=true
>>
>> The rest looks OK.
>>
>> On Sat, Nov 11, 2023 at 9:12 PM arjun s v <arjun.cs...@gmail.com> wrote:
>>
>> > A small correction, I'm not trying to enable mTLS, just a simple
>> > authentication(Digest or Plain) is enough,
>> > Sharing the jaas files and config files,
>> > kafka_server_jaas.conf
>> >
>> > > admin.KafkaServer{
>> > >
>> > > org.apache.kafka.common.security.plain.PlainLoginModule required
>> > >
>> > > username="USERNAME"
>> > >
>> > > password="PASSWORD";
>> > >
>> > > };
>> > >
>> > > KafkaServer{
>> > >
>> > > org.apache.kafka.common.security.plain.PlainLoginModule required
>> > >
>> > > username="USERNAME"
>> > >
>> > > password="PASSWORD";
>> > >
>> > > };
>> > >
>> > > Client{
>> > >
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > >
>> > > username="super"
>> > >
>> > > password="adminsecret";
>> > >
>> > > };
>> > >
>> > server properties
>> >
>> > > zookeeper.sasl.client=false
>> >
>> >
>> java.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf
>> >
>> > zookeeper.connection.timeout.ms=6000
>> >
>> > zookeeper.sync.time.ms=2000
>> >
>> > zookeeper.set.acl=true
>> >
>> >
>> > zk_server_jaas.conf
>> >
>> > > QuorumServer{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > user_test="test";
>> > > };
>> > > QuorumLearner{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > username="test"
>> > > password="test";
>> > > };
>> > > Server{
>> > > org.apache.zookeeper.server.auth.DigestLoginModule required
>> > > username="super"
>> > > password="adminsecret";
>> >
>> > };
>> >
>> >
>> > zoo.cfg
>> >
>> > >
>> > >
>> >
>> zookeeper.authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider
>> >
>> >
>> > env
>> >
>> > > SERVER_JVMFLAGS="$SERVER_JVMFLAGS
>> > > -Djava.security.auth.login.config=$ZOOCFGDIR/zk_server_jaas.conf"
>> > >
>> > On Fri, Nov 10, 2023 at 7:03 PM Alex Brekken <brek...@gmail.com> wrote:
>> >
>> > > Ok, so you're trying to enable both SASL authentication (digest) and
>> TLS,
>> > > using mTLS for Zookeeper? I'm just trying to understand the bigger
>> > > picture. The error you're getting regarding the Sasl token sounds
>> like
>> > > either the jaas config on the Kafka broker side is wrong/missing, or
>> the
>> > > jaas config on the ZK side is wrong/missing. (you need both - in this
>> > case
>> > > the broker is the "client" and ZK is the "server"). Are you able to
>> share
>> > > the jaas config you're using for both Kafka and ZK? Without seeing
>> that
>> > > it's tough to know. Also, to make troubleshooting easier you might
>> want
>> > to
>> > > leave TLS out of it for now and get SASL working first. (or
>> vice-versa)
>> > >
>> > >
>> > >
>> > > On Thu, Nov 9, 2023 at 11:26 PM arjun s v <arjun.cs...@gmail.com>
>> wrote:
>> > >
>> > > > "Digest-MD5 is SASL authentication, so not sure what you mean here."
>> > > > If I set zookeeper.sasl.client=true, zookeeper expects a "saslToken"
>> > and
>> > > > throws the following error,
>> > > >
>> > > > "SASL authentication failed using login context 'Client' with
>> > exception:
>> > > > {}" "javax.security.sasl.SaslException: Error in authenticating
>> with a
>> > > > Zookeeper Quorum member: the quorum member's saslToken is null.
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
>> > > >
>> > > > at
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
>> > > >
>> > > >
>> > > > "Hmm, that config shouldn't have anything to do with TLS. You can
>> set
>> > > ACL's
>> > > >
>> > > > with or without TLS encryption. Were you getting an error?"
>> > > >
>> > > >
>> > > > "Fatal error during KafkaServer startup. Prepare to shutdown"
>> > > > "java.lang.SecurityException: zookeeper.set.acl is true, but
>> ZooKeeper
>> > > > client TLS configuration identifying at least
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslClientEnableProp,
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkClientCnxnSocketProp, and
>> > > > kafka.server.KafkaConfig$@7b22ec89.ZkSslKeyStoreLocationProp was not
>> > > > present and the verification of the JAAS login file failed
>> > > > [java.security.auth.login.config=./../config/kafka_server_jaas.conf,
>> > > > zookeeper.sasl.client=false,
>> > zookeeper.sasl.clientconfig=default:Client]
>> > > >
>> > > > at
>> kafka.server.KafkaServer.initZkClient(KafkaServer.scala:445)
>> > > >
>> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
>> > > >
>> > > > at kafka.Kafka$.main(Kafka.scala:109)
>> > > >
>> > > > at kafka.Kafka.main(Kafka.scala)
>> > > >
>> > > >
>> > > > "This was the 2nd result in a google search:
>> > > >
>> https://docs.confluent.io/platform/current/security/zk-security.html";
>> > > >
>> > > > FYKI, I've googled, asked chat gpt, surfed over many zookeeper and
>> > kafka
>> > > > docs and blog,
>> > > > I remember trying the doc you suggested here about 10 days back in
>> the
>> > > > initial days of this task!
>> > > > About the doc you suggested,
>> > > > I cannot configure SSL as I already mentioned, If I skip ssl config
>> > part
>> > > > from your suggested doc and tried Digest-MD5, I come up "saslToken
>> > > missing"
>> > > > exception which I mentioned above!
>> > > > I don't really understand what saslToken is and how to make it get
>> > > > generated for Digest auth!
>> > > > Please assist!
>> > > >
>> > > > On Thu, Nov 9, 2023 at 7:15 PM Alex Craig <alexcrai...@gmail.com>
>> > wrote:
>> > > >
>> > > > > " I couldn't find any doc by kafka to enable Digest-MD5
>> > > authentication."
>> > > > > This was the 2nd result in a google search:
>> > > > >
>> https://docs.confluent.io/platform/current/security/zk-security.html
>> > > > >
>> > > > > " I don't want to enable SASL."
>> > > > > Digest-MD5 is SASL authentication, so not sure what you mean here.
>> > > > >
>> > > > > " If I set zookeeper.set.acl=true, I'm forced to configure TLS."
>> > > > > Hmm, that config shouldn't have anything to do with TLS. You can
>> set
>> > > > ACL's
>> > > > > with or without TLS encryption. Were you getting an error?
>> > > > >
>> > > > > On Wed, Nov 8, 2023 at 11:35 PM arjun s v <arjun.cs...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Team,
>> > > > > >
>> > > > > > Please consider this as high priority, we need to enable
>> > > authentication
>> > > > > > ASAP. Please assist.
>> > > > > > On Tue, Nov 7, 2023 at 4:38 PM arjun s v <arjun.cs...@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi team,
>> > > > > > >
>> > > > > > > I'm trying to configure *Digest-MD5* authentication between
>> kafka
>> > > and
>> > > > > > > zookeeper.
>> > > > > > > Also I need to set ACL with digest scheme and credentials.
>> > > > > > > I don't want to enable SASL.
>> > > > > > > I tried to follow this
>> > > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
>> > > > > >
>> > > > > > doc
>> > > > > > > from zookeeper,
>> > > > > > >
>> > > > > > > - If I configured a jaas file, I have to set
>> > > > > > zookeeper.sasl.client=true(if
>> > > > > > > not kafka throws error from JaasUtils) which enables sasl
>> > > > > > authentication.
>> > > > > > > - If I set zookeeper.set.acl=true, I'm forced to configure
>> > TLS.
>> > > > > > >
>> > > > > > > I couldn't find any doc by kafka to enable Digest-MD5
>> > > authentication.
>> > > > > > > I cannot configure kerberos or TLS, just a Digest-MD5 is
>> > sufficient
>> > > > for
>> > > > > > my
>> > > > > > > usecase.
>> > > > > > > Please let me know if there are any docs to enable Digest-MD5
>> > auth
>> > > > > > between
>> > > > > > > kafka and zookeeper.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Arjun S V
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
