Spark Structured Streaming not connecting to Kafka using kerberos

[Darshan Pandya]

Hello,

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker
10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="gand...@domain.com";
};

$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf
-Dhdp.version=2.4.2.0-258" \
--conf 
"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"
\
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar

Thanks in advance.


-- 
Sincerely,
Darshan