Re: Spark Structured Streaming not connecting to Kafka using kerberos
Hi Darshan, Did you try passing the config directly as an option, like this: .option("kafka.sasl.jaas.config", saslConfig) Where saslConfig can look like: com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab=true \ storeKey=true \ keyTab="/etc/security/keytabs/kafka_client.keytab" \ principal="kafka-clien...@example.com"; Reference: http://kafka.apache.org/documentation.html# security_kerberos_sasl_clientconfig Thanks, Prashant. On Tue, Oct 17, 2017 at 11:21 AM, Darshan Pandya wrote: > HI Burak, > > Well turns out it worked fine when i submit in cluster mode. I also tried > to convert my app in dstreams. In dstreams too it works well only when > deployed in cluster mode. > > Here is how i configured the stream. > > > val lines = spark.readStream > .format("kafka") > .option("kafka.bootstrap.servers", jobParams.boorstrapServer) > .option("subscribe", jobParams.sourceTopic) > .option("startingOffsets", "latest") > .option("minPartitions", "10") > .option("failOnDataLoss", "true") > .load() > > > > Sincerely, > Darshan > > On Mon, Oct 16, 2017 at 12:08 PM, Burak Yavuz wrote: > >> Hi Darshan, >> >> How are you creating your kafka stream? Can you please share the options >> you provide? >> >> spark.readStream.format("kafka") >> .option(...) // all these please >> .load() >> >> >> On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya >> wrote: >> >>> Hello, >>> >>> I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos >>> >>> I am unable to connect to the kafka broker with the following message >>> >>> >>> 17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker >>> 10.197.19.25:9092 disconnected >>> >>> and is unable to consume any messages. >>> >>> And am using it as follows >>> >>> jaas.conf >>> >>> KafkaClient { >>> com.sun.security.auth.module.Krb5LoginModule required >>> useKeyTab=true >>> keyTab="./gandalf.keytab" >>> storeKey=true >>> useTicketCache=false >>> serviceName="kafka" >>> principal="gand...@domain.com"; >>> }; >>> >>> $SPARK_HOME/bin/spark-submit \ >>> --master yarn \ >>> --files jaas.conf,gandalf.keytab \ >>> --driver-java-options "-Djava.security.auth.login.config=./jaas.conf >>> -Dhdp.version=2.4.2.0-258" \ >>> --conf >>> "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" >>> \ >>> --class com.example.ClassName uber-jar-with-deps-and-hive-site.jar >>> >>> Thanks in advance. >>> >>> >>> -- >>> Sincerely, >>> Darshan >>> >>> >> > > > -- > Sincerely, > Darshan > >
Re: Spark Structured Streaming not connecting to Kafka using kerberos
HI Burak, Well turns out it worked fine when i submit in cluster mode. I also tried to convert my app in dstreams. In dstreams too it works well only when deployed in cluster mode. Here is how i configured the stream. val lines = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", jobParams.boorstrapServer) .option("subscribe", jobParams.sourceTopic) .option("startingOffsets", "latest") .option("minPartitions", "10") .option("failOnDataLoss", "true") .load() Sincerely, Darshan On Mon, Oct 16, 2017 at 12:08 PM, Burak Yavuz wrote: > Hi Darshan, > > How are you creating your kafka stream? Can you please share the options > you provide? > > spark.readStream.format("kafka") > .option(...) // all these please > .load() > > > On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya > wrote: > >> Hello, >> >> I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos >> >> I am unable to connect to the kafka broker with the following message >> >> >> 17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker >> 10.197.19.25:9092 disconnected >> >> and is unable to consume any messages. >> >> And am using it as follows >> >> jaas.conf >> >> KafkaClient { >> com.sun.security.auth.module.Krb5LoginModule required >> useKeyTab=true >> keyTab="./gandalf.keytab" >> storeKey=true >> useTicketCache=false >> serviceName="kafka" >> principal="gand...@domain.com"; >> }; >> >> $SPARK_HOME/bin/spark-submit \ >> --master yarn \ >> --files jaas.conf,gandalf.keytab \ >> --driver-java-options "-Djava.security.auth.login.config=./jaas.conf >> -Dhdp.version=2.4.2.0-258" \ >> --conf >> "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" >> \ >> --class com.example.ClassName uber-jar-with-deps-and-hive-site.jar >> >> Thanks in advance. >> >> >> -- >> Sincerely, >> Darshan >> >> > -- Sincerely, Darshan
Re: Spark Structured Streaming not connecting to Kafka using kerberos
Hi Darshan, How are you creating your kafka stream? Can you please share the options you provide? spark.readStream.format("kafka") .option(...) // all these please .load() On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya wrote: > Hello, > > I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos > > I am unable to connect to the kafka broker with the following message > > > 17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker > 10.197.19.25:9092 disconnected > > and is unable to consume any messages. > > And am using it as follows > > jaas.conf > > KafkaClient { > com.sun.security.auth.module.Krb5LoginModule required > useKeyTab=true > keyTab="./gandalf.keytab" > storeKey=true > useTicketCache=false > serviceName="kafka" > principal="gand...@domain.com"; > }; > > $SPARK_HOME/bin/spark-submit \ > --master yarn \ > --files jaas.conf,gandalf.keytab \ > --driver-java-options "-Djava.security.auth.login.config=./jaas.conf > -Dhdp.version=2.4.2.0-258" \ > --conf > "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" > \ > --class com.example.ClassName uber-jar-with-deps-and-hive-site.jar > > Thanks in advance. > > > -- > Sincerely, > Darshan > >
Spark Structured Streaming not connecting to Kafka using kerberos
Hello, I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos I am unable to connect to the kafka broker with the following message 17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected and is unable to consume any messages. And am using it as follows jaas.conf KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="./gandalf.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="gand...@domain.com"; }; $SPARK_HOME/bin/spark-submit \ --master yarn \ --files jaas.conf,gandalf.keytab \ --driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \ --class com.example.ClassName uber-jar-with-deps-and-hive-site.jar Thanks in advance. -- Sincerely, Darshan