I am working on a spark job which reads from kafka topic and write to HDFS
however while submitting the job using spark-submit command I am getting
following error.
Error log
Caused by: org.apache.kafka.common.KafkaException:
javax.security.auth.login.LoginException: Could not login: the
Hi Bill,
Normally Kerberos principal and keytab should be enough, because keytab
could actually represent the password. Did you configure SASL/GSSAPI or
SASL/Plain for KafkaClient?
http://kafka.apache.org/documentation.html#security_sasl
Actually this is more like a Kafka question and normally
Saisai,
Yea that seems to have helped. Looks like the kerberos ticket when I submit
does not get passed to the executor?
... 3 more
Caused by: org.apache.kafka.common.KafkaException:
javax.security.auth.login.LoginException: Unable to obtain password from
user
at
Hi Bill,
The exception is from executor side. From the gist you provided, looks like
the issue is that you only configured java options in driver side, I think
you should also configure this in executor side. You could refer to here (
I'm working on a poc spark job to pull data from a kafka topic with
kerberos enabled ( required ) brokers.
The code seems to connect to kafka and enter a polling mode. When I toss
something onto the topic I get an exception which I just can't seem to
figure out. Any ideas?
I have a full gist up
