Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-14 Thread Vijayendra Yadav
Thanks for your valuable inputs Robert, it helped me solve the issue. While I tried -yD from flink run, like you mentioned and many other combinations of the same, that didn't work out. Finally it worked when I passed it from flink-conf.yaml with relative path. Like below: env.java.opts.jobmanage

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-14 Thread Robert Metzger
Hi Vijayendra, I'm not sure if -yD is the right argument as you've posted it: It is meant to be used for Flink configuration keys, not for JVM properties. With the Flink configuration "env.java.opts", you should be able to pass JVM properties. This should work: -yD env.java.opts="-D java.security

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-13 Thread Vijayendra Yadav
Hi Yangze, I tried the following: maybe I am missing something. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/cli.html -yt,--yarnship Run: /usr/lib/flink/bin/flink run -m yarn-cluster -yt ${app_install_path}/conf my KRB5.conf is in ${app_install_path}/conf n master node (loc

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-13 Thread Yangze Guo
Hi, When deploying Flink on Yarn, you could ship krb5.conf by "--ship" command. Notice that this command only supports to ship folders now. Best, Yangze Guo On Fri, Aug 14, 2020 at 11:22 AM Vijayendra Yadav wrote: > > Any inputs ? > > On Tue, Aug 11, 2020 at 10:34 AM Vijayendra Yadav > wrote:

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-13 Thread Vijayendra Yadav
Any inputs ? On Tue, Aug 11, 2020 at 10:34 AM Vijayendra Yadav wrote: > Dawid, I was able to resolve the keytab issue by passing the service name, > but now I am facing the KRB5 issue. > > Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: > Failed to create SaslClient with m

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-11 Thread Vijayendra Yadav
Dawid, I was able to resolve the keytab issue by passing the service name, but now I am facing the KRB5 issue. Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism GSSAPI Caused by: javax.security.sasl.SaslException: Failure to initializ

Re: [Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-11 Thread Dawid Wysakowicz
Hi, As far as I know the approach 2) is the supported way of setting up Kerberos authentication in Flink. In the second approach have you tried setting the `sasl.kerberos.service.name` in the configuration of your KafkaConsumer/Producer[1]? I think this might be the issue. Best, Dawid [1] https

[Flink-KAFKA-KEYTAB] Kafkaconsumer error Kerberos

2020-08-09 Thread Vijayendra Yadav
Hi Team, I am trying to stream data from kafkaconsumer using: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html Here my KAFKA is Kerberos secured and SSL enabled. I am running my Flink streaming in yarn-cluster on EMR 5.31. I have tried to pass keytab/principal in