Re: [EXTERNAL] kafka
Thanks a lot for the response Pramod. Our kafka environment is NOT Kerberized! But the kafka topic is secure via SSL certificates! Only hosts with valid SSL certificates can only connect! Any thoughts in this regard will be really helpful!! Thanks a lot!! Regards, Raja. From: Pramod Immaneni <pra...@datatorrent.com> Reply-To: "users@apex.apache.org" <users@apex.apache.org> Date: Tuesday, November 1, 2016 at 3:24 AM To: "users@apex.apache.org" <users@apex.apache.org> Subject: Re: [EXTERNAL] kafka Hi Raja, Yes we have had success reading from kerberized secure Kafka. If I remember correctly we had to set a few attributes like container jvm options in Apex. Is your setup Kerberos secured Kafka? Thanks On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli <raja.aravapa...@target.com<mailto:raja.aravapa...@target.com>> wrote: Hi Team, Can someone pls help me with below requested information ? Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 secure kafka topics? Thanks a lot. Regards, Raja. From: "Raja.Aravapalli" <raja.aravapa...@target.com<mailto:raja.aravapa...@target.com>> Reply-To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>> Date: Monday, October 24, 2016 at 2:29 PM To: "users@apex.apache.org<mailto:users@apex.apache.org>" <users@apex.apache.org<mailto:users@apex.apache.org>> Subject: [EXTERNAL] kafka Hi, Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters? Thanks a lot. Regards, Raja.
Re: [EXTERNAL] kafka
IMO this should be added here: http://apex.apache.org/docs/malhar/operators/kafkaInputOperator/ On Tue, Nov 1, 2016 at 5:24 PM, hsy...@gmail.com <hsy...@gmail.com> wrote: > Hey Raja, > > The setup for secure kafka input operator is not easy. You can follow > these steps. > 1. Follow kafka document to setup your brokers properly ( > https://kafka.apache.org/090/documentation.html#security_overview) > 2. You have to manually create the client JAAS file ( > https://kafka.apache.org/090/documentation.html#security_overview) >a sample file would look like this: > KafkaClient { > > com.sun.security.auth.module.Krb5LoginModule required > useKeyTab=true > storeKey=true > keyTab="/etc/security/keytabs/kafka_client.keytab" > principal="kafka-clien...@example.com"; > }; > > 3. On the operator you have to set the attribute JVM_OPTS > -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf > > 4. On the operator you have to set the consumerProperties, for example > > set dt.operator.$your_operator_name.{consumerProps.security.protocol} to > SASL or other security function you use > set dt.operator.$your_operator_name.{consumerPrope.sasl. > kerberos.service.name} to kafka > > > Hope this would help you! > > > Regards, > Siyuan > > > > On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli < > raja.aravapa...@target.com> wrote: > >> >> >> Hi Team, >> >> >> >> Can someone pls help me with below requested information ? >> >> >> >> Does apache apex have any inbuilt kafka input operator to read from Kafka >> 0.9 secure kafka topics? >> >> >> >> Thanks a lot. >> >> >> >> Regards, >> >> Raja. >> >> >> >> *From: *"Raja.Aravapalli" <raja.aravapa...@target.com> >> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org> >> *Date: *Monday, October 24, 2016 at 2:29 PM >> *To: *"users@apex.apache.org" <users@apex.apache.org> >> *Subject: *[EXTERNAL] kafka >> >> >> >> >> >> Hi, >> >> >> >> Do we have any kaka operator readily available to consume messages from >> secure kafka topics in kafka 0.9 clusters? >> >> >> >> Thanks a lot. >> >> >> >> >> >> Regards, >> >> Raja. >> > >
Re: [EXTERNAL] kafka
Hey Raja, The setup for secure kafka input operator is not easy. You can follow these steps. 1. Follow kafka document to setup your brokers properly ( https://kafka.apache.org/090/documentation.html#security_overview) 2. You have to manually create the client JAAS file ( https://kafka.apache.org/090/documentation.html#security_overview) a sample file would look like this: KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka_client.keytab" principal="kafka-clien...@example.com"; }; 3. On the operator you have to set the attribute JVM_OPTS -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf 4. On the operator you have to set the consumerProperties, for example set dt.operator.$your_operator_name.{consumerProps.security.protocol} to SASL or other security function you use set dt.operator.$your_operator_name.{consumerPrope. sasl.kerberos.service.name} to kafka Hope this would help you! Regards, Siyuan On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli < raja.aravapa...@target.com> wrote: > > > Hi Team, > > > > Can someone pls help me with below requested information ? > > > > Does apache apex have any inbuilt kafka input operator to read from Kafka > 0.9 secure kafka topics? > > > > Thanks a lot. > > > > Regards, > > Raja. > > > > *From: *"Raja.Aravapalli" <raja.aravapa...@target.com> > *Reply-To: *"users@apex.apache.org" <users@apex.apache.org> > *Date: *Monday, October 24, 2016 at 2:29 PM > *To: *"users@apex.apache.org" <users@apex.apache.org> > *Subject: *[EXTERNAL] kafka > > > > > > Hi, > > > > Do we have any kaka operator readily available to consume messages from > secure kafka topics in kafka 0.9 clusters? > > > > Thanks a lot. > > > > > > Regards, > > Raja. >