Re: Is anyone using MX4J loader?

2017-10-03 Thread Dong Lin
Thanks for the information Ted. Yes, it does appear that the issue has already been reported in KAFKA-4946. I will comment on that ticket and see if Ralph can open a git pull request instead. Or we can simply commit Ralph's patch in his name. Thanks, Dong On Fri, Sep 29, 2017 at 4:56 AM, Ted Yu

Re: Using Kafka on DC/OS + Marathon

2017-10-03 Thread Valentin Forst
Hi David, Thank you for your replay! Presumably I wasn’t clear in my previous post. Here an example to visualize what I'm trying to figure out: Imagine we have a data flow propagating massages through a Kafka-Cluster which is happen to consist of 3 brokers (3 partitions, 3 replica). If one of t

Re: Using Kafka on DC/OS + Marathon

2017-10-03 Thread Valentin Forst
Hi Avinash, Thanks for this hint. It would have been great, if someone could share experience using this framework on the production environment. Thanks in advance Valentin > Am 02.10.2017 um 19:39 schrieb Avinash Shahdadpuri : > > There is a a native kafka framework which runs on top of DC/

Re: Using Kafka on DC/OS + Marathon

2017-10-03 Thread Sean Glover
Hi Valentin, Kafka is available on DC/OS in the Catalog (aka Universe) as part of the `kafka` package. Mesosphere has put a lot of effort into making Kafka work on DC/OS. Since Kafka requires persistent disk it's required to make sure after initial deployment brokers stay put on their assigned M

Kafka Streams Avro SerDe version/id caching

2017-10-03 Thread Kristopher Kane
If using a Byte SerDe and schema registry in the consumer configs of a Kafka streams application, does it cache the Avro schemas by ID and version after fetching from the registry once? Thanks, Kris

kafka-console-producer.sh --security-protocol

2017-10-03 Thread Pekka Sarnila
Hi, kafka_2.11-0.11.0.0 If I try to give --security-protocol xyz (xyz any value e.g. SASL_PLAINTEXT, PLAINTEXTSASL, SASL_SSL) I get error security-protocol is not a recognized option Also having security.protocol=xyz in producer.properties gives error org.apache.kafka.common.KafkaExceptio

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Ted Yu
I think in producer.properties you should use: security.protocol=SASL_PLAINTEXT FYI On Tue, Oct 3, 2017 at 7:17 AM, Pekka Sarnila wrote: > Hi, > > kafka_2.11-0.11.0.0 > > If I try to give --security-protocol xyz (xyz any value e.g. > SASL_PLAINTEXT, PLAINTEXTSASL, SASL_SSL) I get error > > s

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Pekka Sarnila
The output below is actually from having security.protocol=SASL_PLAINTEXT in producer.properties. Actual error point I believe is: Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error: Line 5: expected [option key] at sun.security.provider.ConfigFile$S

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Manikumar
looks like syntax issue with "sasl.jaas.config" config property. On Tue, Oct 3, 2017 at 8:06 PM, Pekka Sarnila wrote: > The output below is actually from having > > security.protocol=SASL_PLAINTEXT > > in producer.properties. > > Actual error point I believe is: > > Caused by: java.lang.Securi

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Manikumar
looks like syntax issue with "sasl.jaas.config" config property or jaas conf file On Tue, Oct 3, 2017 at 8:12 PM, Manikumar wrote: > looks like syntax issue with "sasl.jaas.config" config property. > > On Tue, Oct 3, 2017 at 8:06 PM, Pekka Sarnila wrote: > >> The output below is actually from h

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Pekka Sarnila
Ah! Of course. That was it: missing ';' after last option (parser expected more options instead of '};' at line 5. Silly me. Thanks a lot. Pekka On 10/03/17 17:42, Manikumar wrote: looks like syntax issue with "sasl.jaas.config" config property. On Tue, Oct 3, 2017 at 8:06 PM, Pekka Sarni

Re: Kafka Streams Avro SerDe version/id caching

2017-10-03 Thread Ted Yu
I did a quick search in the code base - there doesn't seem to be caching as you described. On Tue, Oct 3, 2017 at 6:36 AM, Kristopher Kane wrote: > If using a Byte SerDe and schema registry in the consumer configs of a > Kafka streams application, does it cache the Avro schemas by ID and version

Re: Kafka Streams Avro SerDe version/id caching

2017-10-03 Thread Damian Guy
If you are using the confluent schema registry then the will be cached by the SchemaRegistryClient. Thanks, Damian On Tue, 3 Oct 2017 at 09:00 Ted Yu wrote: > I did a quick search in the code base - there doesn't seem to be caching as > you described. > > On Tue, Oct 3, 2017 at 6:36 AM, Kristop

Re: Using Kafka on DC/OS + Marathon

2017-10-03 Thread Valentin Forst
Hi Sean, Thanks a lot for this info ! Are you running DC/OS in prod? Regards Valentin > Am 03.10.2017 um 15:29 schrieb Sean Glover : > > Hi Valentin, > > Kafka is available on DC/OS in the Catalog (aka Universe) as part of the > `kafka` package. Mesosphere has put a lot of effort into makin

Re: SSL setup in Kafka 2.10.0.10.2.1 for keystore and truststore files

2017-10-03 Thread Awadhesh Gupta
Hi, I validated the client chain in server log after enabling the SSL log and it was showing entries of both the certificate in chain. I imported server csr (ca-cert file generated from command penssl req -new -x509 -keyout ca-key -out ca-cert -days $VALIDITY) to Client trust store and client csr

Hostname validation in Kafka for SSL authentication

2017-10-03 Thread Awadhesh Gupta
Hi, I have enabled SSL certificate betwen my Kafka server and client communicaiton with correct host name in CSR of certificates. I've multiple hosts (100-200) that can produce the messages on the broker and would liek to share the same certificates for each of the hosts. For this, I want to kn

Re: SSL setup in Kafka 2.10.0.10.2.1 for keystore and truststore files

2017-10-03 Thread Martin Gainty
From: Awadhesh Gupta Sent: Tuesday, October 3, 2017 1:44 PM To: users@kafka.apache.org; ja...@scholz.cz Subject: Re: SSL setup in Kafka 2.10.0.10.2.1 for keystore and truststore files Hi, I validated the client chain in server log after enabling the SSL log an

Re: Using Kafka on DC/OS + Marathon

2017-10-03 Thread Sean Glover
No, I don't. I help others that do :) On Tue, Oct 3, 2017 at 1:12 PM, Valentin Forst wrote: > Hi Sean, > > Thanks a lot for this info ! > Are you running DC/OS in prod? > > Regards > Valentin > > > Am 03.10.2017 um 15:29 schrieb Sean Glover : > > > > Hi Valentin, > > > > Kafka is available on D

Re: Kafka Streams Avro SerDe version/id caching

2017-10-03 Thread Svante Karlsson
I've implemented the same logic for a c++ client - caching is the only way to go since the performance impact of not doing it would be to big. So bet on caching on all clients. 2017-10-03 18:12 GMT+02:00 Damian Guy : > If you are using the confluent schema registry then the will be cached by > th

Re: SSL setup in Kafka 2.10.0.10.2.1 for keystore and truststore files

2017-10-03 Thread Jakub Scholz
> Regarding host name validation, does FQDN with hostname always present in CN (common name) of the certificate? What if I want to use some free form text in CSR for CN field to make it for multiple host? You have two options. Either you can use wildcard certificates as suggested by Martin. Or you