Re: v0.10 MirrorMaker producer cannot send v0.8 message from v0.10 broker

This is a known bug, I think it was fixed in the 0.10.0.1 release. You could alternatively use a custom message handler for the mirror maker, and then use the produce without a timestamp when the timestamp is -1 in the consuming message. On Thu, Sep 15, 2016 at 9:48 AM Samuel Zhou wrote: > Hi, >

SASL_PLAINTEXT Authentication/Connection failure

Hi, I am trying to get SASL_PLAINTEXT or SASL_SSL to work. Sofar I am not successful. I posted the full story on SO: http://stackoverflow.com/questions/39521691/kafka-authentication-producer-unable-to-connect-producer Bottom line is, when I start the server in SASL_PLAINTEXT mode, the below excep

Re: SASL_PLAINTEXT Authentication/Connection failure

Max, I think there is a typo in your configuration. You intended admin password to be admin-secret? KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="alice-secret" *=> Change to **"admin-secret"* use

Large # of Topics/Partitions

Hi, This might also be of interest: http://www.confluent .io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ Cheers, Jens On Monday, August 8, 2016, Daniel Fagnan wrote: > Thanks Tom! This was very helpful and I’ll explore having a more static > set of partitions as that

Lost message in asynchronous kafka producer

Hi, I have 2 brokers with a topic with replication factor = 2. Brokers are configured with min.insync.replicas=2. I use a producer in asynchronous mode to send 10 messages. After some seconds after producer start, I stop one broker. On producer side I got no exceptions, so for producer appli

Re: SASL_PLAINTEXT Authentication/Connection failure

Thanks Rajini. That was the issue. Now I am facing another one. I am not sure why my consumer is trying to use the topic in PLAINTEXT. The consumer config is: security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN KAFKA_OPTS is set to /home/kafka/kafka_client_jaas.conf. I can confirm that this fi

Re: SASL_PLAINTEXT Authentication/Connection failure

Max, You need to use the new consumer since the old consumer does not support security features. For console-consumer, you need to add the option --new-consumer. On Fri, Sep 16, 2016 at 10:14 AM, Max Bridgewater wrote: > Thanks Rajini. That was the issue. Now I am facing another one. I am not >

Re: SASL_PLAINTEXT Authentication/Connection failure

Awesome. After adding --new-consumer and --bootstrap-server, I got it working. Thanks a lot. On Fri, Sep 16, 2016 at 5:33 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Max, > > You need to use the new consumer since the old consumer does not support > security features. For console

Re: Publish to 1 topic, consume from N

Hello Marko, The difference is that the publisher has to know before hand to which queues to publish. With the AMQ model, you have a more decoupled model: The publisher only know the Exchange, and the consumers register themselves without the producer knowing about them. Therefore, let’s say yo

Schema for json converter

I am trying to use jdbc connector to send records from Kafka 0.9 to DB. I am using jsonConverter to convert the records. My connector is failing when its checking the Schema I am using. Please let me know what is the issue with my json schema. Configuration used: key.converter=org.apache.kafka.con

Slow machine disrupting the cluster

We just had an interesting issue, luckily this was only on our test cluster. Because of some reason one of the machines in a cluster became really slow. Because it was still alive, it stil was the leader for some topic-partitions. Our mirror maker reads and writes to multiple topic-partitions on ea

Error kafka-stream method punctuate in context.forward()

Good morning, I have a problem with a kafka-stream application. In fact I 've created already two kafka stream applications : StatsByMinute : entry topic : uplinks, out topic : statsM. StatsByHour : entrey topic : statsM, out topic : statsH. StatsByDay : entry topic : statsH, out topic : stats

Re: Slow machine disrupting the cluster

To remediate, you could start another broker, rebalance, and then shut down the busted broker. But, you really should put some monitoring on your system (to help diagnose the actual problem). Datadog has a pretty good set of articles for using jmx to do this: https://www.datadoghq.com/blog/mo

Slow 'rolled new log segment' ?

Hey! I'm trying to determine why I had a weird hiccup in publishing to my Kafka cluster. I'm running Kafka 0.10.0.1. Here's a graph of my producer times: You can see at around ~10:08 we had a large spike in kafka's internal publishing, waiting for the leader.

Re: Lost message in asynchronous kafka producer

You need to use one of the options you've already outlined. Without this you are just doing fire-and-forget. On Fri, 16 Sep 2016 at 09:25 Agostino Calamita wrote: > Hi, > I have 2 brokers with a topic with replication factor = 2. > Brokers are configured with min.insync.replicas=2. > > I use a p

java.nio.channels.ClosedChannelException in console-consumer.sh

I've created a 3 broker kafka cluster, changing only the config values for broker id, log.dirs, and zookeeper connect. I left the remaining fields as default. The broker ids are 1, 2, 3. I opened the port 9092 on AWS. I then created a topic 'test' with replication factor of 2, and 3 partitions.

Re: java.nio.channels.ClosedChannelException in console-consumer.sh

Some googling indicates that there are issues on AWS / EC2 when using the private IP, and its recommended to use the public ip as the advertised hostname instead. I have zookeeper and Kafka both running on EC2, and both are in the same availability zone, so both should be able to talk to each othe

Re: Exception while deserializing in kafka streams

Guozhang, Any clues on this one? Walter On Wed, Sep 14, 2016 at 9:46 PM, Walter rakoff wrote: > Guozhang, > > I am using 0.10.0.0. Could the below log be the cause? > > 16/09/14 17:24:35 WARN ConsumerConfig: The configuration > schema.registry.url = http://192.168.50.6:

Re: Performance issue with KafkaStreams

Is there a specific way that I need to build kafka for that to work? bash$ export INCLUDE_TEST_JARS=true; ./bin/kafka-run-class.sh org.apache.kafka.streams.perf.SimpleBenchmark Error: Could not find or load main class org.apache.kafka.streams.perf.SimpleBenchmark bash$ find . -name SimpleBenchma

Re: Time of derived records in Kafka Streams

On Sat, Sep 10, 2016 at 9:17 AM, Eno Thereska wrote: > > For aggregations, the timestamp will be that of the latest record being > aggregated. > How does that account for out of order records? What about kstream-kstream joins? The output from the join could be triggered by a record received fr

Upgrading from 0.8 to 0.10

We are planning to upgrade from 0.8 to 0.10. I am reading about possible performance impact of the upgrade. Do the performance hit due to conversation to 0.8 format or its related to something else? Would someone suggest best way to measure the performance hit? Thanks so much in advance, Vadim

any update on this?

https://issues.apache.org/jira/browse/KAFKA-1793 It would be great to use Consul instead of Zookeeper for Kafka and I think it would benefit Kafka a lot from the exponentially growing consul community.