Re: Failed on publish to Kafka

2018-05-23 Thread Omer Litov
Hi, Thanks, I will try it. I just don’t understand a couple of things: 1. how could reducing the buffer size can help here. 2. What is using a dummy topic relevant? Can’t I just continue to send to the existing topic? Thanks On 23/05/2018, 9:33 AM, "M. Manna" wrote: Can you try following?

Re: Failed on publish to Kafka

2018-05-22 Thread M. Manna
Can you try following? 1) reduce buffer size to exactly 2x of your batch size ? 2) back up and save you old properties file. 3) create a dummy topic 4) produce and consume messages using the new config. Let us know. On Wed, 23 May 2018 at 07:02, Omer Litov wrote: > Hi, > Thanks for the quic

Re: Failed on publish to Kafka

2018-05-22 Thread Omer Litov
Hi, Thanks for the quick response. What I meant is, that had those exceptions on both versions (0.10.1 and 1.0.1), so upgrading didn't solve the problem. And the same for increasing the cluster size. Here are the properties we set: "key.serializer" = "org.apache.kafka.common.serialization.Stri

Re: Failed on publish to Kafka

2018-05-22 Thread M. Manna
Hi, You said "On all of those setting there was no change in the described behavior" - this is slightly confusing. Could you please clarify this? If there is no change, that means everything is working :) ? >From the provided exception stack, it seems as if you are waiting to batch a lot of reque

Failed on publish to Kafka

2018-05-22 Thread Omer Litov
Hi, I have 4 producers, writing to a kafka cluster with 12 partitions. Periodically (about once per hour or two), we get exceptions while trying to send messages to kafka. The exceptions are “Expiring 12 record(s) for -: 49193 ms has passed since batch creation plus linger time”. We get that for