Consumer Pause & Scheduled Resume

Hi, I have a requirement to have kafka streaming start at scheduled time and then pause the stream when the consumer poll returns empty fetches for 3 or more polls. I am starting a consumer poll loop during application startup using a singled thread executor and then pausing the consumer when the

Re: Open Source Schema Registry

Have a look at https://github.com/confluentinc/schema-registry > On Oct 23, 2018, at 9:28 AM, chinchu chinchu wrote: > > Hi folks, > We are looking to use open source schema registry with apache kafka 1.0.1 > and avro. *Do we need to write a

Open Source Schema Registry

Hi folks, We are looking to use open source schema registry with apache kafka 1.0.1 and avro. *Do we need to write a serialzer/deserialzier similar to confluent's KafkAvroSerializer to achieve this ?*.Our schemas are large ,so one of the reason that we are looking to use the registry is to decr

Re: RocksDB not closed on error during CachingKeyValueStore.flush?

Hi Patrik, Thanks for reporting this, it does indeed look like this could be an issue. I've created https://issues.apache.org/jira/browse/KAFKA-7534 to look into the issue and if required create a patch. -Bill On Tue, Oct 23, 2018 at 9:24 AM Patrik Kleindl wrote: > Hello > > Can someone pleas

RocksDB not closed on error during CachingKeyValueStore.flush?

Hello Can someone please verify if my assumption is correct? In CachingKeyValueStore, if an exception happens during flush() the store will not be closed properly. @Override public void flush() { lock.writeLock().lock(); try { cache.flush(cacheName); underlying.flush();

RE: kafka client 1.1.0 broker compatibility

Thanks Matthias. Your info was helpful. -Original Message- From: Matthias J. Sax Sent: 22 October 2018 22:15 To: users@kafka.apache.org Subject: Re: kafka client 1.1.0 broker compatibility If Logstash's internal client is 1.1.0, it is be compatible with Kafka brokers 2.0.0. Note: Brok

The limit on the number of consumers in a group.

Dear all. Is there any limit on the number of consumers in a group? I want to utilize about 300 or more consumers in a group, but rebalancing hangs and never get finished. When I invoke only 130~140 consumers in a group, it works fine. Buf from 150 consumers, rebalancing is never stopped. Is this