kafka 0.8.2 broker behaviour

2016-05-22 Thread Anishek Agarwal
Hello, we are using 4 kafka machines in production with 4 topics and each topic either 16/32 partitions and replication factor of 2. each machine has 3 disks for kafka logs. we see a strange behaviour where we see high disk usage spikes on one of the disks on all machines. it varies over time, wi

Kafka Streams change log behavior

2016-05-22 Thread Manu Zhang
Hi All, I'm new to Kafka Streams and have a question on change log. If a StreamTask fails and is restarted, will the change log be written to the old change log's topic partition ? Is it possible for some change log topic partition to have duplicate records so that log compaction is required ? T

Re: KAFKA-3470: treat commits as member heartbeats #1206

2016-05-22 Thread allen chan
Thank you for confirming! On Sunday, May 22, 2016, Guozhang Wang wrote: > Hello, > > KAFKA-3470 is a mainly a broker-side change, which handles the commit > request to also "reset" the timer for heartbeat as well. > > Guozhang > > On Sat, May 21, 2016 at 4:02 PM, allen chan > > wrote: > > > Hi,

Re: KAFKA-3470: treat commits as member heartbeats #1206

2016-05-22 Thread Guozhang Wang
Hello, KAFKA-3470 is a mainly a broker-side change, which handles the commit request to also "reset" the timer for heartbeat as well. Guozhang On Sat, May 21, 2016 at 4:02 PM, allen chan wrote: > Hi, > > Does anyone know if this is a broker side implementation or consumer side? > We deal with

KafkaStreams / Processor API - How to retrieve offsets

2016-05-22 Thread Florian Hussonnois
Hi everyone, Is there any way to retrieve the offset attached to each message with the APIs of kafka streams ? Currently the Processor interface exposes only the method process(K key, V value); It would be usefull to get the offsets in order to implement DSL extensions with some methods which nee

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

2016-05-22 Thread Kamal C
Andy, Kafka 0.9.0 server supports the previous versions of the clients (0.8.2, 0.8.1..). But, new clients won't work properly with the older version of Kafka server. You should upgrade your server / broker first. --Kamal On Fri, May 20, 2016 at 10:58 PM, Andy Davidson < a...@santacruzintegratio