Hi can anyone help with this? On Fri, Jan 29, 2016 at 11:50 PM, allen chan <allen.michael.c...@gmail.com> wrote:
> Use case: We are using kafka as broker in one of our elasticsearch > clusters. Kafka caches the logs if elasticsearch has any performance > issues. I have Kafka set to delete logs pretty quickly to keep things in > the file cache to limit IO. > > Questions: > 1. in 0.9 it seems like consumer offers are stored only in Kafka. Is there > a way to configure Kafka to delete my production logs pretty quickly but > have a different retention behavior for the consumer offsets? > > 2. Our consumer lag monitoring show us that a lot of times our consumers > are behind somewhere between 500 to 1000 messages. Looking at the JMX > metrics requestSizeAvg and requestSizeMax, it shows our average request > size is 500 bytes and max request size is 800,000 bytes. I assume the lag > is because that batch could only hold one message given the max is 1000000 > bytes. I plan to enable compression and increase the max.bytes to 10mb to > fix this short term. In a few blogs, people mentioned the ultimate fix > should be splitting the message into smaller chunks in the producer and > then having the consumer put it back together. Is that handled in the kafka > producer/consumer natively or has to be handled outside of it? > > Thanks for the attention. > Allen Chan > > > -- Allen Michael Chan
