Use case: We are using kafka as broker in one of our elasticsearch clusters. Kafka caches the logs if elasticsearch has any performance issues. I have Kafka set to delete logs pretty quickly to keep things in the file cache to limit IO.
Questions: 1. in 0.9 it seems like consumer offers are stored only in Kafka. Is there a way to configure Kafka to delete my production logs pretty quickly but have a different retention behavior for the consumer offsets? 2. Our consumer lag monitoring show us that a lot of times our consumers are behind somewhere between 500 to 1000 messages. Looking at the JMX metrics requestSizeAvg and requestSizeMax, it shows our average request size is 500 bytes and max request size is 800,000 bytes. I assume the lag is because that batch could only hold one message given the max is 1000000 bytes. I plan to enable compression and increase the max.bytes to 10mb to fix this short term. In a few blogs, people mentioned the ultimate fix should be splitting the message into smaller chunks in the producer and then having the consumer put it back together. Is that handled in the kafka producer/consumer natively or has to be handled outside of it? Thanks for the attention. Allen Chan
