Re: Kafka broker slow down when consumer try to fetch large messages from topic

Prateek, hope you looked at compression? On Thu, Jun 2, 2016 at 10:26 AM, Tom Crayford wrote: > The article says ideal is about 10KB, which holds up well with what we've > seen in practice as well. > > On Thu, Jun 2, 2016 at 6:25 PM, prateek arora

Re: Kafka broker slow down when consumer try to fetch large messages from topic

Hi there, Firstly, a note that Kafka isn't really designed for this kind of large message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ covers a lot of tips around this use case however, and covers some tuning that will likely improve your usage. In particular, I expect tuning up

Kafka broker slow down when consumer try to fetch large messages from topic

I have 4 node kafka broker with following configuration : Default Number of Partitions : num.partitions : 1 Default Replication Factor : default.replication.factor : 1 Maximum Message Size : message.max.bytes : 10 MB Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB Right now I have