Thank you Ismael On Tue, Dec 12, 2017 at 11:49 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> Yes, the broker will store the batches it receives from the producer. By > default, it won't recompress, but it can be configured to recompress. > Either way, it won't merge multiple batches into a single one. > > Ismael > > On Tue, Dec 12, 2017 at 6:45 PM, Dmitry Minkovsky <dminkov...@gmail.com> > wrote: > > > I would like to follow up with this more concrete question: > > > > For the purpose of achieving decent compression ratios, I am having > > difficulty finding information on whether the broker can perform > additional > > record batching beyond the batching received from a producer. It seems > like > > in a low-latency application where the producer recorder buffer and > linger > > are small, it is difficult or impossible to achieve effective compression > > on the broker side, because the producer sending very small batches. > > > > So, am I correct in my understand that the broker appends record batches > > exactly as they are received from the producer? Or can it do its own > > batching to improve compression? > > > > Thank you, > > Dmitry > > > > On Sun, Dec 10, 2017 at 7:44 PM, Dmitry Minkovsky <dminkov...@gmail.com> > > wrote: > > > > > This is hopefully my final question for a while. > > > > > > I noticed that compression is disabled by default. Why is this? My best > > > guess is that compression doesn't work well for short messages > > > <https://www.elastic.co/blog/store-compression-in-lucene- > > and-elasticsearch>, > > > which was maybe identified as the majority use-case for Kafka. But, > > > producers batch records based on buffer/linger, and in my understanding > > the > > > whole record batch is compressed together. > > > > > > So, what's the deal? Should I turn on compression in production? Does > it > > > depends on my anticipated batch size? I am using Kafka Streams with a > > very > > > low linger, so most of my batches will likely be very small. > > > > > > Thank you! > > > > > > > > > > > >
