This is hopefully my final question for a while. I noticed that compression is disabled by default. Why is this? My best guess is that compression doesn't work well for short messages <https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch>, which was maybe identified as the majority use-case for Kafka. But, producers batch records based on buffer/linger, and in my understanding the whole record batch is compressed together.
So, what's the deal? Should I turn on compression in production? Does it depends on my anticipated batch size? I am using Kafka Streams with a very low linger, so most of my batches will likely be very small. Thank you!
