Hi Becket, Thanks for the kip. I think one of the risks here is that when compression estimation is disabled, you could have much smaller batches than expected, and throughput could be hurt. It would be worth adding this to the documentation of this setting.
Also, one of the rejected alternatives states that per topic estimations would not work when the compression of individual messages is variable. This is true in theory, but in practice one would expect Kafka topics to have fairly homogenous data, and hence should compress evenly. I was curious if you have data which shows otherwise. Thanks, Apurva On Tue, Feb 21, 2017 at 12:30 PM, Becket Qin <becket....@gmail.com> wrote: > Hi folks, > > I would like to start the discussion thread on KIP-126. The KIP propose > adding a new configuration to KafkaProducer to allow batching based on > uncompressed message size. > > Comments are welcome. > > The KIP wiki is following: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 126+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size > > Thanks, > > Jiangjie (Becket) Qin >
