Hi I would like to know if anyone has seen any benefit in specifying a compression.type for Kafka Producer sending avro binary (not JSON) encoded messages to Kafka cluster. I can appreciate the compression benefits for JSON but is there really any benefit in compressing binary avro data?
Kafka Producer has a user thread(s) and a sender thread and the user thread uses a compressor but I am not clear whether the whole batch is compressed or each message is compressed before adding to the batch. I imagine compressing a batch would provide better compression ratio compared to compressing individual messages? Does kafka compress individual messages or the whole batch of messages? Many thanks! S
