Vikash Mishra created KAFKA-15160: ------------------------------------- Summary: Message bytes duplication in Kafka headers when compression is enabled Key: KAFKA-15160 URL: https://issues.apache.org/jira/browse/KAFKA-15160 Project: Kafka Issue Type: Bug Components: clients, compression, consumer Affects Versions: 3.3.2, 3.2.3 Reporter: Vikash Mishra Attachments: java heap dump.png, wireshark-min.png
I created a spring Kafka consumer using @KafkaListener. During this, I encounter a scenario where when data is compressed ( any compression snappy/gzip) and consumed by the consumer then I see that in a heap dump, there is a " byte" occupying the same amount of memory as in Message value. This behavior is seen only in cases when compressed data is consumed by consumers not in the case of uncompressed data. Tried to capture Kafka's message through Wireshark, there it shows the proper size of data incoming from Kafka server & no extra bytes in headers. So, this is definitely something in Kafka client. Spring doesn't do any actions about compression; the whole functionality is done internally in the Kafka client library. Attached is the screenshot of the heap dump and Wireshark. This seems like a critical issue as message size in memory almost gets doubles impacting consumer memory and performance. Somewhere it feels like the actual message value is copied to headers? *To Reproduce* # Produce compressed data on any topic. # Create a simple consumer consuming from the above-created topic. # Capture heap dump. *Expected behavior* Headers should not show bytes consuming memory equivalent to value. -- This message was sent by Atlassian Jira (v8.20.10#820010)