Yes, there is a 10 byte overhead per message. Here is the exact message format.
message length : 4 bytes (value: 1+1+4+n) "magic" value : 1 byte compression attr: 1 byte crc : 4 bytes payload : n bytes The data sent over the wire also include topic name, partition name, total length of the request, in addition to the message. Those are shared by messages in the same message set. So, if you send more messages in a single produce request, you should see overhead per message closer to 10 bytes. Thanks, Jun On Wed, Oct 26, 2011 at 12:12 AM, Bao Thai Ngo <[email protected]> wrote: > Hi, > > We have just made inspection on the message format and found some > information unclear to us. Before the investigation, I did know that each > Kafka message has an overhead of only 10 bytes. > > Below is what we have done with Kafka 0.7 RC2 on a CentOS 5.6 server: > - start kafka server on port 4444 > - tcpdump -i lo -w test.pcap port 4444 and host localhost -vvvvv > - start kafka producer > - have kafka producer send some sort of message (hello, aaa) to the server > > We found 2 important items: > 1. Before sending an actual message, kafka producer do send a (control) > message of 4 bytes to the server. Kafka producer always does this action > before send some message to the server. > 2. Sending a message of 15 bytes (10 bytes for overhead + 5 bytes for the > message payload) from the producer to the server gives us an IP packet of 83 > bytes (IP header: 20, TCP header: 32, Data: 31 bytes). The data of the IP > packet is 31 instead of 15 bytes > > Could you please give us a detailed explanation about these 2 items? Please > also see the attachment for further information. > > Thanks, > ~Thai > > > > >
