Rajiv Kurian created KAFKA-3200:
-----------------------------------
Summary: MessageSet from broker seems invalid
Key: KAFKA-3200
URL: https://issues.apache.org/jira/browse/KAFKA-3200
Project: Kafka
Issue Type: Bug
Affects Versions: 0.9.0.0
Environment: Linux, running Oracle JVM 1.8
Reporter: Rajiv Kurian
I am writing a java consumer client for Kafka and using the protocol guide at
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
to parse buffers. I am currently running into a problem parsing certain fetch
responses. Many times it works fine but some other times it does not. It might
just be a bug with my implementation in which case I apologize.
My messages are uncompressed and exactly 23 bytes in length and has null keys.
So each Message in my MessageSet is exactly size 4 (crc) + 1(magic_bytes) + 1
(attributes) + 4(key_num_bytes) + 0 (key is null) + 4(num_value_bytes) +
23(value_bytes) = 37 bytes.
So each element of the MessageSet itself is exactly 37 (size of message) + 8
(offset) + 4 (message_size) = 49 bytes.
In comparison an empty message set element should be of size 8 (offset) + 4
(message_size) + 4 (crc) + 1(magic_bytes) + 1 (attributes) + 4(key_num_bytes) +
0 (key is null) + 4(num_value_bytes) + 0(value is null) = 26 bytes
I occasionally receive a MessageSet which says size is 1000. A size of 1000 is
not divisible by my MessageSet element size which is 49 bytes. When I parse
such a message set I can actually read 20 of message set elements(49 bytes)
which is 980 bytes. I have 20 extra bytes to parse now which is actually less
than even an empty message (26 bytes). At this moment I don't know how to parse
the messages any more.
I will attach a file for a response that can actually cause me to run into this
problem as well as the sample ccde.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
