[
https://issues.apache.org/jira/browse/KAFKA-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131142#comment-15131142
]
Dana Powers commented on KAFKA-3200:
------------------------------------
the minimum Message payload size is 26 bytes (8 offset, 4 size, 14 for an
'empty' message), so generally I would break if there are less than 26 bytes
left and then also break if the decoded size is larger than the remaining
buffer.
for reference, the code I wrote to handle message set decoding in kafka-python
is here:
https://github.com/dpkp/kafka-python/blob/master/kafka/protocol/message.py#L123-L158
> MessageSet from broker seems invalid
> ------------------------------------
>
> Key: KAFKA-3200
> URL: https://issues.apache.org/jira/browse/KAFKA-3200
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0
> Environment: Linux, running Oracle JVM 1.8
> Reporter: Rajiv Kurian
>
> I am writing a java consumer client for Kafka and using the protocol guide at
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> to parse buffers. I am currently running into a problem parsing certain
> fetch responses. Many times it works fine but some other times it does not.
> It might just be a bug with my implementation in which case I apologize.
> My messages are uncompressed and exactly 23 bytes in length and has null
> keys. So each Message in my MessageSet is exactly size 4 (crc) +
> 1(magic_bytes) + 1 (attributes) + 4(key_num_bytes) + 0 (key is null) +
> 4(num_value_bytes) + 23(value_bytes) = 37 bytes.
> So each element of the MessageSet itself is exactly 37 (size of message) + 8
> (offset) + 4 (message_size) = 49 bytes.
> In comparison an empty message set element should be of size 8 (offset) + 4
> (message_size) + 4 (crc) + 1(magic_bytes) + 1 (attributes) + 4(key_num_bytes)
> + 0 (key is null) + 4(num_value_bytes) + 0(value is null) = 26 bytes
> I occasionally receive a MessageSet which says size is 1000. A size of 1000
> is not divisible by my MessageSet element size which is 49 bytes. When I
> parse such a message set I can actually read 20 of message set elements(49
> bytes) which is 980 bytes. I have 20 extra bytes to parse now which is
> actually less than even an empty message (26 bytes). At this moment I don't
> know how to parse the messages any more.
> I will attach a file for a response that can actually cause me to run into
> this problem as well as the sample ccde.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
