[ https://issues.apache.org/jira/browse/KAFKA-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235636#comment-13235636 ]
Jun Rao commented on KAFKA-310: ------------------------------- ByteBufferMessageSet.validBytes currently makes a deep iteration of all messages, which means that we need to decompress messages. To avoid this overhead, we should change ByteBufferMessageSet.validBytes to use a shallow iterator. > Incomplete message set validation checks in kafka.log.Log's append API can > corrupt on disk log > ---------------------------------------------------------------------------------------------- > > Key: KAFKA-310 > URL: https://issues.apache.org/jira/browse/KAFKA-310 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.7 > Reporter: Neha Narkhede > Priority: Critical > Attachments: kafka-310.patch > > > The behavior of the ByteBufferMessageSet's iterator is to ignore and return > false if some trailing bytes are found that cannot be de serialized into a > Kafka message. The append API in Log, iterates through a ByteBufferMessageSet > and validates the checksum of each message. Though, while appending data to > the log, it just uses the underlying ByteBuffer that forms the > ByteBufferMessageSet. Now, due to some bug, if the ByteBuffer has some > trailing data, that will get appended to the on-disk log too. This can cause > corruption of the log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira