[
https://issues.apache.org/jira/browse/KAFKA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235679#comment-13235679
]
Jun Rao commented on KAFKA-309:
-------------------------------
+1 on the patch.
> Bug in FileMessageSet's append API can corrupt on disk log
> ----------------------------------------------------------
>
> Key: KAFKA-309
> URL: https://issues.apache.org/jira/browse/KAFKA-309
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Critical
> Attachments: kafka-309-test.patch, kafka-309.patch
>
>
> In FileMessageSet's append API, we write a ByteBufferMessageSet to a log in
> the following manner -
> while(written < messages.sizeInBytes)
> written += messages.writeTo(channel, 0, messages.sizeInBytes)
> In ByteBufferMessageSet, the writeTo API uses buffer.duplicate() to append to
> a channel -
> def writeTo(channel: GatheringByteChannel, offset: Long, size: Long): Long =
> channel.write(buffer.duplicate)
> If the channel doesn't write the ByteBuffer in one call, then we call it
> again until sizeInBytes bytes are written. But the next call will use
> buffer.duplicate() to write to the FileChannel, which will write the entire
> ByteBufferMessageSet again to the file.
> Effectively, we have a corrupted set of messages on disk.
> Thinking about it, FileChannel is a blocking channel, so ideally, the entire
> ByteBuffer should be written to the FileChannel in one call. I wrote a test
> (attached here) and saw that it does. But I'm not aware if there are some
> corner cases when it doesn't do so. In those cases, Kafka will end up
> corrupting on disk log segment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
