[
https://issues.apache.org/jira/browse/KAFKA-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-4298.
------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.10.0.2)
Issue resolved by pull request 2019
[https://github.com/apache/kafka/pull/2019]
> LogCleaner writes inconsistent compressed message set if topic message format
> != message format
> -----------------------------------------------------------------------------------------------
>
> Key: KAFKA-4298
> URL: https://issues.apache.org/jira/browse/KAFKA-4298
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.0.1
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
> Priority: Critical
> Fix For: 0.10.1.0
>
>
> When cleaning the log, we don't want to convert messages to the format
> configured for the topic due to KAFKA-3915. However, the cleaner logic for
> writing compressed messages (in case some messages in the message set were
> not retained) writes the topic message format version in the magic field of
> the outer message instead of the actual message format. The choice of the
> absolute/relative offset for the inner messages will also be based on the
> topic message format version.
> For example, if there is an old compressed message set with magic=0 in the
> log and the topic is configured for magic=1, then after cleaning, the new
> message set will have a wrapper with magic=1, the nested messages will still
> have magic=0, but the message offsets will be relative. If this happens,
> there does not seem to be an easy way to recover without manually fixing up
> the log.
> The offsets still work correctly as both the clients and broker use the outer
> message format version to decide if the relative offset needs to be converted
> to an absolute offset. So the main problem turns out to be that
> `ByteBufferMessageSet.deepIterator` throws an exception if there is a
> mismatch between outer and inner message format version.
> {code}
> if (newMessage.magic != wrapperMessage.magic)
> throw new IllegalStateException(s"Compressed message has magic
> value ${wrapperMessage.magic} " +
> s"but inner message has magic value ${newMessage.magic}")
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
