James Ranson created KAFKA-3772:
-----------------------------------
Summary: MirrorMaker crashes on Corrupted Message
Key: KAFKA-3772
URL: https://issues.apache.org/jira/browse/KAFKA-3772
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.9.0.1
Reporter: James Ranson
We recently came across an issue where a message on our source kafka cluster
became corrupted. When MirrorMaker tried to consume this message, the thread
crashed and caused the entire process to also crash. Each time we attempted to
restart MM, it crashed on the same message. There is no information in the MM
logs about which message it was trying to consume (what topic, what offset,
etc). So the only way we were able to get past the issue was to go into the
zookeeper tree for our mirror consumer group and increment the offset for every
partition on every topic until the MM process could start without crashing.
This is not a tenable operational solution. MirrorMaker should gracefully skip
corrupt messages since they will never be able to be replicated anyway.
```2016-05-26 20:02:26,787 FATAL MirrorMaker$MirrorMakerThread - [{}]
[mirrormaker-thread-3] Mirror maker thread failure due to
kafka.message.InvalidMessageException: Message is corrupt (stored crc =
33747148, computed crc = 3550736267)
at kafka.message.Message.ensureValid(Message.scala:167)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at
kafka.tools.MirrorMaker$MirrorMakerOldConsumer.hasData(MirrorMaker.scala:483)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:394)
2016-05-26 20:02:27,580 FATAL MirrorMaker$MirrorMakerThread - [{}]
[mirrormaker-thread-3] Mirror maker thread exited abnormally, stopping the
whole mirror maker.```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
