Thank you very much.
I will check that, but it does seem to stop on the same message.
Jörg
On 01.10.2015 22:49, Lance Laursen wrote:
Hey Jörg,
Unfortunately when the high level consumer hits a corrupt message, it
enters an invalid state and closes. The only way around this is to iterate
your offset by 1 in order to skip the corrupt message. This is currently
not automated. You can catch this exception if you are using the simple
consumer client, but unfortunately mirrormaker uses the high level client.
There have been some corrupt producer message bugs related to using snappy
compression recently, but this does not seem to be the same as your problem.
Does MM stop on the exact same message each time (
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker
)? I would suggest triple checking that your configurations are the same
across all DC's (you mentioned that MM mirrors successfully to another DC
with no problem), as well as examine the problem message to see if you can
find anything different about it when compared to the others (See:
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-SimpleConsumerShell
). Your only other recourse is to iterate past the problem offset.
On Thu, Oct 1, 2015 at 1:22 AM, Jörg Wagner <joerg.wagn...@1und1.de> wrote:
Hey everyone,
I've been having some issues with corrupted messages and mirrormaker as I
wrote previously. Since there was no feedback, I want to ask a new question:
Did you ever have corrupted messages in kafka? Did things break? How did
you recover or work around that?
Thanks
Jörg
--
Mit freundlichem Gruß
Jörg Wagner
Systemadministrator
Search & Account Security
1&1 Mail & Media Development & Technology GmbH | Sapporobogen 6-8 | 80637
München | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagn...@1und1.de | Web: www.1und1.de
Amtsgericht Montabaur, HRB 5452
Geschäftsführer: Frank Einhellinger, Hans-Henning Kettler, Jan Oetjen
Member of United Internet
