We often replay data days old, and have never seen any issues like
this. We are running 0.72.

Philip

On Mon, Jun 10, 2013 at 11:17 AM, Todd Bilsborrow
<[email protected]> wrote:
> We've been running Kafka 0.7.0 in production for several months and have been 
> quite happy. Our use case to date has been to pull from the head of our 
> topics, so we're normally consuming within seconds of message production 
> using the high level consumer which is working great as far as I can tell.
>
> Recently I've started pulling older data (usually a few hours old) using the 
> low level consumer, and I'm running into what appears to be corruption in the 
> data files. The consumer pauses for several seconds and then throws 
> "java.io.EOFException: Received -1 when reading from channel, socket has 
> likely been closed". The server log shows "ERROR Closing socket for 
> /xx.xx.xx.xx because of error (kafka.network.Processor) java.io.IOException: 
> Input/output error". And DumpLogSegments reads up to the problematic offset 
> and then stops and reports that the tail of the log is at offset: <bad 
> offset> even though there is more data in the file (the next segment file's 
> starting offset is much higher).
>
> I learned here on the mailing list last month that I can skip the rest of the 
> corrupted segment, but I'd rather not be doing that because then I'm losing 
> messages. This has happened 5-6 times in the past month; I've seen it on 
> different brokers, different topics, different partitions, and different 
> segments.
>
> So finally, my questions are:
>
> - is anyone else pulling older data without issues, or is everyone pretty 
> much always consuming as fast as possible?
> - is there a known bug that would be fixed with an upgrade to a newer kafka 
> version? I don't know if it's the same problem but I see jira 309 and 310 are 
> marked as fixed but I don't know in which version
> - is there any way to examine a corrupt file to see what went wrong? Or any 
> way to diagnose why it's happening?
>
> Thanks!

Reply via email to