Blake Smith created KAFKA-980:
---------------------------------
Summary: Crash during log recovery can cause full recovery to
never run
Key: KAFKA-980
URL: https://issues.apache.org/jira/browse/KAFKA-980
Project: Kafka
Issue Type: Bug
Affects Versions: 0.7.1, 0.8
Reporter: Blake Smith
After an unclean shutdown of the Kafka server, if the broker throws an
unhandled exception during log recovery, the broker can get in a state where
recovery never runs on a log file.
We saw this problem manifest in production and is summarized on the mailing
list here:
http://mail-archives.apache.org/mod_mbox/kafka-users/201307.mbox/%3CCAKSpikjgp2sW2ycuf86JrjtAPxWBp92OOEmigVed=u=jfop...@mail.gmail.com%3E
Because recovery state is not tracked explicitly, our kafka broker started
writing data even when the log files were not fully recovered. It feels to me
like a separate state flag for recovery should also be tracked in cases where
recovery does not fully run. What do you guys think?
Steps to reproduce:
1. Shutdown the kafka broker
2. Create a directory named 'bogus' under the kafka log directory (won't parse
since it has no partition number)
3. Remove .kafka_cleanshutdown from the log directory to force a recovery
4. Start the kafka broker, observe:
- Recovery will run on partition segments until it reaches the bogus
directory
- Exception will be thrown during log loading from the bogus directory
- Kafka will initiate a clean shutdown after the exception is thrown
5. Once the Kafka server is cleanly shutdown, start it again, observe:
- Recovery will not try to run, since kafka was shutdown cleanly
- Some partition log files have never been recovered
6. Remove the bogus log directory
7. Start Kafka broker, observe:
- Recovery will not run
- Kafka will start cleanly and begin accepting writes again, even though
recovery has never run and logs might be in a corrupt state
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
