I dug deeper and saw this during normal operation: In the kafka broker log:
[2014-11-03 21:39:25,658] ERROR [KafkaApi-8] Error when processing fetch request for partition [activity.stream,5] offset 7475239 from consumer with correlation id 69 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRangeException: Request for offset 7475239 but we only have log segments in the range 8014610 to 10227768. And on the client side I saw: Nov 03 21:39:25 INFO kafka consumer.py: Commit offset 10227769 in SimpleConsumer: group=ActivityStream, topic=activity.stream, partition=5 ... ... Nov 03 21:39:26 ERROR demux.consumer_stream consumer_stream.py: OffsetOutOfRangeError(FetchResponse(topic='activity.stream', partition=5, error=1, highwaterMark=-1, messages=<generator object _decode_message_set_iter at 0x3dcf870>),) I am using the python kafka client. Why would the kafka client commit offset 10227769, and then, 1 second later, turn around and ask kafka for the offset 7475239? thx jim
