Hi Arthur,
I'm running into a very similar issue even with the latest version (
kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a
topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka
brokers). I've published 100 messages to the topic (see Reference below). Now
each time when I run the following consumer test
----------------------8<------------------------------------------import
loggingimport timefrom kafka.client import KafkaClientfrom kafka.consumer
import SimpleConsumerfrom kafka.producer import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic", auto_commit=True,
auto_commit_every_n=10)
for message in consumer: time.sleep(1)
print(message)----------------------8<------------------------------------------
I get back all the 100 messages. You mentioned that with kafka 0.8 there will
be an offset stored in zookeeper (via broker) which will prevent consumers from
getting older messages. I'm curious how to use this feature. I also want to run
multiple consumers (on different machines) with exactly the same test code as
above and get only one message delivered to only one of the consumers in
"my-group" (multiple consumers per queue behavior).
Thanks,jsh
REFERENCE:
I've used following code to publish
messages---------------------8<-------------------------------rom kafka.client
import KafkaClientfrom kafka.consumer import SimpleConsumerfrom kafka.producer
import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
producer = SimpleProducer(kafka, "my-topic")for i in range(0,100):
producer.send_messages("some message
{0}".format(i))---------------------8<-------------------------------
