Hi Arthur,
I'm running into a very similar issue even with the latest version (
kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a
topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka
brokers). I've published 100 messages to the topic (see Reference below). Now
each time when I run the following consumer test
----------------------8<------------------------------------------
import logging
import time
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic", auto_commit=True,
auto_commit_every_n=10)
for message in consumer:
time.sleep(1)
print(message)
----------------------8<------------------------------------------
I get back all the 100 messages. You mentioned that with kafka 0.8 there will
be an offset stored in zookeeper (via broker) which will prevent consumers from
getting older messages. I'm curious how to use this feature. I also want to run
multiple consumers (on different machines) with exactly the same test code as
above and get only one message delivered to only one of the consumers in
"my-group" (multiple consumers per queue behavior).
Thanks,
jsh
REFERENCE:
I've used following code to publish messages
---------------------8<-------------------------------
rom kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
producer = SimpleProducer(kafka, "my-topic")
for i in range(0,100):
producer.send_messages("some message {0}".format(i))
---------------------8<-------------------------------
