jsh,
Please open an issue at https://github.com/mumrah/kafka-python so other
users/devs of this library have visibility
Thanks!
-David
On 1/16/14 9:18 PM, Jagbir Hooda wrote:
Hi Arthur,
I'm running into a very similar issue even with the latest version (
kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a
topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka
brokers). I've published 100 messages to the topic (see Reference below). Now
each time when I run the following consumer test
----------------------8<------------------------------------------
import logging
import time
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic", auto_commit=True,
auto_commit_every_n=10)
for message in consumer:
time.sleep(1)
print(message)
----------------------8<------------------------------------------
I get back all the 100 messages. You mentioned that with kafka 0.8 there will be an
offset stored in zookeeper (via broker) which will prevent consumers from getting older
messages. I'm curious how to use this feature. I also want to run multiple consumers (on
different machines) with exactly the same test code as above and get only one message
delivered to only one of the consumers in "my-group" (multiple consumers per
queue behavior).
Thanks,
jsh
REFERENCE:
I've used following code to publish messages
---------------------8<-------------------------------
rom kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer
kafka = KafkaClient("kafkabroker2", 9092)
producer = SimpleProducer(kafka, "my-topic")
for i in range(0,100):
producer.send_messages("some message {0}".format(i))
---------------------8<-------------------------------