Re: python and kafka - how to use as a queue

[David Arthur]

jsh,

Please open an issue at https://github.com/mumrah/kafka-python so other 
users/devs of this library have visibility

Thanks!
-David

On 1/16/14 9:18 PM, Jagbir Hooda wrote:
Hi Arthur,

I'm running into a very similar issue even with the latest version ( 
kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a 
topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka 
brokers). I've published 100 messages to the topic (see Reference below). Now 
each time when I run the following consumer test

----------------------8<------------------------------------------
import logging
import time
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer

kafka = KafkaClient("kafkabroker2", 9092)

consumer = SimpleConsumer(kafka, "my-group", "my-topic", auto_commit=True, 
auto_commit_every_n=10)

for message in consumer:
     time.sleep(1)
     print(message)
----------------------8<------------------------------------------

I get back all the 100 messages. You mentioned that with kafka 0.8 there will be an 
offset stored in zookeeper (via broker) which will prevent consumers from getting older 
messages. I'm curious how to use this feature. I also want to run multiple consumers (on 
different machines) with exactly the same test code as above and get only one message 
delivered to only one of the consumers in "my-group" (multiple consumers per 
queue behavior).

Thanks,
jsh

REFERENCE:

I've used following code to publish messages
---------------------8<-------------------------------
rom kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
from kafka.producer import SimpleProducer, KeyedProducer

kafka = KafkaClient("kafkabroker2", 9092)

producer = SimpleProducer(kafka, "my-topic")
for i in range(0,100):
     producer.send_messages("some message {0}".format(i))
---------------------8<-------------------------------