Re: python and kafka - how to use as a queue

2014-02-08 Thread David Arthur
jsh, Please open an issue at https://github.com/mumrah/kafka-python so other users/devs of this library have visibility Thanks! -David On 1/16/14 9:18 PM, Jagbir Hooda wrote: Hi Arthur, I'm running into a very similar issue even with the latest version ( kafka-python @ V. 0.8.1_1 used with

RE: python and kafka - how to use as a queue

2014-01-16 Thread Jagbir Hooda
Hi Arthur, I'm running into a very similar issue even with the latest version ( kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka brokers). I've published 100 messages to the topic (see

Re: python and kafka - how to use as a queue

2014-01-15 Thread Jagbir Hooda
Hi Arthur, I'm running into a very similar issue even with the latest version ( kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka brokers). I've published 100 messages to the topic (see

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Montgomery
Great. I will follow your progress. I will use kafka 7.x in the meantime for development. I am sure you will announce in then group with the 8.x has a RC for your client? On Sun, Feb 17, 2013 at 1:23 PM, David Arthur wrote: > It is indeed pure python > > > On 2/17/13 12:20 AM, David Montgome

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Arthur
It is indeed pure python On 2/17/13 12:20 AM, David Montgomery wrote: Key issue with gevent is there can be no C bindings. If pure python then the sockets can be monkey patched as long as pure python code. I use gevent to run redis-py to make async calls to redis even though the client in natu

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Montgomery
Key issue with gevent is there can be no C bindings. If pure python then the sockets can be monkey patched as long as pure python code. I use gevent to run redis-py to make async calls to redis even though the client in nature is blocking. I do believe your client is pure python? Thanks On Su

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Arthur
Replying to both messages inline: On 2/16/13 9:07 PM, David Montgomery wrote: By the way..I assume that python-kafka is gevent safe? No idea, I've never used greenlet/gevent before. If the question is "are you doing anything unusual in kafka-python", then the answer is no. Just basic method ca

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Montgomery
By the way..I assume that python-kafka is gevent safe? Thanks On Sun, Feb 17, 2013 at 10:04 AM, David Montgomery < davidmontgom...@gmail.com> wrote: > Ok...great...I see now about offsets. I see how I can manage on > restarts. Thanks! > > So...considering I am in a disposable machine world th

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Montgomery
Ok...great...I see now about offsets. I see how I can manage on restarts. Thanks! So...considering I am in a disposable machine world then I will consider redis as a centralized store. Makes sense? What is the time frame for v8 release? On Sun, Feb 17, 2013 at 3:27 AM, David Arthur wr

Re: python and kafka - how to use as a queue

2013-02-16 Thread David Arthur
Greetings! I am the maintainer of kafka-python. Very cool to see it used in the wild. The kafka-python library supports the low-level protocols of Kafka 0.7 (Produce/Fetch/MultiProduce/MultiFetch). When you ask Kafka for messages via a Fetch request, you specify an offset + range (much like re

Re: python and kafka - how to use as a queue

2013-02-16 Thread Philip O'Toole
You need to read the Kafka design docs. Kafka does not delete messages just because a Consumer reads it. It does not track what messages have been consumed by any Consumer. It is up to Consumers to start off where they left off, by always asking for the right message (via offsets). Philip O