I'm still not getting the necessary behavior. If I run on the command line, I
get a series of messages:
--------
$ ./kafka-console-consumer.sh --zookeeper w.x.y.z:p --topic test
--from-beginning
Test
test
tests
asdf
--------
If I exclude the --from-beginning argument then it hangs, which indicates to me
that the offset is currently at the end and awaiting new messages. If I run
through python, it also hangs, which is why I suspect it is insistently reading
from the end. See below:
--------
print "Begin constructing SimpleConsumer"
client = KafkaClient(servers)
s_consumer = SimpleConsumer(client,
topic,
group_id,
partitions=[0], #Not sure I need this
auto_commit=False, #Tried with and without,
doesn't fix the problem
auto_offset_reset='smallest' #Tried with and
without, doesn't fix the problem
)
print "End constructing SimpleConsumer\n"
print "Begin reading messages"
try:
for message in s_consumer:
print " New message"
print " " + message.topic
print " " + message.partition
print " " + message.offset
print " " + message.key
print " " + message.value
except Exception as e:
print "Exception: ", e
print "End reading messages\n"
print "End all"
--------
Output:
Begin all
Begin constructing SimpleConsumer
End constructing SimpleConsumer
Begin reading messages
--------
It just hangs after that. I also tried with a KafkaConsumer instead of a
SimpleConsumer and it does exactly the same thing. I'm not sure what to do.
----------------------------------------------------------------------------------------
Keith Wiley
Senior Software Engineer, Atigeo
[email protected]
________________________________________
From: Dana Powers <[email protected]>
Sent: Tuesday, July 28, 2015 09:58 PM
To: [email protected]
Subject: Re: kafka-python message offset?
Hi Keith,
you can use the `auto_offset_reset` parameter to kafka-python's
KafkaConsumer. It behaves the same as the java consumer configuration of
the same name. See
http://kafka-python.readthedocs.org/en/latest/apidoc/kafka.consumer.html#kafka.consumer.KafkaConsumer.configure
for more details on how to configure a KafkaConsumer instance.
For fine-grained control wrt configuring topic/partition offsets, use
KafkaConsumer.set_topic_partitions() . For the most control, pass a
dictionary of {(topic, partition): offset, ...} .
see
http://kafka-python.readthedocs.org/en/latest/apidoc/kafka.consumer.html#kafka.consumer.kafka.KafkaConsumer.set_topic_partitions
-Dana
On Tue, Jul 28, 2015 at 8:47 PM, Keith Wiley <[email protected]> wrote:
> I haven’t found a way to specify that a consumer should read from the
> beginning, or from any other explicit offset, or that the offset should be
> “reset” in any way. The command-line shell scripts (which I believe simply
> wrap the Scala tools) have flags for this sort of thing. Is there any way
> to do this through the python library?
>
> Thanks.
