Hi folks,
Reading through the Java examples for Kafka consumers, it appeared to
me as though I have the following options when it comes to setting up
consumers for a topic with more than one partition:
(a) Write a consumer that kicks off N threads, one thread per
partition, to read messages from the broker in parallel.
(b) Write a single threaded consumer and run N different JVMs, each
reading one of N partitions.
(c) Write a single threaded consumer that iterates through each
partition, using a consumer.timeout.ms setting to trigger a
timeout so that I don't block forever on a single partition that
is not being written to.
Are there are options? Is there a way to set up a non-blocking poll
of a KafkaStream to see whether or not it has a message available?
Does anyone actually do either (b) or (c), or is (a) really the only
realistic option? Is it possible/likely that a topic with N
partitions might have one partition that is never updated?
Jim
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
James A. Robinson [email protected]
Stanford University HighWire Press http://highwire.stanford.edu/
+1 650 7237294 (Work) +1 650 7259335 (Fax)
