Is there a better way to do this where I can build one big requests with broker -> partition mappings and call a consumer.fetch() with one giant request?
Unfortunately, not until the 0.9 consumer is released. Until then, you can take a look at ZookeeperConsumerConnector on how it manages multiple SimpleConsumers. Thanks, Neha On Fri, Oct 24, 2014 at 1:46 PM, Rajiv Kurian <ra...@signalfuse.com> wrote: > I saw the example at > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > and it answered most of my questions. I am still trying to figure out the > pattern to be used when I want to use a single simple consumer to fetch > messages from different partitions (possibly from different offsets) and > possibly managed by different leaders. > > My use case right now is that I have a consumer which dynamically decides > which partitions it is responsible for and decides to fetch messages from > them at potentially different offsets. > > Right now it seems like I would need a new SimpleConsumer for each broker > since the SimpleConsumer takes the leadBroker in it's constructor. Then I'd > have to build a FetchRequest for each broker and ensure that the > addFetch(...) calls are made with partitions that correspond to the leader > broker that a SimpleConsumer is managing. Finally I'll need to make up to > numBrokers number of consumer.fetch(req) calls since each request is for a > separate broker. > > Is there a better way to do this where I can build one big requests with > broker -> partition mappings and call a consumer.fetch() with one giant > request? Otherwise if I am doing this in a single thread I have head of > line blocking with one request blocking another. > > Thanks! >
