My experience is solely with 0.72. More inline. On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites <[email protected]> wrote: > Hello everyone, > > I have a very low volume topic that has 2 consumers in the same group. How do > I get each consumer to only consume 1 message at a time and if the the first > consumer is busy get the other consumer to consume the message?
You can't, not if you only have one partition. Each consumer is dedicated to a single partition. Unless you deliberately tear down the consumer and let another take over that partition (if you are using the high-level consumer). > > Currently what I am doing is: > > First consumer connects to Kafka waits for 300 milliseconds then disconnects, > waits for 10 seconds, then reconnects to see if there is a waiting message. I don't think you need to do this. The high-level has a API that allows you to set this timeout (I think). > > The messages kick off a long task on each server, each server can handle > multiple tasks up to a limit so first I am trying to balance the tasks across > multiple servers and if they are maxed out don't consume any messages. > > This will give the other server or servers a chances to pickup a message and > do the task. > > I would not disconnect if I can ensure I don't have messages waiting in the > queue for a server to consume them without the other servers being able to > see them. I think a better design would be to have a basic consumer that drains the topic and hands jobs to the set of available workers. *Those* workers perform the long-running job. Only if there are no available workers does the consumer block. You may be trying to do too much in the consumer. > > Thanks for the help... > > Cheers, > Eric Sites > >
