I'm wondering how much of this can be done using careful system design vs building it within the consumer itself. You could distribute the several consumer instances across machines since it is built for distributed load balancing. That will sufficiently isolate the resources required to run the various consumers. But probably you have a specific use case in mind for running several consumer groups on the same machine. Would you mind giving more details?
On Thu, Oct 23, 2014 at 12:55 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Imagine exposing Kafka over various remoting protocols, where incoming > poll/read requests may come in concurrently for different consumer groups, > especially in a case with lots of different consumer groups. > If you create and destroy KafkaConsumer for each such request, response > times and throughput will be very low, and doing that is one of the ways to > reproduce https://issues.apache.org/jira/browse/KAFKA-1716 > > It would be better if one could reuse a (pool of) Consumer instances, and > through a read operation parameter specify for which consumer group should > read be performed. > > Kind regards, > Stevo Slavic. > > On Tue, Oct 14, 2014 at 6:17 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > > Stevo, > > > > The new consumer API is planned for 0.9, not 0.8.2. You can take a look > at > > a detailed javadoc here > > < > > > http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html > > > > > . > > > > Can you explain why you would like to poll messages across consumer > groups > > using just one instance? > > > > Thanks, > > Neha > > > > On Tue, Oct 14, 2014 at 1:03 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > > > Hello Apache Kafka community, > > > > > > Current (Kafka 0.8.1.1) high-level API's KafkaConsumer is not > lightweight > > > object, it's creation takes some time and resources, and it does not > seem > > > to be thread-safe. It's API also does not support reuse, for consuming > > > messages from different consumer groups. > > > > > > I see even in the coming (0.8.2) redesigned API it will not be possible > > to > > > reuse consumer instance to poll messages from different consumer > groups. > > > > > > Can something be done to support this? > > > > > > Would it help if there was consumer group as a separate entity from > > > consumer, for all the subscription management tasks? > > > > > > Kind regards, > > > Stevo Slavic > > > > > >
