Anyone? On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger < vinicius.scheideg...@gmail.com> wrote:
> Does anyone know how could I perform a load balance to distribute equally > the messages to all consumers within the same consumer group having > multiple producers? > > Is this a conceptual flaw on Kafka, wasn't it thought for equal > distribution with multiple producers or am I missing something? > I've asked on Stack Overflow, on Kafka users mailing group, here (on Kafka > Devs) and on Slack - and still have no definitive answer (actually most of > the time I got no answer at all) > > Would something like this even be possible in the way Kafka is currently > designed? > How does proposing for a KIP work? > > Thanks, > > > > On Thu, May 28, 2020, 3:44 PM Vinicius Scheidegger < > vinicius.scheideg...@gmail.com> wrote: > >> Hi, >> >> I'm trying to understand a little bit more about how Kafka works. >> I have a design with multiple producers writing to a single topic and >> multiple consumers in a single Consumer Group consuming message from this >> topic. >> >> My idea is to distribute the messages from all producers equally. From >> reading the documentation I understood that the partition is always >> selected by the producer. Is that correct? >> >> I'd also like to know if there is an out of the box option to assign the >> partition via a round robin *on the broker side *to guarantee equal >> distribution of the load - if possible to each consumer, but if not >> possible, at least to each partition. >> >> If my understanding is correct, it looks like in a multiple producer >> scenario there is lack of support from Kafka regarding load balancing and >> customers have to either stick to the hash of the key (random distribution, >> although it would guarantee same key goes to the same partition) or they >> have to create their own logic on the producer side (i.e. by sharing memory) >> >> Am I missing something? >> >> Thank you, >> >> Vinicius Scheidegger >> >
