Hi Liam, First of all, thanks for checking my doubt. I understand that the reason I notice this behavior is because our case differs from yours in one point: I'm not writing terabytes in a single day. I'm writing MB, but distribution matters in these MBs, this because processing in the consumer side is CPU intensive (complex machine learning algo), so a real equally distributed load is a requirement, otherwise messages start queuing (which is what I don't want).
Our system is also in production and this bad distribution is generating queuing in some consumers while others are idle. I tried both approaches: Round Robbin - actually gives me a distributed load in a big time window, but as I grow the number of producers I get several messages in a single partition while others don't get anything. Hash of the key - here we used a random UUID as the key - still a bad distribution [image: image.png] When I start, let's say 10 producers, we can get 5 messages going to one partition while others have none - i understand that this is because the round robin is internal to the producer. [image: image.png] (The picture above is what I believe is happening) it would surprise me that this hasn't come up before, that's why I'm pretty sure I'm missing something here... We're currently analyzing some solutions, one of them is building our own partitioner with shared memory (yes, that's how far we got on this), although I believe a better solution would be to have this on Kafka broker side and not depend on custom code. [image: image.png] Above is the draft of one of our current ideas of a possible design. Based on the shared memory we could decide the partition and send the messages directly there (the number of producers, partitions and consumers here are simplified). This if we don't find a solution within Kafka to mitigate this distribution issue - even though the thought design imposes limits and add layers we didn't had in our initial design. My question is, do we really need to develop this?? Is equal distribution in a scenario with multiple producers something that can be achieved in Kafka without custom development? Having never checked out the broker code when receiving a message - is partition definition even possible on the broker side? If this really does not exist, would a feature like that benefit other people, is it worth checking out instead of the above solution? Should I move this message to the dev forum? - no one gave me much attention there too (but maybe my messages are too big/boring - hahaha) Thanks On Tue, Jun 2, 2020 at 10:47 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Vinicius, > > As you note, the cluster doesn't load balance producers, it relies on them > using a partition strategy to do so. > > In production, I've never had actual broker load skew develop from multiple > independent producers using round robining - and we're talking say 20 - 50 > producers (depending on scaling) writing terabytes over a day. > > And load skew / hot brokers is something I monitor closely. > > The only time I've seen load skew is when a key based partition strategy > was used, and keys weren't evenly distributed. > > So in other words, in theory there's no guarantee, but in my experience, > round robining multiple producers works fine. > > Cheers, > > Liam Clarke > > On Mon, 1 Jun. 2020, 11:55 pm Vinicius Scheidegger, < > vinicius.scheideg...@gmail.com> wrote: > > > Hey guys, I need some help here... > > > > Is this a flaw in the design (maybe a discussion point for a KIP?), is > > Kafka not supposed to perform equal load balancing with multiple > producers > > or am I missing something (which is what I believe is happening)? > > > > On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger < > > vinicius.scheideg...@gmail.com> wrote: > > > >> Does anyone know whether we could really have an "out of the box" > >> solution to do round robin over the partitions when we have multiple > >> producers? > >> By that I mean, a round robin on the broker side (or maybe some way to > >> synchronize all producers). > >> > >> Thank you, > >> > >> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger < > >> vinicius.scheideg...@gmail.com> wrote: > >> > >>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound to > >>> the producer. In a scenario with multiple producers it doesn't > guarantee > >>> equal distribution - from what I understood and from my tests, the > >>> following situation happens with it: > >>> > >>> [image: image.png] > >>> > >>> Of course, the first partition is not always 1 and each producer may > >>> start in a different point in time, anyway my point is that it does not > >>> guarantee equal distribution. > >>> > >>> The other option pointed out is to select the partition myself - either > >>> a shared memory on the producers (assuming that this is possible - I > mean I > >>> would need to guarantee that producers CAN share a synchronized > memory) or > >>> include an intermediate topic with a single partition and a > >>> dispatcher/producer using RoundRobinPartitioner (but this would > include a > >>> single point of failure). > >>> > >>> [image: image.png] > >>> [image: image.png] > >>> > >>> None of these seem to be ideal as a Broker side round robin solution > >>> would. > >>> Am I missing something? Any other ideas? > >>> > >>> Thanks > >>> > >>> On Tue, May 26, 2020 at 11:34 AM M. Manna <manme...@gmail.com> wrote: > >>> > >>>> Hey Vinicius, > >>>> > >>>> > >>>> On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger < > >>>> vinicius.scheideg...@gmail.com> wrote: > >>>> > >>>> > In a scenario with multiple independent producers (imagine ephemeral > >>>> > dockers, that do not know the state of each other), what should be > the > >>>> > approach for the messages being sent to be equally distributed over > a > >>>> topic > >>>> > partition? > >>>> > > >>>> > From what I understood the partition election is always on the > >>>> Producer. Is > >>>> > this understanding correct? > >>>> > > >>>> > If that's the case, how should one achieve an equally distributed > load > >>>> > balancing (round robin) over the partitions in a scenario with > >>>> multiple > >>>> > producers? > >>>> > > >>>> > Thank you, > >>>> > > >>>> > Vinicius Scheidegger > >>>> > >>>> > >>>> Have you checked RoundRobinPartitioner ? Also, you can always specify > >>>> which partition you are writing to, so you can control the > partitioning > >>>> in > >>>> your way. > >>>> > >>>> Regards, > >>>> > >>>> Regards, > >>>> > >>>> > > >>>> > > >>>> > >>> >
