Hi, I can't see your images, they're not making it through the mailing list, sorry.
On Tue, 2 Jun. 2020, 11:09 pm Vinicius Scheidegger, < vinicius.scheideg...@gmail.com> wrote: > Hi Liam, > > First of all, thanks for checking my doubt. > I understand that the reason I notice this behavior is because our case > differs from yours in one point: I'm not writing terabytes in a single day. > I'm writing MB, but distribution matters in these MBs, this > because processing in the consumer side is CPU intensive (complex machine > learning algo), so a real equally distributed load is a requirement, > otherwise messages start queuing (which is what I don't want). > > Our system is also in production and this bad distribution is generating > queuing in some consumers while others are idle. > I tried both approaches: > Round Robbin - actually gives me a distributed load in a big time window, > but as I grow the number of producers I get several messages in a single > partition while others don't get anything. > Hash of the key - here we used a random UUID as the key - still a bad > distribution > > > [image: image.png] > > > > When I start, let's say 10 producers, we can get 5 messages going to one > partition while others have none - i understand that this is because the > round robin is internal to the producer. > [image: image.png] > > (The picture above is what I believe is happening) > > > it would surprise me that this hasn't come up before, that's why I'm > pretty sure I'm missing something here... > We're currently analyzing some solutions, one of them is building our own > partitioner with shared memory (yes, that's how far we got on this), > although I believe a better solution would be to have this on Kafka broker > side and not depend on custom code. > > [image: image.png] > Above is the draft of one of our current ideas of a possible design. Based > on the shared memory we could decide the partition and send the messages > directly there (the number of producers, partitions and consumers here are > simplified). > This if we don't find a solution within Kafka to mitigate this > distribution issue - even though the thought design imposes limits and add > layers we didn't had in our initial design. > > My question is, do we really need to develop this?? > Is equal distribution in a scenario with multiple producers something that > can be achieved in Kafka without custom development? > Having never checked out the broker code when receiving a message - is > partition definition even possible on the broker side? > If this really does not exist, would a feature like that benefit other > people, is it worth checking out instead of the above solution? > Should I move this message to the dev forum? - no one gave me much > attention there too (but maybe my messages are too big/boring - hahaha) > > Thanks > > > On Tue, Jun 2, 2020 at 10:47 AM Liam Clarke-Hutchinson < > liam.cla...@adscale.co.nz> wrote: > >> Hi Vinicius, >> >> As you note, the cluster doesn't load balance producers, it relies on them >> using a partition strategy to do so. >> >> In production, I've never had actual broker load skew develop from >> multiple >> independent producers using round robining - and we're talking say 20 - 50 >> producers (depending on scaling) writing terabytes over a day. >> >> And load skew / hot brokers is something I monitor closely. >> >> The only time I've seen load skew is when a key based partition strategy >> was used, and keys weren't evenly distributed. >> >> So in other words, in theory there's no guarantee, but in my experience, >> round robining multiple producers works fine. >> >> Cheers, >> >> Liam Clarke >> >> On Mon, 1 Jun. 2020, 11:55 pm Vinicius Scheidegger, < >> vinicius.scheideg...@gmail.com> wrote: >> >> > Hey guys, I need some help here... >> > >> > Is this a flaw in the design (maybe a discussion point for a KIP?), is >> > Kafka not supposed to perform equal load balancing with multiple >> producers >> > or am I missing something (which is what I believe is happening)? >> > >> > On Wed, May 27, 2020 at 2:40 PM Vinicius Scheidegger < >> > vinicius.scheideg...@gmail.com> wrote: >> > >> >> Does anyone know whether we could really have an "out of the box" >> >> solution to do round robin over the partitions when we have multiple >> >> producers? >> >> By that I mean, a round robin on the broker side (or maybe some way to >> >> synchronize all producers). >> >> >> >> Thank you, >> >> >> >> On Tue, May 26, 2020 at 1:41 PM Vinicius Scheidegger < >> >> vinicius.scheideg...@gmail.com> wrote: >> >> >> >>> Yes, I checked it. The issue is that RoundRobbinPartitioner is bound >> to >> >>> the producer. In a scenario with multiple producers it doesn't >> guarantee >> >>> equal distribution - from what I understood and from my tests, the >> >>> following situation happens with it: >> >>> >> >>> [image: image.png] >> >>> >> >>> Of course, the first partition is not always 1 and each producer may >> >>> start in a different point in time, anyway my point is that it does >> not >> >>> guarantee equal distribution. >> >>> >> >>> The other option pointed out is to select the partition myself - >> either >> >>> a shared memory on the producers (assuming that this is possible - I >> mean I >> >>> would need to guarantee that producers CAN share a synchronized >> memory) or >> >>> include an intermediate topic with a single partition and a >> >>> dispatcher/producer using RoundRobinPartitioner (but this would >> include a >> >>> single point of failure). >> >>> >> >>> [image: image.png] >> >>> [image: image.png] >> >>> >> >>> None of these seem to be ideal as a Broker side round robin solution >> >>> would. >> >>> Am I missing something? Any other ideas? >> >>> >> >>> Thanks >> >>> >> >>> On Tue, May 26, 2020 at 11:34 AM M. Manna <manme...@gmail.com> wrote: >> >>> >> >>>> Hey Vinicius, >> >>>> >> >>>> >> >>>> On Tue, 26 May 2020 at 10:27, Vinicius Scheidegger < >> >>>> vinicius.scheideg...@gmail.com> wrote: >> >>>> >> >>>> > In a scenario with multiple independent producers (imagine >> ephemeral >> >>>> > dockers, that do not know the state of each other), what should be >> the >> >>>> > approach for the messages being sent to be equally distributed >> over a >> >>>> topic >> >>>> > partition? >> >>>> > >> >>>> > From what I understood the partition election is always on the >> >>>> Producer. Is >> >>>> > this understanding correct? >> >>>> > >> >>>> > If that's the case, how should one achieve an equally distributed >> load >> >>>> > balancing (round robin) over the partitions in a scenario with >> >>>> multiple >> >>>> > producers? >> >>>> > >> >>>> > Thank you, >> >>>> > >> >>>> > Vinicius Scheidegger >> >>>> >> >>>> >> >>>> Have you checked RoundRobinPartitioner ? Also, you can always >> specify >> >>>> which partition you are writing to, so you can control the >> partitioning >> >>>> in >> >>>> your way. >> >>>> >> >>>> Regards, >> >>>> >> >>>> Regards, >> >>>> >> >>>> > >> >>>> > >> >>>> >> >>> >> >
