Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-26 Thread Mathieu Amblard
Hi Sophie and David, Thank you both for your replies, You are in the best position to know if this can be useful or not ;) and I completely agree with your comments, it seems to be too specific to our use cases. Therefore, I propose to end the discussion and see if in the future any other people

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-25 Thread Sophie Blee-Goldman
Thanks for clarifying Mathieu, I hadn't seen your PR yet and assumed from your earlier comments that this was still a WIP. I do tend to agree with David on this though, I was also struggling to understand the use case/motivation and felt it was quite specific. As I think David said in his PR

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-25 Thread David Jacot
Hi Mathieu, Thanks for the effort that you have put in creating this KIP. I just read it again and I am still confused by the use cases and the motivation. I suppose that this works for your data model but it does not seem to be a general pattern. Overall, I stick to the comment that I made in

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-24 Thread Mathieu Amblard
Hi Sophie, Thank you for your message. The TopicRoundRobinAssignor has been already implemented (because the existing ones do not address our use cases correctly), it implements the interface you mentionned, and I have made a PR to the Kafka repo where you can see the source code. It was in a

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-24 Thread Sophie Blee-Goldman
Hey Mathieu, Apologies if you already know this, but the partition assignor interface is fully pluggable. That means you can plug in a custom PartitionAssignor implementation, without having to go through a KIP or commit any code to the AK repo. I suspect some of the current unknowns and

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-21 Thread Mathieu Amblard
Hello everybody, Just to let you know that I have added a chapter about having multiple containers (multiple pods for Kubernetes) running the same application :

Re: [DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-11 Thread Mathieu Amblard
Hi Hector, First, thank you for your questions ! *If the goal is to do the partition assignments at a topic level, wouldn't having single-partition topics solve this problem?* We are in a microservices environment; therefore, we can have multiple containers running the same application.

Re:[DISCUSS] KIP-874: TopicRoundRobinAssignor

2022-10-07 Thread Hector Geraldino (BLOOMBERG/ 919 3RD A)
Hi Mathieu. I took a look at your KIP and have a couple questions. If the goal is to do the partition assignments at a topic level, wouldn't having single-partition topics solve this problem? You also mentioned that your goal is to minimize the potential of a poison pill message breaking all