[
https://issues.apache.org/jira/browse/KAFKA-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mathieu Amblard updated KAFKA-14269:
------------------------------------
Description:
*The context :*
I have :
* only one type of message per topic
* the same number of consumers and topics
* each consumer subscribes to all topics in the same microservice
* a strategy where I stopped the consumer if the consumption failed
*The need :*
I would like to have a Topic Round Robin Assignor in order to assign all
partitions of same topic to exactly one consumer, therefore I will be able to
continue the consumption of one topic even if one failed.
If there are exactly the same number of consumers and topics, then each
consumer will get all partitions of one topic.
If there are more consumers than topics, then some consumer will not have any
partitions to consume.
If there are less consumers than topics, then some consumer will have multiple
topics to consume.
As far as I know, there are currently 4 different strategies :
CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
Therefore, I have written my own Topic Round Robin Assignor that assigns all
partitions from each topic to exactly one consumer.
For example, suppose there are two consumers *C0* and {*}C1{*}, two topics *t0*
and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
{*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
The assignment will be:
C0: [t0p0, t0p1 t0p1]
C1: [t1p0, t1p1, t1p2]
First of all, I would like to know if this is a legitimate need.
If this is the case, if you are interested to have a Pull Request about it.
Thank you in advance.
was:
*The context :*
I have only one type of message per topic, multiple consumers in the same
microservice and I am using a strategy where I stopped the consumer if the
consumption failed.
*The need :*
I would like to have a Topic Round Robin Assignor in order to continue to
consume other topics even if one failed.
If there are exactly the same number of consumers and topics, then each
consumer will get all partitions of one topic.
If there are more consumers than topics, then each consumer will get a small
number of partitions of one topic.
If there are less consumers than topics, then each consumer will get all
partitions of one topic, some topics will not be consumed at all.
As far as I know, there are currently 4 different strategies :
CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
Therefore, I have written my own Topic Round Robin Assignor that assigns all
partitions from each topic to one specific consumer.
For example, suppose there are two consumers *C0* and {*}C1{*}, two topics *t0*
and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
{*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
The assignment will be:
C0: [t0p0, t0p1 t0p1]
C1: [t1p0, t1p1, t1p2]
First of all, I would like to know if this is a legitimate need.
If this is the case, if you are interested to have a Pull Request about it.
Thank you in advance.
> Partition Assignment Strategy - Topic Round Robin Assignor
> ----------------------------------------------------------
>
> Key: KAFKA-14269
> URL: https://issues.apache.org/jira/browse/KAFKA-14269
> Project: Kafka
> Issue Type: Wish
> Reporter: Mathieu Amblard
> Priority: Major
>
> *The context :*
> I have :
> * only one type of message per topic
> * the same number of consumers and topics
> * each consumer subscribes to all topics in the same microservice
> * a strategy where I stopped the consumer if the consumption failed
>
> *The need :*
> I would like to have a Topic Round Robin Assignor in order to assign all
> partitions of same topic to exactly one consumer, therefore I will be able to
> continue the consumption of one topic even if one failed.
> If there are exactly the same number of consumers and topics, then each
> consumer will get all partitions of one topic.
> If there are more consumers than topics, then some consumer will not have any
> partitions to consume.
> If there are less consumers than topics, then some consumer will have
> multiple topics to consume.
>
> As far as I know, there are currently 4 different strategies :
> CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
> Therefore, I have written my own Topic Round Robin Assignor that assigns all
> partitions from each topic to exactly one consumer.
>
> For example, suppose there are two consumers *C0* and {*}C1{*}, two topics
> *t0* and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
> {*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
> The assignment will be:
> C0: [t0p0, t0p1 t0p1]
> C1: [t1p0, t1p1, t1p2]
>
> First of all, I would like to know if this is a legitimate need.
> If this is the case, if you are interested to have a Pull Request about it.
> Thank you in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
