[
https://issues.apache.org/jira/browse/KAFKA-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612181#comment-17612181
]
Mathieu Amblard commented on KAFKA-14269:
-----------------------------------------
Hello [~guozhang] ,
Thanks for your comment that's a good question,
All topics have not the same number of partition, they are sized accordingly to
the data traffic.
If we have to consume a large amount of data, we simply scale up the number of
pods (so the number of consumer). Using this partition assignor, the partitions
are balanced uniformly across the pods.
For information, we do not use Kafka Streams and we have the need to handle
only 10K (max 100K) of records a day per topic. I do not know if a such
assignor would work correctly with millions of records.
> Partition Assignment Strategy - Topic Round Robin Assignor
> ----------------------------------------------------------
>
> Key: KAFKA-14269
> URL: https://issues.apache.org/jira/browse/KAFKA-14269
> Project: Kafka
> Issue Type: Wish
> Components: clients
> Reporter: Mathieu Amblard
> Priority: Major
>
> *The context :*
> I have :
> * only one type of message per topic
> * the same number of consumers and topics
> * each consumer subscribes to all topics in the same microservice
> * a strategy where I stopped the consumer if the consumption failed
>
> *The need :*
> I would like to have a Topic Round Robin Assignor in order to assign all
> partitions of same topic to exactly one consumer, therefore I will be able to
> continue the consumption of one topic even if one failed.
> If there are exactly the same number of consumers and topics, then each
> consumer will get all partitions of one topic.
> If there are more consumers than topics, then some consumer will not have any
> partitions to consume.
> If there are less consumers than topics, then some consumer will have
> multiple topics to consume.
>
> As far as I know, there are currently 4 different strategies :
> CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
> Therefore, I have written my own Topic Round Robin Assignor that assigns all
> partitions from each topic to exactly one consumer.
>
> For example, suppose there are two consumers *C0* and {*}C1{*}, two topics
> *t0* and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
> {*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
> The assignment will be:
> C0: [t0p0, t0p1 t0p1]
> C1: [t1p0, t1p1, t1p2]
>
> First of all, I would like to know if this is a legitimate need.
> If this is the case, if you are interested to have a Pull Request about it.
> Thank you in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
