Also, want to clarify one more doubt, is there any way for the client to explicitly trigger a rebalance without dying itself?
On Thu, Jan 30, 2020 at 7:54 PM Devaki, Srinivas <me@eightnoteight.space> wrote: > Hi All, > > We have a set of logstash consumer groups running under the same set of > instances, we have decided to run separate consumer groups subscribing > multiple topics instead of running single consumer group for all topics(the > reasoning behind this decision is because of how our elasticsearch cluster > is designed). > > Since we are running multiple consumer groups, sometimes we have detected > that a few ec2 nodes are receiving multiple high throughput topics in > different consumer groups. which was expected based on the implementation > of round robin assignor. > > So I've decided to make a partition assignor which will consider the > assignment based on other consumer group assignment. > > Could you please give me some pointers on how to proceed. This is my > initial ideas on the problem. > > Solution #0: > write an assignor, and use a specific consumer id pattern across all > consumer groups, and in the assignor do a describe on all consumer groups > and based on the topic throughput and the other consumer group assignment > decide the assignment of this topic > > > > > Thanks >
