On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote: > Hi Colin, > > Thanks for considering the idea and sharing your feedback. > > The improvements I proposed can be achieved, to some extend, using the > AdminClient API and the Consumer Group CLI tool. But they won't fully > support the proposal. > > For example, > Regular expressions are not supported on the groups > Topic / Client filtering is not supported across all groups > > So the reason for proposing the idea was to see if other Kafka users are > also interested in some of these features so we can remove the burden of > them writing custom code around existing consumer group features, and make > those features built into Kafka Consumer Group Command and AdminClient > API.
Hmm. If you're writing Java code that calls the APIs, though, this is easy, right? You can filter the groups in the cluster with a regular expression with just a single line of code in Java 8. > adminClient.listConsumerGroups().all().stream(). > filter(listing -> listing.groupId().matches(myRegex)). > collect(Collectors.toList()); An option to filter ListConsumerGroups by a regular expression might actually be less easy-to-use for most users than this simple filter, since users would have to read the JavaDoc for our APIs. Maybe there are some use-cases where it makes sense to add regex support to the command-line tools, though. I guess the reason why I am pushing back on this is that regular expressions add a lot of complexity to the API and make it harder for us to meet our backwards compatibility guarantees. For example, Java regular expressions changed slightly between Java 7 and Java 8 best, Colin > > Thanks again! > --Vahid > > > > From: Colin McCabe <[email protected]> > To: [email protected] > Date: 08/03/2018 04:16 PM > Subject: Re: ConsumerGroupCommand tool improvement? > > > > Hi Vahid, > > Interesting idea. > > It seems like if you're using the AdminClient APIs programmatically, you > can just do the filtering yourself in a more flexible way than what we > could provide. > > On the other hand, if you're using the ./bin/consumer-groups.sh > command-line tool, why not use grep or a similar tool to filter the > output? Maybe there is some additional functionality in supporting > regexes in the command-line tool, but it also seems like it might be kind > of complex as well. Do you have some examples where having regex support > int the tool would be much easier than the traditional way of piping the > output to grep, awk, and sed? > > best, > Colin > > > On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote: > > Hi all, > > > > A requirement has been raised by a colleague and I wanted to see if > there > > is any interest in the community in adding the functionality to Apache > > Kafka. > > > > ConsumerGroupCommand tool in describe ('--describe' or '--describe > > --offsets') mode currently lists all topics the group has consumed from > > and all consumers with assigned partitions for a single group. > > The idea is to allow filtering of topics, consumers (client ids), and > even > > groups using regular expressions. This will allow the tool to handle use > > > cases such as > > What's the status of a particular consumer (or consumers) in all the > > groups they are consuming from? (for example to check if they are > lagging > > behind in all groups) > > What consumer groups are consuming from a topic (or topics) and what's > the > > lag for each group? > > Limit the existing result to the topics/consumers of interest (for > groups > > with several topics/consumers) > > ... > > > > This would potentially lead to enhancing the AdminClient API as well. > > > > If the community also sees a value in this, I could start drafting a > KIP. > > > > Thanks for your feedback. > > --Vahid > > > > > > >
