Hey Vahid, Hmm... If possible, it would be nice to avoid cluttering the default option too much, especially if it is information which is going to be the same for all members (such as the generation). My preference would be to use the --state option that you've suggested for that info so that we can represent it more concisely.
The reason I prefer the current output is that it is clear every entry corresponds to a partition for which we have committed offset. Entries like this look strange: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID - - - - - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff /127.0.0.1 consumer4 - - - - - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea /127.0.0.1 consumer5 It makes me think that the consumers have committed offsets for an unknown partition. The --members option seems like a clearer way to communicate the fact that there are some members with no assigned partitions. A few additional suggestions: 1. Maybe we can rename --partitions to --offsets or --committed-offsets and the output could match the default output (in other words, --offsets is treated as the default switch). Seems no harm including the assignment information if we have it. 2. Along the lines of Onur's comment, it would be nice if the --members option included the list of assignment strategies that the consumer joined with (round-robin, range, etc). This list should always be small. 3. Thinking a little more, I'm not sure how necessary a --topics option is. The --partitions (or --offsets) option already shows the current assignment. Maybe --topics could be --subscription and just list the topics that the members subscribed to? Thanks, Jason On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Jason, Onur, thank you for reviewing the KIP. > > Regarding the default `--describe` option, so far there have been a few > suggestions that conflict a bit. Here are the suggestions: > - Keep the current behavior exactly as is (Edo, Jeff) > - Remove members with no assignments from the current result set (Jason) > - Add additional status info to the result set (Onur) -- I assume the > additional status (which are group related info, rather than group member > related) will appear in the result separate from the member table (e.g., > before the table) > > One thing we could do to remain as close as possible to these suggestions > is trim the resulting rows as per Jason's suggestion, and add the > additional details that Onur suggested. Would this work for everyone? Edo, > Jeff, what do you think? > If so, I'll update the KIP accordingly. > > Some of the other updates based on the feedback received: > * "--describe --members" will not include a topic(partitions) column. > Instead there will be a #Partitions (number of partitions assigned to this > member) column > * "--describe --topics" will be added to list topic partitions in the > group and the relevant info > * "--describe --state" will be added to report group related info, such as > state, protocol, ... > > Thanks. > --Vahid > > > > From: Onur Karaman <onurkaraman.apa...@gmail.com> > To: d...@kafka.apache.org > Cc: Kafka Users <users@kafka.apache.org> > Date: 07/14/2017 11:40 AM > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > In other words, I think the default should be the exact behavior we have > today plus the remaining group information from DescribeGroupResponse. > > On Fri, Jul 14, 2017 at 11:36 AM, Onur Karaman > <onurkaraman.apa...@gmail.com > > wrote: > > > I think if we had the opportunity to start from scratch, --describe > would > > have been the following: > > --describe --offsets: shows all offsets committed for the group as well > as > > lag > > --describe --state (or maybe --members): shows the full > > DescribeGroupResponse output (including things like generation id, > state, > > protocol type, etc) > > --describe: shows the merged version of the above two. > > > > On Fri, Jul 14, 2017 at 10:56 AM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > >> Hey Vahid, > >> > >> Thanks for the KIP. Looks like a nice improvement. One minor > suggestion: > >> Since consumers can be subscribed to a large number of topics, I'm > >> wondering if it might be better to leave out the topic list from the > >> "describe members" option so that the output remains concise? Perhaps > we > >> could list only the number of assigned partitions so that users have an > >> easy way to check the overall balance and we can add a separate > "describe > >> topics" switch to see the topic breakdown? > >> > >> As for the default --describe, it seems safest to keep its current > >> behavior. In other words, we should list all partitions which have > >> committed offsets for the group even if the partition is not currently > >> assigned. However, I don't think we need to try and fit members without > >> any > >> assigned partitions into that view. > >> > >> Thanks, > >> Jason > >> > >> On Fri, Jul 7, 2017 at 10:49 AM, Vahid S Hashemian < > >> vahidhashem...@us.ibm.com> wrote: > >> > >> > Thanks Jeff for your feedback on the usefulness of the current tool. > >> > > >> > --Vahid > >> > > >> > > >> > > >> > > >> > From: Jeff Widman <j...@netskope.com> > >> > To: d...@kafka.apache.org > >> > Cc: Kafka User <users@kafka.apache.org> > >> > Date: 07/06/2017 02:25 PM > >> > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views > for > >> > ConsumerGroupCommand > >> > > >> > > >> > > >> > Thanks for the KIP Vahid. I think it'd be useful to have these > filters. > >> > > >> > That said, I also agree with Edo. > >> > > >> > We don't currently rely on the output, but there's been more than one > >> time > >> > when debugging an issue that I notice something amiss when I see all > the > >> > data at once but if it wasn't present in the default view I probably > >> would > >> > have missed it as I wouldn't have thought to look at that particular > >> > filter. > >> > > >> > This would also be more consistent with the API of the > kafka-topics.sh > >> > where "--describe" gives everything and then can be filtered down. > >> > > >> > > >> > > >> > On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> > >> wrote: > >> > > >> > > Hi Vahid, > >> > > no we are not relying on parsing the current output. > >> > > > >> > > I just thought that keeping the full output isn't necessarily that > bad > >> > as > >> > > it shows some sort of history of how a group was used. > >> > > > >> > > ciao > >> > > Edo > >> > > -------------------------------------------------- > >> > > > >> > > Edoardo Comar > >> > > > >> > > IBM Message Hub > >> > > > >> > > IBM UK Ltd, Hursley Park, SO21 2JN > >> > > > >> > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 > >> > > 17:11:43: > >> > > > >> > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > >> > > > To: d...@kafka.apache.org > >> > > > Cc: "Kafka User" <users@kafka.apache.org> > >> > > > Date: 04/07/2017 17:12 > >> > > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > >> > > > ConsumerGroupCommand > >> > > > > >> > > > Hi Edo, > >> > > > > >> > > > Thanks for reviewing the KIP. > >> > > > > >> > > > Modifying the default behavior of `--describe` was suggested in > the > >> > > > related JIRA. > >> > > > We could poll the community to see whether they go for that > option, > >> > or, > >> > > as > >> > > > you suggested, introducing a new `--only-xxx` ( can't also think > of > >> a > >> > > > proper name right now :) ) option instead. > >> > > > > >> > > > Are you making use of the current `--describe` output and relying > on > >> > the > >> > > > >> > > > full data set? > >> > > > > >> > > > Thanks. > >> > > > --Vahid > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > From: Edoardo Comar <eco...@uk.ibm.com> > >> > > > To: d...@kafka.apache.org > >> > > > Cc: "Kafka User" <users@kafka.apache.org> > >> > > > Date: 07/04/2017 03:17 AM > >> > > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' > views > >> > for > >> > > > >> > > > ConsumerGroupCommand > >> > > > > >> > > > > >> > > > > >> > > > Thanks Vahid, I like the KIP. > >> > > > > >> > > > One question - could we keep the current "--describe" behavior > >> > unchanged > >> > > > >> > > > and introduce "--only-xxx" options to filter down the full output > as > >> > you > >> > > > >> > > > proposed ? > >> > > > > >> > > > ciao, > >> > > > Edo > >> > > > -------------------------------------------------- > >> > > > > >> > > > Edoardo Comar > >> > > > > >> > > > IBM Message Hub > >> > > > > >> > > > IBM UK Ltd, Hursley Park, SO21 2JN > >> > > > > >> > > > > >> > > > > >> > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > >> > > > To: dev <d...@kafka.apache.org>, "Kafka User" > >> > > <users@kafka.apache.org> > >> > > > Date: 04/07/2017 00:06 > >> > > > Subject: [DISCUSS] KIP-175: Additional '--describe' views > for > >> > > > ConsumerGroupCommand > >> > > > > >> > > > > >> > > > > >> > > > Hi, > >> > > > > >> > > > I created KIP-175 to make some improvements to the > >> > ConsumerGroupCommand > >> > > > tool. > >> > > > The KIP can be found here: > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A > >> > > > +Additional+%27--describe%27+views+for+ConsumerGroupCommand > >> > > > > >> > > > > >> > > > > >> > > > Your review and feedback is welcome! > >> > > > > >> > > > Thanks. > >> > > > --Vahid > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > Unless stated otherwise above: > >> > > > IBM United Kingdom Limited - Registered in England and Wales with > >> > number > >> > > > >> > > > 741598. > >> > > > Registered office: PO Box 41, North Harbour, Portsmouth, > Hampshire > >> PO6 > >> > > 3AU > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > Unless stated otherwise above: > >> > > IBM United Kingdom Limited - Registered in England and Wales with > >> number > >> > > 741598. > >> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 > >> > 3AU > >> > > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > >
