Hi Viktor, I'll admit I've only had an extremely brief look at your code and jira, but it sounds absolutely awesome! Plus it will give us a chance at a fresh start without worrying about breaking existing code using the tools. Of course...at the risk of code duplication..
I'd be absolutely in favor of creating something like you have proposed! Best regards, Sönke On Wed, May 8, 2019 at 11:38 AM Viktor Somogyi-Vass <viktorsomo...@gmail.com> wrote: > Just to add: I wasn't too pushy by raising a KIP for this as so far I had > the experience that the community is fine with them or at least people > learned to live with the current tools but if there's a need I'd be happy > to jump back working on it or helping you if you have time :) > > On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Hi Sönke, > > > > In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately > > didn't get much attention so far from the creators of the jira. It works > > similarly to what Colin mentioned, like "kafka.sh topics create -n > my-topic > > -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic > > --partitions 3 --replicas 3". The bootstrap server everywhere defaults to > > :9092 or reads up the configs from a path so in many scenarios you can > omit > > it, also it uses the admin client of course and all in all provides a > much > > better experience than the current tools. It has interactive mode and > help > > too. Wanted to implement "code completion" too but for that I'd have to > > exercise the code a little bit more :). > > It is currently based on a quite old trunk but if you think it's > > interesting I can rebase it and we can continue with raising a KIP. > > https://github.com/viktorsomogyi/kafka/tree/kafka_shell > > > > Viktor > > > > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau > > <soenke.lie...@opencore.com.invalid> wrote: > > > >> Hi Colin, > >> > >> thanks! > >> While I personally don't like the current command line tools I > >> realistically think we'll be stuck with them for a while yet, so a > cleanup > >> might make sense. > >> So I'll start looking into that. > >> > >> Regarding a central entrypoint, that would indeed be brilliant and I'd > >> love > >> to work on that, but I currently have enough other open things to look > at, > >> so I won't draw that one to myself as well for now. > >> > >> But I'll keep it in mind for when some time frees up :) > >> > >> Best regards, > >> Sönke > >> > >> > >> > >> Colin McCabe <cmcc...@apache.org> schrieb am Di., 7. Mai 2019, 00:56: > >> > >> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote: > >> > > Hi Colin, > >> > > > >> > > it was my intention to keep the structure of the commands mostly > >> intact > >> > > while doing the refactoring - if that is possible, have not really > >> > checked > >> > > yet to be honest. > >> > > > >> > > But what I wanted to try and do is recreate the current parsing with > >> > > argparse as much as possible. And in the process simply adding > >> synonyms, > >> > > for example make the kafka-console-producer understand a > >> > > bootstrap-parameter in addition to broker-list. > >> > > There is a bit of custom logic about which parameters go together > >> etc. in > >> > > the current classes, so output may look different here and there, > but > >> in > >> > > principle I do believe that it should be possible to recreate the > >> current > >> > > structure. > >> > > >> > Sounds like a good idea. Thanks for the clarification. > >> > > >> > > > >> > > If there is an appetite for a new, hadoop-like entrypoint anyway, > then > >> > all > >> > > of this might be "wasted" effort, or rather effort better spent > >> though, > >> > you > >> > > are right. > >> > > >> > I don't think anyone is working on a new entry point right now -- or > if > >> > they are, they haven't said anything yet :) > >> > > >> > I just wanted to mention it as a possible approach in case you wanted > to > >> > do a bigger project. > >> > > >> > best, > >> > Colin > >> > > >> > > > >> > > Best regards, > >> > > Sönke > >> > > > >> > > > >> > > > >> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cmcc...@apache.org> > >> wrote: > >> > > > >> > > > Hi Sönke, > >> > > > > >> > > > #2 is a bit tough because people have come to rely on the way the > >> > commands > >> > > > are structured right now. > >> > > > > >> > > > If we want to make big changes, it might be easier just to create > a > >> > > > separate tool and deprecate the old one(s). One thing we've > talked > >> > about > >> > > > doing in the past is creating a single entry point for all the > tool > >> > > > functionality, kind of like hadoop did with the "hadoop" command > Or > >> > git > >> > > > with the "git" command, etc. Then we could deprecate the > standalone > >> > > > commands and remove them after enough time had passed-- kind of > like > >> > the > >> > > > old consumer. > >> > > > > >> > > > On the other hand, a more incremental change would be > standardizing > >> > flags > >> > > > a bit. So for example, at least setting it up so that there is a > >> > standard > >> > > > way of supplying bootstrap brokers, etc. We could keep the old > >> flags > >> > > > around for a while as variants to ease the transition. > >> > > > > >> > > > best, > >> > > > Colin > >> > > > > >> > > > > >> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > >> > > > > Hi Colin, > >> > > > > > >> > > > > I totally agree! Especially the differently named bootstrap > server > >> > > > options > >> > > > > have been annoying me a long time. > >> > > > > > >> > > > > I'd propose a two-step approach: > >> > > > > 1. Add new default options objects similar to CommandLineUtils > and > >> > > > > CommandDefaultOptions (based on argparse4j) but in the clients > >> > project, > >> > > > as > >> > > > > this is referenced by all command line tools as far as I can > tell > >> > > > > 2. Refactor tools one by one to use these new helper classes > (and > >> > thus > >> > > > > argparse) and add standardized synonyms for parameters as > >> necessary > >> > > > > > >> > > > > I think for step 1 we can get away with no KIP, as this doesn't > >> > change > >> > > > any > >> > > > > public interfaces or behavior. > >> > > > > Step 2 probably needs a KIP as we are adding new parameters? We > >> can > >> > pick > >> > > > up > >> > > > > KIP-14 again for that I think. A lot of work has been done on > that > >> > > > already. > >> > > > > > >> > > > > Does that sound useful to everybody? > >> > > > > > >> > > > > Best regards, > >> > > > > Sönke > >> > > > > > >> > > > > > >> > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe < > cmcc...@apache.org> > >> > wrote: > >> > > > > > >> > > > > > If we are going to standardize on one argument parsing > library, > >> it > >> > > > should > >> > > > > > certainly be argparse4j, I think. > >> > > > > > argparse4j is simply a better argument parsing library with > >> > support > >> > > > for > >> > > > > > more features. One example is mutually exclusive options. > >> > argparse4j > >> > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't > support > >> > this, > >> > > > so > >> > > > > > when it is needed, we have to add extra code to manually check > >> that > >> > > > > > mutually exclusive options are not set. > >> > > > > > > >> > > > > > argparse4j also has subcommands. If you want something like > >> "git > >> > add" > >> > > > > > with some set of flags, and "git remove" with another, you can > >> do > >> > this > >> > > > with > >> > > > > > argparse4j, but not with jopt. This would be very helpful for > >> > > > clearing up > >> > > > > > confusion in a lot of our shell scripts which have accumulated > >> > dozens > >> > > > of > >> > > > > > arguments, most of which are only relevant to a very specific > >> > > > operation. > >> > > > > > But you just can't do it with jopt. > >> > > > > > > >> > > > > > Just to give an example, argparse4j with subcommands would > allow > >> > you to > >> > > > > > run something like ./kafka-topics.sh list --help and get just > >> > options > >> > > > that > >> > > > > > were relevant for listing topics, not the full dozens of > options > >> > that > >> > > > might > >> > > > > > relate to adding topics, removing them, etc. > >> > > > > > > >> > > > > > To be honest, though, what would help users the most is > >> > standardizing > >> > > > the > >> > > > > > option flags across tools. We should have a standard way of > >> > specifying > >> > > > > > bootstrap brokers, for example. (We can continue to support > the > >> > old > >> > > > > > synonyms for a while, of course.) > >> > > > > > > >> > > > > > best, > >> > > > > > Colin > >> > > > > > > >> > > > > > > >> > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote: > >> > > > > > > I took another look at the PR itself and I think it would be > >> > great to > >> > > > > > have > >> > > > > > > this cleanup too -- I cannot remember at the beginning why > we > >> > > > gradually > >> > > > > > > moved to different mechanism (argparse4j) for different > cmds, > >> if > >> > > > there's > >> > > > > > no > >> > > > > > > rationales behind it we should just make them consistent. > >> > > > > > > > >> > > > > > > Thanks for driving this! > >> > > > > > > > >> > > > > > > Guozhang > >> > > > > > > > >> > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan < > >> > ryannedo...@gmail.com> > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep > >> track > >> > of > >> > > > which > >> > > > > > > > commands use which form -- I always seem to guess wrong. > >> > > > > > > > > >> > > > > > > > Though I don't think there is any reason to deprecate > >> existing > >> > > > forms, > >> > > > > > e.g. > >> > > > > > > > consumer.config vs consumer-config. I think it's perfectly > >> > > > reasonable > >> > > > > > to > >> > > > > > > > have multiple spellings of the same arguments. I don't > >> really > >> > see a > >> > > > > > > > downside to keeping the aliases around indefinitely. > >> > > > > > > > > >> > > > > > > > Ryanne > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > >> > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > >> > > > > > > > > >> > > > > > > > > Hi everybody, > >> > > > > > > > > > >> > > > > > > > > Jason and I were recently discussing command line > argument > >> > > > parsing on > >> > > > > > > > > KAFKA-8131 (or rather the related pull request) [1]. > >> > > > > > > > > > >> > > > > > > > > Command line tools and their arguments are somewhat > >> diverse > >> > at > >> > > > the > >> > > > > > > > moment. > >> > > > > > > > > Most of the tools use joptsimple for argument parsing, > >> some > >> > newer > >> > > > > > java > >> > > > > > > > > tools use argparse4j instead and some tools use nothing > at > >> > all. > >> > > > > > > > > I've looked for a reason as to why there are two > libraries > >> > being > >> > > > > > used, > >> > > > > > > > but > >> > > > > > > > > couldn't really find anything. Paolo brought up the same > >> > > > question on > >> > > > > > the > >> > > > > > > > > mailing list a while back [7], but got no response > either. > >> > > > > > > > > Does anybody know why this is the case? > >> > > > > > > > > > >> > > > > > > > > This results in no central place to add universal > >> parameters > >> > like > >> > > > > > help > >> > > > > > > > and > >> > > > > > > > > version, as well as the help output looking different > >> between > >> > > > some > >> > > > > > of the > >> > > > > > > > > tools. > >> > > > > > > > > Also, there are a number of parameters that should be > >> > renamed to > >> > > > > > adhere > >> > > > > > > > to > >> > > > > > > > > defaults. > >> > > > > > > > > > >> > > > > > > > > There have been a few discussions and initiatives around > >> > this in > >> > > > the > >> > > > > > > > past. > >> > > > > > > > > Just of the top of my head (and a 5 minute jira search) > >> there > >> > > > are: > >> > > > > > > > > - KIP-14 [2] > >> > > > > > > > > - KAFKA-2111 [3] > >> > > > > > > > > - KIP-316 [4] > >> > > > > > > > > - KAFKA-1292 [5] > >> > > > > > > > > - KAFKA-3530 [6] > >> > > > > > > > > - and probably many more > >> > > > > > > > > > >> > > > > > > > > Would people generally be in favor of revisiting this > >> topic? > >> > > > > > > > > > >> > > > > > > > > What I'd propose to do is: > >> > > > > > > > > - comb through jira and KIPs, clean up old stuff and > >> creae a > >> > new > >> > > > > > umbrella > >> > > > > > > > > issue to track this (maybe reuse KIP-4 as well) > >> > > > > > > > > - agree on one library for parsing command line > arguments > >> > (don't > >> > > > care > >> > > > > > > > which > >> > > > > > > > > one, but two is one too many I think) > >> > > > > > > > > - refactor tools to use one library and default way of > >> > argument > >> > > > > > parsing > >> > > > > > > > > with central help and version parameter > >> > > > > > > > > - add aliases for options that should be renamed > >> according to > >> > > > KIP-4 > >> > > > > > (and > >> > > > > > > > > maybe others) so that both new and old work for a while, > >> > > > deprecate > >> > > > > > old > >> > > > > > > > > parameters for a cycle or two and then remove them > >> > > > > > > > > > >> > > > > > > > > I'll shut up now and see if people would consider this > >> > useful or > >> > > > > > have any > >> > > > > > > > > other input :) > >> > > > > > > > > > >> > > > > > > > > Best regards, > >> > > > > > > > > Sönke > >> > > > > > > > > > >> > > > > > > > > [1] > >> > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003 > >> > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131> > >> > > > > > > > > [2] > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization > >> > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111 > >> > > > > > > > > [4] > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties > >> > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292 > >> > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530 > >> > > > > > > > > [7] > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> > https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > -- Guozhang > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Sönke Liebau > >> > > > > Partner > >> > > > > Tel. +49 179 7940878 > >> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > >> Germany > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Sönke Liebau > >> > > Partner > >> > > Tel. +49 179 7940878 > >> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - > Germany > >> > > > >> > > >> > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
