For the background on why “external” was selected, you have to go back to a lengthy discussion in Feb. 2014.
Here’s the start of the thread: http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3e <http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3E> It continues into March: http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e I’m -1 on renaming “external”. That’s the name chosen by the community and it has been the norm for 3 years. Changing it would likely confuse users. One of the ideas behind “external” was that it would contain components that were not essential to running storm. That line has recently blurred with some non-connector code sneaking in, so I’m okay with moving non-connector code out of external. Another point in that thread was a desire to avoid cluttering up the root directory, so we need to be careful about what the destination for those components is. -Taylor > On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <[email protected]> > wrote: > > +1 non-connectors to top level > +1 to renaming external to connectors > > As for storm-kaka, if we are already touching the external modules, all the > modules should be a submodule of a parent module called storm-kafka. I don’t > think we should have 3 parent modules as we currently have (storm-kafka, > storm-kafka-client, storm-kafka-monitor) > > The structure should be something along the lines (I don’t mean the exact > names; we should find better ones. storm-kafka and storm-kafka-client are > not very self explanatory in my opinion) > > + storm-kafka > + monitoring > + new-client > + old-client > > If we have to create new modules or submodules (e.g. under utils) so be it. > The code should be in a module that is named after what its doing. > > Hugo > >> On Mar 24, 2017, at 11:15 AM, Priyank Shah <[email protected]> wrote: >> >> +1 to moving non-conncectors to top level. I think we should keep >> stom-kafka-monitor under external or connectors(after renaming). >> >> Jungtaek, just to clarify on what you said regarding storm core referencing >> storm-kafka-monitor. Like you said its just calling the script from ui jvm. >> There is no dependency in terms of class files needed to run the script from >> ui. The script itself adds a –cp argument and all it needs is >> storm-kafka-monitor jar in classpath. As far as packaging the script is >> concerned we can do what Satish suggested. i.e. move it to >> storm-kafka-monitor in source and while packaging put it under bin. >> Reiterating to make sure I am not mis-understanding anything. >> >> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <[email protected]> wrote: >> >> +1 on moving non-connectors to top-level like sql and storm-perf. >> Regarding storm-kafka-monitor we can move this into "util" folder or keep >> in the external. >> -Harsha >> >> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana <[email protected]> >> wrote: >> >>> storm-kafka-monitor is not a connector by itself but it is related to kafka >>> connectors. So, any utility related to that connector should be part of >>> that connector module(can be a submodule) instead of a top level module. >>> core/ui uses this utility referring directly in a hacky way, which we may >>> want to fix later. storm-kafka-monitor script exists in bin directory which >>> can be moved to storm-kafka-monitor module and the same script can be >>> packaged as part of storm/bin directory while packaging the distribution. >>> >>> Thanks, >>> ~Satish. >>> >>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <[email protected]> wrote: >>> >>>> storm-kafka-monitor is referred by storm-core, though it's referenced via >>>> executing command. Yes it's a bit odd to place it as top directory, but >>>> it's not a connector for that reason too. Neither is ideal for me, so >>>> ironically, either is fine. >>>> >>>> - Jungtaek Lim (HeartSaVioR) >>>> >>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <[email protected]>님이 >>> 작성: >>>> >>>>> +1 except for storm-kafka-monitor module as this utility is more about >>>>> querying topic/partition offsets of kafka spouts in a topology. Do not >>> we >>>>> want to push this module into connectors/kafka as a submodule along >>> with >>>>> other submodules including old/new kafka spout modules? >>>>> >>>>> Thanks, >>>>> Satish. >>>>> >>>>> >>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <[email protected]> >>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> Makes sense to move the non-connectors to top level and keep only the >>>>>> connectors under “connectors” folder. >>>>>> >>>>>> >>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <[email protected]> wrote: >>>>>> >>>>>>> (Sent this yesterday but can't find this from storm-dev mbox... >>>> sending >>>>> it >>>>>>> again) >>>>>>> >>>>>>> Hi dev, >>>>>>> >>>>>>> I'd like to start discussion regarding moving non-connectors modules >>>> out >>>>>> of >>>>>>> external, maybe top directory. >>>>>>> >>>>>>> "external" directory has non-connectors (SQL, Flux, >>>> storm-kafka-monitor, >>>>>>> storm-submit-tools), and except Flux, others should be placed to the >>>>>> binary >>>>>>> dist. since Storm itself (not from user topology) needs to refer >>> them. >>>>>>> >>>>>>> They're actually tied to the core of Storm, so I feel that it would >>> be >>>>>>> better to treat them (including Flux) as non-external, maybe same >>>> level >>>>> as >>>>>>> storm-core. >>>>>>> (I'm not sure what "external" actually means for Storm project btw.) >>>>>>> >>>>>>> In addition, after doing that I'd like to change the directory name >>>>>>> "external" to "connector" or so, so that the name could be >>>>> self-describing >>>>>>> and we can only place connectors to that directory. >>>>>>> (I know it would be painful for already opened pull requests, so no >>>>> strong >>>>>>> opinion regarding this.) >>>>>>> >>>>>>> Looking forward to your opinion! >>>>>>> >>>>>>> Thanks, >>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>> >>>>>> >>>>> >>>> >>> >> >> >
