If we decide to change the structure of the distribution like this, I think we should do it in masrwe/2.0. If we want this for 1.1.0 we need to cut a new release candidate.
Changing the structure of the distribution file structure can be disruptive for users. Even the change to no longer include connector binaries, as we've learned, will be a headache for some users. IMHO, from an ops perspective, changes like this should be handled like API changes. -Taylor > On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> > wrote: > > Another possibility is to keep the ‘external' module, and create sub modules > under it. The legacy structure would remain intact, while making things more > modular. An idea would be: > > + external > + connectors > + tools > + monitoring > + etc > > Hugo > >> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote: >> >> For the background on why “external” was selected, you have to go back to a >> lengthy discussion in Feb. 2014. >> >> Here’s the start of the thread: >> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e >> >> <http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E> >> >> It continues into March: >> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e >> >> I’m -1 on renaming “external”. That’s the name chosen by the community and >> it has been the norm for 3 years. Changing it would likely confuse users. >> >> One of the ideas behind “external” was that it would contain components that >> were not essential to running storm. That line has recently blurred with >> some non-connector code sneaking in, so I’m okay with moving non-connector >> code out of external. Another point in that thread was a desire to avoid >> cluttering up the root directory, so we need to be careful about what the >> destination for those components is. >> >> -Taylor >> >>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> >>> wrote: >>> >>> +1 non-connectors to top level >>> +1 to renaming external to connectors >>> >>> As for storm-kaka, if we are already touching the external modules, all the >>> modules should be a submodule of a parent module called storm-kafka. I >>> don’t think we should have 3 parent modules as we currently have >>> (storm-kafka, storm-kafka-client, storm-kafka-monitor) >>> >>> The structure should be something along the lines (I don’t mean the exact >>> names; we should find better ones. storm-kafka and storm-kafka-client are >>> not very self explanatory in my opinion) >>> >>> + storm-kafka >>> + monitoring >>> + new-client >>> + old-client >>> >>> If we have to create new modules or submodules (e.g. under utils) so be it. >>> The code should be in a module that is named after what its doing. >>> >>> Hugo >>> >>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com> wrote: >>>> >>>> +1 to moving non-conncectors to top level. I think we should keep >>>> stom-kafka-monitor under external or connectors(after renaming). >>>> >>>> Jungtaek, just to clarify on what you said regarding storm core >>>> referencing storm-kafka-monitor. Like you said its just calling the script >>>> from ui jvm. There is no dependency in terms of class files needed to run >>>> the script from ui. The script itself adds a –cp argument and all it needs >>>> is storm-kafka-monitor jar in classpath. As far as packaging the script is >>>> concerned we can do what Satish suggested. i.e. move it to >>>> storm-kafka-monitor in source and while packaging put it under bin. >>>> Reiterating to make sure I am not mis-understanding anything. >>>> >>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <st...@harsha.io> wrote: >>>> >>>> +1 on moving non-connectors to top-level like sql and storm-perf. >>>> Regarding storm-kafka-monitor we can move this into "util" folder or keep >>>> in the external. >>>> -Harsha >>>> >>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana <satish.dugg...@gmail.com> >>>> wrote: >>>> >>>>> storm-kafka-monitor is not a connector by itself but it is related to >>>>> kafka >>>>> connectors. So, any utility related to that connector should be part of >>>>> that connector module(can be a submodule) instead of a top level module. >>>>> core/ui uses this utility referring directly in a hacky way, which we may >>>>> want to fix later. storm-kafka-monitor script exists in bin directory >>>>> which >>>>> can be moved to storm-kafka-monitor module and the same script can be >>>>> packaged as part of storm/bin directory while packaging the distribution. >>>>> >>>>> Thanks, >>>>> ~Satish. >>>>> >>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <kabh...@gmail.com> wrote: >>>>>> >>>>>> storm-kafka-monitor is referred by storm-core, though it's referenced via >>>>>> executing command. Yes it's a bit odd to place it as top directory, but >>>>>> it's not a connector for that reason too. Neither is ideal for me, so >>>>>> ironically, either is fine. >>>>>> >>>>>> - Jungtaek Lim (HeartSaVioR) >>>>>> >>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <satish.dugg...@gmail.com>님이 >>>>> 작성: >>>>>> >>>>>>> +1 except for storm-kafka-monitor module as this utility is more about >>>>>>> querying topic/partition offsets of kafka spouts in a topology. Do not >>>>> we >>>>>>> want to push this module into connectors/kafka as a submodule along >>>>> with >>>>>>> other submodules including old/new kafka spout modules? >>>>>>> >>>>>>> Thanks, >>>>>>> Satish. >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <ai...@hortonworks.com> >>>>>> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> Makes sense to move the non-connectors to top level and keep only the >>>>>>>> connectors under “connectors” folder. >>>>>>>> >>>>>>>> >>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> (Sent this yesterday but can't find this from storm-dev mbox... >>>>>> sending >>>>>>> it >>>>>>>>> again) >>>>>>>>> >>>>>>>>> Hi dev, >>>>>>>>> >>>>>>>>> I'd like to start discussion regarding moving non-connectors modules >>>>>> out >>>>>>>> of >>>>>>>>> external, maybe top directory. >>>>>>>>> >>>>>>>>> "external" directory has non-connectors (SQL, Flux, >>>>>> storm-kafka-monitor, >>>>>>>>> storm-submit-tools), and except Flux, others should be placed to the >>>>>>>> binary >>>>>>>>> dist. since Storm itself (not from user topology) needs to refer >>>>> them. >>>>>>>>> >>>>>>>>> They're actually tied to the core of Storm, so I feel that it would >>>>> be >>>>>>>>> better to treat them (including Flux) as non-external, maybe same >>>>>> level >>>>>>> as >>>>>>>>> storm-core. >>>>>>>>> (I'm not sure what "external" actually means for Storm project btw.) >>>>>>>>> >>>>>>>>> In addition, after doing that I'd like to change the directory name >>>>>>>>> "external" to "connector" or so, so that the name could be >>>>>>> self-describing >>>>>>>>> and we can only place connectors to that directory. >>>>>>>>> (I know it would be painful for already opened pull requests, so no >>>>>>> strong >>>>>>>>> opinion regarding this.) >>>>>>>>> >>>>>>>>> Looking forward to your opinion! >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >
