Agree on change like this would be confusing to the users. Lets keep the original plan of moving non-connectors modules of external instead of introducing new changes that are not in scope of this discussion. My +1 still stands on keeping the current external/storm-* in place and move just sql and storm-perf into top-level. We can have discussion for storm 2.0 if we want to do more changes.
-Harsha On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <[email protected]> wrote: > If we decide to change the structure of the distribution like this, I > think we should do it in masrwe/2.0. If we want this for 1.1.0 we need to > cut a new release candidate. > > Changing the structure of the distribution file structure can be > disruptive for users. Even the change to no longer include connector > binaries, as we've learned, will be a headache for some users. > > IMHO, from an ops perspective, changes like this should be handled like > API changes. > > -Taylor > > > On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <[email protected]> > wrote: > > > > Another possibility is to keep the ‘external' module, and create sub > modules under it. The legacy structure would remain intact, while making > things more modular. An idea would be: > > > > + external > > + connectors > > + tools > > + monitoring > > + etc > > > > Hugo > > > >> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <[email protected]> > wrote: > >> > >> For the background on why “external” was selected, you have to go back > to a lengthy discussion in Feb. 2014. > >> > >> Here’s the start of the thread: > >> > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3e > < > http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%[email protected]%3E > > > >> > >> It continues into March: > >> > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e > >> > >> I’m -1 on renaming “external”. That’s the name chosen by the community > and it has been the norm for 3 years. Changing it would likely confuse > users. > >> > >> One of the ideas behind “external” was that it would contain components > that were not essential to running storm. That line has recently blurred > with some non-connector code sneaking in, so I’m okay with moving > non-connector code out of external. Another point in that thread was a > desire to avoid cluttering up the root directory, so we need to be careful > about what the destination for those components is. > >> > >> -Taylor > >> > >>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro < > [email protected]> wrote: > >>> > >>> +1 non-connectors to top level > >>> +1 to renaming external to connectors > >>> > >>> As for storm-kaka, if we are already touching the external modules, > all the modules should be a submodule of a parent module called > storm-kafka. I don’t think we should have 3 parent modules as we currently > have (storm-kafka, storm-kafka-client, storm-kafka-monitor) > >>> > >>> The structure should be something along the lines (I don’t mean the > exact names; we should find better ones. storm-kafka and > storm-kafka-client are not very self explanatory in my opinion) > >>> > >>> + storm-kafka > >>> + monitoring > >>> + new-client > >>> + old-client > >>> > >>> If we have to create new modules or submodules (e.g. under utils) so > be it. The code should be in a module that is named after what its doing. > >>> > >>> Hugo > >>> > >>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <[email protected]> > wrote: > >>>> > >>>> +1 to moving non-conncectors to top level. I think we should keep > stom-kafka-monitor under external or connectors(after renaming). > >>>> > >>>> Jungtaek, just to clarify on what you said regarding storm core > referencing storm-kafka-monitor. Like you said its just calling the script > from ui jvm. There is no dependency in terms of class files needed to run > the script from ui. The script itself adds a –cp argument and all it needs > is storm-kafka-monitor jar in classpath. As far as packaging the script is > concerned we can do what Satish suggested. i.e. move it to > storm-kafka-monitor in source and while packaging put it under bin. > Reiterating to make sure I am not mis-understanding anything. > >>>> > >>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <[email protected]> wrote: > >>>> > >>>> +1 on moving non-connectors to top-level like sql and storm-perf. > >>>> Regarding storm-kafka-monitor we can move this into "util" folder or > keep > >>>> in the external. > >>>> -Harsha > >>>> > >>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana < > [email protected]> > >>>> wrote: > >>>> > >>>>> storm-kafka-monitor is not a connector by itself but it is related > to kafka > >>>>> connectors. So, any utility related to that connector should be part > of > >>>>> that connector module(can be a submodule) instead of a top level > module. > >>>>> core/ui uses this utility referring directly in a hacky way, which > we may > >>>>> want to fix later. storm-kafka-monitor script exists in bin > directory which > >>>>> can be moved to storm-kafka-monitor module and the same script can be > >>>>> packaged as part of storm/bin directory while packaging the > distribution. > >>>>> > >>>>> Thanks, > >>>>> ~Satish. > >>>>> > >>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <[email protected]> > wrote: > >>>>>> > >>>>>> storm-kafka-monitor is referred by storm-core, though it's > referenced via > >>>>>> executing command. Yes it's a bit odd to place it as top directory, > but > >>>>>> it's not a connector for that reason too. Neither is ideal for me, > so > >>>>>> ironically, either is fine. > >>>>>> > >>>>>> - Jungtaek Lim (HeartSaVioR) > >>>>>> > >>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <[email protected] > >님이 > >>>>> 작성: > >>>>>> > >>>>>>> +1 except for storm-kafka-monitor module as this utility is more > about > >>>>>>> querying topic/partition offsets of kafka spouts in a topology. Do > not > >>>>> we > >>>>>>> want to push this module into connectors/kafka as a submodule along > >>>>> with > >>>>>>> other submodules including old/new kafka spout modules? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Satish. > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <[email protected] > > > >>>>>> wrote: > >>>>>>> > >>>>>>>> +1 > >>>>>>>> > >>>>>>>> Makes sense to move the non-connectors to top level and keep only > the > >>>>>>>> connectors under “connectors” folder. > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <[email protected]> wrote: > >>>>>>>>> > >>>>>>>>> (Sent this yesterday but can't find this from storm-dev mbox... > >>>>>> sending > >>>>>>> it > >>>>>>>>> again) > >>>>>>>>> > >>>>>>>>> Hi dev, > >>>>>>>>> > >>>>>>>>> I'd like to start discussion regarding moving non-connectors > modules > >>>>>> out > >>>>>>>> of > >>>>>>>>> external, maybe top directory. > >>>>>>>>> > >>>>>>>>> "external" directory has non-connectors (SQL, Flux, > >>>>>> storm-kafka-monitor, > >>>>>>>>> storm-submit-tools), and except Flux, others should be placed to > the > >>>>>>>> binary > >>>>>>>>> dist. since Storm itself (not from user topology) needs to refer > >>>>> them. > >>>>>>>>> > >>>>>>>>> They're actually tied to the core of Storm, so I feel that it > would > >>>>> be > >>>>>>>>> better to treat them (including Flux) as non-external, maybe same > >>>>>> level > >>>>>>> as > >>>>>>>>> storm-core. > >>>>>>>>> (I'm not sure what "external" actually means for Storm project > btw.) > >>>>>>>>> > >>>>>>>>> In addition, after doing that I'd like to change the directory > name > >>>>>>>>> "external" to "connector" or so, so that the name could be > >>>>>>> self-describing > >>>>>>>>> and we can only place connectors to that directory. > >>>>>>>>> (I know it would be painful for already opened pull requests, so > no > >>>>>>> strong > >>>>>>>>> opinion regarding this.) > >>>>>>>>> > >>>>>>>>> Looking forward to your opinion! > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Jungtaek Lim (HeartSaVioR) > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > > >
