The question remains if we want to do this in the 1.1.0 release, or later. If it's the 1.1.0 release we need to make the changes and cut another RC. I'm fine with that, but want to make sure we have consensus before going down that road.
-Taylor > On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <st...@harsha.io> wrote: > > Agree on change like this would be confusing to the users. Lets keep the > original plan of moving non-connectors modules of external instead of > introducing new changes > that are not in scope of this discussion. > My +1 still stands on keeping the current external/storm-* in place and > move just sql and storm-perf into top-level. We can have discussion for > storm 2.0 if we want to do > more changes. > > -Harsha > >> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com> wrote: >> >> If we decide to change the structure of the distribution like this, I >> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need to >> cut a new release candidate. >> >> Changing the structure of the distribution file structure can be >> disruptive for users. Even the change to no longer include connector >> binaries, as we've learned, will be a headache for some users. >> >> IMHO, from an ops perspective, changes like this should be handled like >> API changes. >> >> -Taylor >> >>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> >> wrote: >>> >>> Another possibility is to keep the ‘external' module, and create sub >> modules under it. The legacy structure would remain intact, while making >> things more modular. An idea would be: >>> >>> + external >>> + connectors >>> + tools >>> + monitoring >>> + etc >>> >>> Hugo >>> >>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com> >> wrote: >>>> >>>> For the background on why “external” was selected, you have to go back >> to a lengthy discussion in Feb. 2014. >>>> >>>> Here’s the start of the thread: >>>> >>>> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e >> < >> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E >>> >>>> >>>> It continues into March: >>>> >>>> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e >>>> >>>> I’m -1 on renaming “external”. That’s the name chosen by the community >> and it has been the norm for 3 years. Changing it would likely confuse >> users. >>>> >>>> One of the ideas behind “external” was that it would contain components >> that were not essential to running storm. That line has recently blurred >> with some non-connector code sneaking in, so I’m okay with moving >> non-connector code out of external. Another point in that thread was a >> desire to avoid cluttering up the root directory, so we need to be careful >> about what the destination for those components is. >>>> >>>> -Taylor >>>> >>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro < >> hlo...@hortonworks.com> wrote: >>>>> >>>>> +1 non-connectors to top level >>>>> +1 to renaming external to connectors >>>>> >>>>> As for storm-kaka, if we are already touching the external modules, >> all the modules should be a submodule of a parent module called >> storm-kafka. I don’t think we should have 3 parent modules as we currently >> have (storm-kafka, storm-kafka-client, storm-kafka-monitor) >>>>> >>>>> The structure should be something along the lines (I don’t mean the >> exact names; we should find better ones. storm-kafka and >> storm-kafka-client are not very self explanatory in my opinion) >>>>> >>>>> + storm-kafka >>>>> + monitoring >>>>> + new-client >>>>> + old-client >>>>> >>>>> If we have to create new modules or submodules (e.g. under utils) so >> be it. The code should be in a module that is named after what its doing. >>>>> >>>>> Hugo >>>>> >>>>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com> >> wrote: >>>>>> >>>>>> +1 to moving non-conncectors to top level. I think we should keep >> stom-kafka-monitor under external or connectors(after renaming). >>>>>> >>>>>> Jungtaek, just to clarify on what you said regarding storm core >> referencing storm-kafka-monitor. Like you said its just calling the script >> from ui jvm. There is no dependency in terms of class files needed to run >> the script from ui. The script itself adds a –cp argument and all it needs >> is storm-kafka-monitor jar in classpath. As far as packaging the script is >> concerned we can do what Satish suggested. i.e. move it to >> storm-kafka-monitor in source and while packaging put it under bin. >> Reiterating to make sure I am not mis-understanding anything. >>>>>> >>>>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <st...@harsha.io> wrote: >>>>>> >>>>>> +1 on moving non-connectors to top-level like sql and storm-perf. >>>>>> Regarding storm-kafka-monitor we can move this into "util" folder or >> keep >>>>>> in the external. >>>>>> -Harsha >>>>>> >>>>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana < >> satish.dugg...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> storm-kafka-monitor is not a connector by itself but it is related >> to kafka >>>>>>> connectors. So, any utility related to that connector should be part >> of >>>>>>> that connector module(can be a submodule) instead of a top level >> module. >>>>>>> core/ui uses this utility referring directly in a hacky way, which >> we may >>>>>>> want to fix later. storm-kafka-monitor script exists in bin >> directory which >>>>>>> can be moved to storm-kafka-monitor module and the same script can be >>>>>>> packaged as part of storm/bin directory while packaging the >> distribution. >>>>>>> >>>>>>> Thanks, >>>>>>> ~Satish. >>>>>>> >>>>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <kabh...@gmail.com> >> wrote: >>>>>>>> >>>>>>>> storm-kafka-monitor is referred by storm-core, though it's >> referenced via >>>>>>>> executing command. Yes it's a bit odd to place it as top directory, >> but >>>>>>>> it's not a connector for that reason too. Neither is ideal for me, >> so >>>>>>>> ironically, either is fine. >>>>>>>> >>>>>>>> - Jungtaek Lim (HeartSaVioR) >>>>>>>> >>>>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana <satish.dugg...@gmail.com >>> 님이 >>>>>>> 작성: >>>>>>>> >>>>>>>>> +1 except for storm-kafka-monitor module as this utility is more >> about >>>>>>>>> querying topic/partition offsets of kafka spouts in a topology. Do >> not >>>>>>> we >>>>>>>>> want to push this module into connectors/kafka as a submodule along >>>>>>> with >>>>>>>>> other submodules including old/new kafka spout modules? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Satish. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer <ai...@hortonworks.com >>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> Makes sense to move the non-connectors to top level and keep only >> the >>>>>>>>>> connectors under “connectors” folder. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> (Sent this yesterday but can't find this from storm-dev mbox... >>>>>>>> sending >>>>>>>>> it >>>>>>>>>>> again) >>>>>>>>>>> >>>>>>>>>>> Hi dev, >>>>>>>>>>> >>>>>>>>>>> I'd like to start discussion regarding moving non-connectors >> modules >>>>>>>> out >>>>>>>>>> of >>>>>>>>>>> external, maybe top directory. >>>>>>>>>>> >>>>>>>>>>> "external" directory has non-connectors (SQL, Flux, >>>>>>>> storm-kafka-monitor, >>>>>>>>>>> storm-submit-tools), and except Flux, others should be placed to >> the >>>>>>>>>> binary >>>>>>>>>>> dist. since Storm itself (not from user topology) needs to refer >>>>>>> them. >>>>>>>>>>> >>>>>>>>>>> They're actually tied to the core of Storm, so I feel that it >> would >>>>>>> be >>>>>>>>>>> better to treat them (including Flux) as non-external, maybe same >>>>>>>> level >>>>>>>>> as >>>>>>>>>>> storm-core. >>>>>>>>>>> (I'm not sure what "external" actually means for Storm project >> btw.) >>>>>>>>>>> >>>>>>>>>>> In addition, after doing that I'd like to change the directory >> name >>>>>>>>>>> "external" to "connector" or so, so that the name could be >>>>>>>>> self-describing >>>>>>>>>>> and we can only place connectors to that directory. >>>>>>>>>>> (I know it would be painful for already opened pull requests, so >> no >>>>>>>>> strong >>>>>>>>>>> opinion regarding this.) >>>>>>>>>>> >>>>>>>>>>> Looking forward to your opinion! >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>
