We should this in next release of 1.x or 2.0. I am +1 on continue with current release. -Harsha On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:
> The question remains if we want to do this in the 1.1.0 release, or later. > > If it's the 1.1.0 release we need to make the changes and cut another RC. > I'm fine with that, but want to make sure we have consensus before going > down that road. > > -Taylor > > > On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <st...@harsha.io> > wrote: > > > > Agree on change like this would be confusing to the users. Lets keep the > > original plan of moving non-connectors modules of external instead of > > introducing new changes > > that are not in scope of this discussion. > > My +1 still stands on keeping the current external/storm-* in place and > > move just sql and storm-perf into top-level. We can have discussion for > > storm 2.0 if we want to do > > more changes. > > > > -Harsha > > > >> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com> > wrote: > >> > >> If we decide to change the structure of the distribution like this, I > >> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need > to > >> cut a new release candidate. > >> > >> Changing the structure of the distribution file structure can be > >> disruptive for users. Even the change to no longer include connector > >> binaries, as we've learned, will be a headache for some users. > >> > >> IMHO, from an ops perspective, changes like this should be handled like > >> API changes. > >> > >> -Taylor > >> > >>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro < > hlo...@hortonworks.com> > >> wrote: > >>> > >>> Another possibility is to keep the ‘external' module, and create sub > >> modules under it. The legacy structure would remain intact, while making > >> things more modular. An idea would be: > >>> > >>> + external > >>> + connectors > >>> + tools > >>> + monitoring > >>> + etc > >>> > >>> Hugo > >>> > >>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com> > >> wrote: > >>>> > >>>> For the background on why “external” was selected, you have to go back > >> to a lengthy discussion in Feb. 2014. > >>>> > >>>> Here’s the start of the thread: > >>>> > >>>> > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e > >> < > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E > >>> > >>>> > >>>> It continues into March: > >>>> > >>>> > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e > >>>> > >>>> I’m -1 on renaming “external”. That’s the name chosen by the community > >> and it has been the norm for 3 years. Changing it would likely confuse > >> users. > >>>> > >>>> One of the ideas behind “external” was that it would contain > components > >> that were not essential to running storm. That line has recently blurred > >> with some non-connector code sneaking in, so I’m okay with moving > >> non-connector code out of external. Another point in that thread was a > >> desire to avoid cluttering up the root directory, so we need to be > careful > >> about what the destination for those components is. > >>>> > >>>> -Taylor > >>>> > >>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro < > >> hlo...@hortonworks.com> wrote: > >>>>> > >>>>> +1 non-connectors to top level > >>>>> +1 to renaming external to connectors > >>>>> > >>>>> As for storm-kaka, if we are already touching the external modules, > >> all the modules should be a submodule of a parent module called > >> storm-kafka. I don’t think we should have 3 parent modules as we > currently > >> have (storm-kafka, storm-kafka-client, storm-kafka-monitor) > >>>>> > >>>>> The structure should be something along the lines (I don’t mean the > >> exact names; we should find better ones. storm-kafka and > >> storm-kafka-client are not very self explanatory in my opinion) > >>>>> > >>>>> + storm-kafka > >>>>> + monitoring > >>>>> + new-client > >>>>> + old-client > >>>>> > >>>>> If we have to create new modules or submodules (e.g. under utils) so > >> be it. The code should be in a module that is named after what its > doing. > >>>>> > >>>>> Hugo > >>>>> > >>>>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com> > >> wrote: > >>>>>> > >>>>>> +1 to moving non-conncectors to top level. I think we should keep > >> stom-kafka-monitor under external or connectors(after renaming). > >>>>>> > >>>>>> Jungtaek, just to clarify on what you said regarding storm core > >> referencing storm-kafka-monitor. Like you said its just calling the > script > >> from ui jvm. There is no dependency in terms of class files needed to > run > >> the script from ui. The script itself adds a –cp argument and all it > needs > >> is storm-kafka-monitor jar in classpath. As far as packaging the script > is > >> concerned we can do what Satish suggested. i.e. move it to > >> storm-kafka-monitor in source and while packaging put it under bin. > >> Reiterating to make sure I am not mis-understanding anything. > >>>>>> > >>>>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <st...@harsha.io> wrote: > >>>>>> > >>>>>> +1 on moving non-connectors to top-level like sql and storm-perf. > >>>>>> Regarding storm-kafka-monitor we can move this into "util" folder or > >> keep > >>>>>> in the external. > >>>>>> -Harsha > >>>>>> > >>>>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana < > >> satish.dugg...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> storm-kafka-monitor is not a connector by itself but it is related > >> to kafka > >>>>>>> connectors. So, any utility related to that connector should be > part > >> of > >>>>>>> that connector module(can be a submodule) instead of a top level > >> module. > >>>>>>> core/ui uses this utility referring directly in a hacky way, which > >> we may > >>>>>>> want to fix later. storm-kafka-monitor script exists in bin > >> directory which > >>>>>>> can be moved to storm-kafka-monitor module and the same script can > be > >>>>>>> packaged as part of storm/bin directory while packaging the > >> distribution. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> ~Satish. > >>>>>>> > >>>>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim <kabh...@gmail.com> > >> wrote: > >>>>>>>> > >>>>>>>> storm-kafka-monitor is referred by storm-core, though it's > >> referenced via > >>>>>>>> executing command. Yes it's a bit odd to place it as top > directory, > >> but > >>>>>>>> it's not a connector for that reason too. Neither is ideal for me, > >> so > >>>>>>>> ironically, either is fine. > >>>>>>>> > >>>>>>>> - Jungtaek Lim (HeartSaVioR) > >>>>>>>> > >>>>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana < > satish.dugg...@gmail.com > >>> 님이 > >>>>>>> 작성: > >>>>>>>> > >>>>>>>>> +1 except for storm-kafka-monitor module as this utility is more > >> about > >>>>>>>>> querying topic/partition offsets of kafka spouts in a topology. > Do > >> not > >>>>>>> we > >>>>>>>>> want to push this module into connectors/kafka as a submodule > along > >>>>>>> with > >>>>>>>>> other submodules including old/new kafka spout modules? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Satish. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer < > ai...@hortonworks.com > >>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 > >>>>>>>>>> > >>>>>>>>>> Makes sense to move the non-connectors to top level and keep > only > >> the > >>>>>>>>>> connectors under “connectors” folder. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <kabh...@gmail.com> > wrote: > >>>>>>>>>>> > >>>>>>>>>>> (Sent this yesterday but can't find this from storm-dev mbox... > >>>>>>>> sending > >>>>>>>>> it > >>>>>>>>>>> again) > >>>>>>>>>>> > >>>>>>>>>>> Hi dev, > >>>>>>>>>>> > >>>>>>>>>>> I'd like to start discussion regarding moving non-connectors > >> modules > >>>>>>>> out > >>>>>>>>>> of > >>>>>>>>>>> external, maybe top directory. > >>>>>>>>>>> > >>>>>>>>>>> "external" directory has non-connectors (SQL, Flux, > >>>>>>>> storm-kafka-monitor, > >>>>>>>>>>> storm-submit-tools), and except Flux, others should be placed > to > >> the > >>>>>>>>>> binary > >>>>>>>>>>> dist. since Storm itself (not from user topology) needs to > refer > >>>>>>> them. > >>>>>>>>>>> > >>>>>>>>>>> They're actually tied to the core of Storm, so I feel that it > >> would > >>>>>>> be > >>>>>>>>>>> better to treat them (including Flux) as non-external, maybe > same > >>>>>>>> level > >>>>>>>>> as > >>>>>>>>>>> storm-core. > >>>>>>>>>>> (I'm not sure what "external" actually means for Storm project > >> btw.) > >>>>>>>>>>> > >>>>>>>>>>> In addition, after doing that I'd like to change the directory > >> name > >>>>>>>>>>> "external" to "connector" or so, so that the name could be > >>>>>>>>> self-describing > >>>>>>>>>>> and we can only place connectors to that directory. > >>>>>>>>>>> (I know it would be painful for already opened pull requests, > so > >> no > >>>>>>>>> strong > >>>>>>>>>>> opinion regarding this.) > >>>>>>>>>>> > >>>>>>>>>>> Looking forward to your opinion! > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Jungtaek Lim (HeartSaVioR) > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >
