I'm hesitant to change the layout of the 1.x release. People do some crazy things when it comes to operations that we can't predict. I'm okay with doing this on the master/2.0 branch, but I'm hesitant on the 1.x branch.
-Taylor > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > > Once we just released Storm 1.1.0, I guess we can continue discussion here. > > I checked the PR list, and there're no open PRs for among Flux, Storm SQL, > and Storm submit tool. So it's good to go. > > I think we have consensus to move non-connectors modules to out of > external. There're also some interest about renaming "external" to > "connectors", but given that "external" is chosen by community and has been > the norm for years, so I agree it would be better not to rename it. We can > do it later when there's another chance of doing it. > > We didn't decide where to move, but most of us seems to be OK to move to > the top directory. > Taylor, any further opinion regarding this? > Suppose we're moving them to the top directory (or whatever the same base > directory), what would be good names for them? Flux doesn't have prefix > 'storm' so a bit different, but if we're OK we skip renaming it. > > I'll do the work when Taylor is OK for changing this. > > - Jungtaek Lim (HeartSaVioR) > > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim <kabh...@gmail.com>님이 작성: > >> 1. apply versions >> >> First plan was applying this only for master, but realized that >> contributors should make two patches for every PRs when we apply this to >> only master. >> >> So in order to make less inconvenience, it would be better to apply this >> for both 1.x and master. It also affects opened pull requests so we would >> like to check that relevant PRs are open, and apply it later than reviewing >> them. >> >> I agree with Harsha. No need to make change for current release vote. >> >> 2. naming issue for "external" >> >> "external" makes me feel that it's related to "external" component, say, >> outside of Storm. That's why I suggest moving non-connectors to out of >> "external". IMHO "connector" is still more intuitive and self-describing, >> but I understand that renaming the directory structure would be painful, >> and "storm-kafka-monitor" is an example of what it's not a "connector" but >> an "external". So I'm OK to keep it as "external". >> >> 3. where to move non-connectors >> >> Except Flux they're directly supported by Storm. I mean "storm.py" is >> aware of them and supports them, so for me they are eligible to move to the >> top directory. I'm open to other suggestion as well. >> >> - Jungtaek Lim (HeartSaVioR) >> >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani <st...@harsha.io>님이 작성: >> >> We should this in next release of 1.x or 2.0. I am +1 on continue with >> current release. >> -Harsha >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <ptgo...@gmail.com> wrote: >>> >>> The question remains if we want to do this in the 1.1.0 release, or >> later. >>> >>> If it's the 1.1.0 release we need to make the changes and cut another RC. >>> I'm fine with that, but want to make sure we have consensus before going >>> down that road. >>> >>> -Taylor >>> >>>> On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <st...@harsha.io> >>> wrote: >>>> >>>> Agree on change like this would be confusing to the users. Lets keep >> the >>>> original plan of moving non-connectors modules of external instead of >>>> introducing new changes >>>> that are not in scope of this discussion. >>>> My +1 still stands on keeping the current external/storm-* in place and >>>> move just sql and storm-perf into top-level. We can have discussion for >>>> storm 2.0 if we want to do >>>> more changes. >>>> >>>> -Harsha >>>> >>>>> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com> >>> wrote: >>>>> >>>>> If we decide to change the structure of the distribution like this, I >>>>> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need >>> to >>>>> cut a new release candidate. >>>>> >>>>> Changing the structure of the distribution file structure can be >>>>> disruptive for users. Even the change to no longer include connector >>>>> binaries, as we've learned, will be a headache for some users. >>>>> >>>>> IMHO, from an ops perspective, changes like this should be handled >> like >>>>> API changes. >>>>> >>>>> -Taylor >>>>> >>>>>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro < >>> hlo...@hortonworks.com> >>>>> wrote: >>>>>> >>>>>> Another possibility is to keep the ‘external' module, and create sub >>>>> modules under it. The legacy structure would remain intact, while >> making >>>>> things more modular. An idea would be: >>>>>> >>>>>> + external >>>>>> + connectors >>>>>> + tools >>>>>> + monitoring >>>>>> + etc >>>>>> >>>>>> Hugo >>>>>> >>>>>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com> >>>>> wrote: >>>>>>> >>>>>>> For the background on why “external” was selected, you have to go >> back >>>>> to a lengthy discussion in Feb. 2014. >>>>>>> >>>>>>> Here’s the start of the thread: >>>>>>> >>>>>>> >>>>> >>> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e >>>>> < >>>>> >>> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E >>>>>> >>>>>>> >>>>>>> It continues into March: >>>>>>> >>>>>>> >>>>> >>> >> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e >>>>>>> >>>>>>> I’m -1 on renaming “external”. That’s the name chosen by the >> community >>>>> and it has been the norm for 3 years. Changing it would likely confuse >>>>> users. >>>>>>> >>>>>>> One of the ideas behind “external” was that it would contain >>> components >>>>> that were not essential to running storm. That line has recently >> blurred >>>>> with some non-connector code sneaking in, so I’m okay with moving >>>>> non-connector code out of external. Another point in that thread was a >>>>> desire to avoid cluttering up the root directory, so we need to be >>> careful >>>>> about what the destination for those components is. >>>>>>> >>>>>>> -Taylor >>>>>>> >>>>>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro < >>>>> hlo...@hortonworks.com> wrote: >>>>>>>> >>>>>>>> +1 non-connectors to top level >>>>>>>> +1 to renaming external to connectors >>>>>>>> >>>>>>>> As for storm-kaka, if we are already touching the external modules, >>>>> all the modules should be a submodule of a parent module called >>>>> storm-kafka. I don’t think we should have 3 parent modules as we >>> currently >>>>> have (storm-kafka, storm-kafka-client, storm-kafka-monitor) >>>>>>>> >>>>>>>> The structure should be something along the lines (I don’t mean the >>>>> exact names; we should find better ones. storm-kafka and >>>>> storm-kafka-client are not very self explanatory in my opinion) >>>>>>>> >>>>>>>> + storm-kafka >>>>>>>> + monitoring >>>>>>>> + new-client >>>>>>>> + old-client >>>>>>>> >>>>>>>> If we have to create new modules or submodules (e.g. under utils) >> so >>>>> be it. The code should be in a module that is named after what its >>> doing. >>>>>>>> >>>>>>>> Hugo >>>>>>>> >>>>>>>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com >>> >>>>> wrote: >>>>>>>>> >>>>>>>>> +1 to moving non-conncectors to top level. I think we should keep >>>>> stom-kafka-monitor under external or connectors(after renaming). >>>>>>>>> >>>>>>>>> Jungtaek, just to clarify on what you said regarding storm core >>>>> referencing storm-kafka-monitor. Like you said its just calling the >>> script >>>>> from ui jvm. There is no dependency in terms of class files needed to >>> run >>>>> the script from ui. The script itself adds a –cp argument and all it >>> needs >>>>> is storm-kafka-monitor jar in classpath. As far as packaging the >> script >>> is >>>>> concerned we can do what Satish suggested. i.e. move it to >>>>> storm-kafka-monitor in source and while packaging put it under bin. >>>>> Reiterating to make sure I am not mis-understanding anything. >>>>>>>>> >>>>>>>>> On 3/24/17, 9:14 AM, "Harsha Chintalapani" <st...@harsha.io> >> wrote: >>>>>>>>> >>>>>>>>> +1 on moving non-connectors to top-level like sql and storm-perf. >>>>>>>>> Regarding storm-kafka-monitor we can move this into "util" folder >> or >>>>> keep >>>>>>>>> in the external. >>>>>>>>> -Harsha >>>>>>>>> >>>>>>>>> On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana < >>>>> satish.dugg...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> storm-kafka-monitor is not a connector by itself but it is >> related >>>>> to kafka >>>>>>>>>> connectors. So, any utility related to that connector should be >>> part >>>>> of >>>>>>>>>> that connector module(can be a submodule) instead of a top level >>>>> module. >>>>>>>>>> core/ui uses this utility referring directly in a hacky way, >> which >>>>> we may >>>>>>>>>> want to fix later. storm-kafka-monitor script exists in bin >>>>> directory which >>>>>>>>>> can be moved to storm-kafka-monitor module and the same script >> can >>> be >>>>>>>>>> packaged as part of storm/bin directory while packaging the >>>>> distribution. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> ~Satish. >>>>>>>>>> >>>>>>>>>>> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim < >> kabh...@gmail.com> >>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> storm-kafka-monitor is referred by storm-core, though it's >>>>> referenced via >>>>>>>>>>> executing command. Yes it's a bit odd to place it as top >>> directory, >>>>> but >>>>>>>>>>> it's not a connector for that reason too. Neither is ideal for >> me, >>>>> so >>>>>>>>>>> ironically, either is fine. >>>>>>>>>>> >>>>>>>>>>> - Jungtaek Lim (HeartSaVioR) >>>>>>>>>>> >>>>>>>>>>> 2017년 3월 24일 (금) 오후 4:19, Satish Duggana < >>> satish.dugg...@gmail.com >>>>>> 님이 >>>>>>>>>> 작성: >>>>>>>>>>> >>>>>>>>>>>> +1 except for storm-kafka-monitor module as this utility is >> more >>>>> about >>>>>>>>>>>> querying topic/partition offsets of kafka spouts in a topology. >>> Do >>>>> not >>>>>>>>>> we >>>>>>>>>>>> want to push this module into connectors/kafka as a submodule >>> along >>>>>>>>>> with >>>>>>>>>>>> other submodules including old/new kafka spout modules? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Satish. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer < >>> ai...@hortonworks.com >>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> Makes sense to move the non-connectors to top level and keep >>> only >>>>> the >>>>>>>>>>>>> connectors under “connectors” folder. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 3/24/17, 12:00 PM, "Jungtaek Lim" <kabh...@gmail.com> >>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> (Sent this yesterday but can't find this from storm-dev >> mbox... >>>>>>>>>>> sending >>>>>>>>>>>> it >>>>>>>>>>>>>> again) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi dev, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd like to start discussion regarding moving non-connectors >>>>> modules >>>>>>>>>>> out >>>>>>>>>>>>> of >>>>>>>>>>>>>> external, maybe top directory. >>>>>>>>>>>>>> >>>>>>>>>>>>>> "external" directory has non-connectors (SQL, Flux, >>>>>>>>>>> storm-kafka-monitor, >>>>>>>>>>>>>> storm-submit-tools), and except Flux, others should be placed >>> to >>>>> the >>>>>>>>>>>>> binary >>>>>>>>>>>>>> dist. since Storm itself (not from user topology) needs to >>> refer >>>>>>>>>> them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> They're actually tied to the core of Storm, so I feel that it >>>>> would >>>>>>>>>> be >>>>>>>>>>>>>> better to treat them (including Flux) as non-external, maybe >>> same >>>>>>>>>>> level >>>>>>>>>>>> as >>>>>>>>>>>>>> storm-core. >>>>>>>>>>>>>> (I'm not sure what "external" actually means for Storm >> project >>>>> btw.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> In addition, after doing that I'd like to change the >> directory >>>>> name >>>>>>>>>>>>>> "external" to "connector" or so, so that the name could be >>>>>>>>>>>> self-describing >>>>>>>>>>>>>> and we can only place connectors to that directory. >>>>>>>>>>>>>> (I know it would be painful for already opened pull requests, >>> so >>>>> no >>>>>>>>>>>> strong >>>>>>>>>>>>>> opinion regarding this.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looking forward to your opinion! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> >>
