+ 1 for release-3 branch instead of release-3.8 I would suggest we vote on two proposals (4.x as well as the option to change names and start at 1.0). Any other suggestions? I'm planning to start a VOTE thread by EOD.
Thanks, Thomas On Sat, Aug 19, 2017 at 1:56 PM, Pramod Immaneni <pra...@datatorrent.com> wrote: > I think if there is a will to allow older apps to continue to benefit from > improvements to operators, without changes, it can be done along with the > proposed move, especially since we are not taking about all operators. > > However, since that is not the case, I am fine with starting a 4.0 release > line as long as we don't seal the future of 3.x now. What I would prefer is > to create a release-3 branch and not a release-3.8 branch now and let the > future determine if 3.8 will be the last 3.x minor release. > > Thanks > > On Fri, Aug 18, 2017 at 5:04 PM Thomas Weise <t...@apache.org> wrote: > > > The proposed change does NOT require changes to existing apps. It follows > > what you see for example in the JDK where older classes are retired > without > > making breaking major release changes. > > > > Even though not required technically, it looks like the preference is to > > make this change with a 4.0 release. I will therefore propose to change > the > > master branch to 4.0-SNAPSHOT and make this and other changes deemed > > worthwhile. > > > > To me it is more important that to newcomers the codebase does not look > > like an outdated legacy situation and therefore if I have to make a > choice > > than pretty much any version number will do for that. So far I have seen > > very little interest and initiative in addressing maintenance work like > the > > packages, checkstyle, CI stability, documentation etc. So it would be > great > > if people interested would step up and contribute. > > > > Here is the next proposal how to proceed: > > > > - create release-3.8 branch for last 3.8 release right away > > - update master to 4.0-SNAPSHOT as part of the open PR #664 > > - identify further cleanup work for master > > - release 4.0 > > > > Thanks, > > Thomas > > > > > > > > > > > > > > > > > > > > On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <a...@datatorrent.com> > wrote: > > > > > This following pull request should be taken up in 4.0.0. See my > comments > > in > > > https://github.com/apache/apex-malhar/pull/664 > > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com% > > > 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg= > > > AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ> > > > https://github.com/apache/apex-malhar/pull/662 > > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com% > > > 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg= > > > AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg> > > > > > > This merge should not be done without a consensus. This will require > code > > > changes to existing apps. > > > > > > Thks, > > > Amol > > > > > > > > > > > > E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* > > > > > > www.datatorrent.com > > > > > > > > > On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <t...@apache.org> wrote: > > > > > > > Hi, > > > > > > > > I opened following PRs for the package change: > > > > > > > > https://github.com/apache/apex-malhar/pull/662 > > > > > > > > Moves all classes with history retained (hence 2 commits). Also > > contains > > > > checkstyle and other mechanical changes. > > > > > > > > https://github.com/apache/apex-malhar/pull/664 > > > > > > > > Adds backward compatibility jar. > > > > > > > > Once above PRs are merged the new artifact can be deployed and > > introduced > > > > as dependency in malhar-library. > > > > > > > > Please review. > > > > > > > > Thanks, > > > > Thomas > > > > > > > > > > > > > > > > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <t...@apache.org> > wrote: > > > > > > > > > My original list of work items contained the b/w compatibility > > aspect, > > > I > > > > > don't think there should be any confusion of whether it will be > > covered > > > > > here or not. > > > > > > > > > > The proposed shading will provide the old classes and they will be > > > frozen > > > > > as of release 3.7. That's the same as making a copy of the code and > > > never > > > > > again making changes to the original classes. This cannot be > > > accomplished > > > > > by using the older 3.7 release in your project because you cannot > > use 2 > > > > > different versions of Malhar in parallel unless you apply shading. > > > > > > > > > > The shaded artifact will only expose the com.datatorrent classes, > and > > > > they > > > > > will be self-contained as the rest of the classes that they may > > depend > > > on > > > > > are shaded. The shaded artifact does not evolve, there are not more > > > > changes > > > > > to com.datatorrent classes after they are relocated in master. > > > > > > > > > > Thanks, > > > > > Thomas > > > > > > > > > > > > > > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni < > > > pra...@datatorrent.com > > > > > > > > > > wrote: > > > > > > > > > >> I don't think we can limit the topic strictly to relocation > without > > > > having > > > > >> a good b/w compatibility story or at least one that goes far > enough. > > > > >> > > > > >> The shading idea sounds interesting. Why not let the shaded > version > > > move > > > > >> forward with each release till we hit a major release. If it is > > going > > > to > > > > >> remain pegged at 3.7.0, why shade in the first place as the > regular > > > > 3.7.0 > > > > >> release would do the same job and it would be same as the loss of > > b/w > > > > >> compatibility with newer releases. > > > > >> > > > > >> Thanks > > > > >> > > > > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <t...@apache.org> > > wrote: > > > > >> > > > > >> > Discussing what in the future might become stable needs to be a > > > > separate > > > > >> > thread, it will be a much bigger discussion. > > > > >> > > > > > >> > The topic here is to relocate the packages. With a few > exceptions > > > > >> > relocation won't affect the semantic versioning. Semantic > > versioning > > > > is > > > > >> > essentially not effective for Malhar because almost everything > is > > > > >> @Evolving > > > > >> > (and there are reasons for that.. -> separate topic) > > > > >> > > > > > >> > I don't really like the idea of creating bw compatibility stubs > > for > > > > the > > > > >> > follow up PR. It creates even more clutter in the source tree > than > > > > there > > > > >> > already is and so here is an alternative suggestion: > > > > >> > > > > > >> > https://github.com/tweise/apex-malhar/blob/malhar37- > > > > >> > compat/shaded-malhar37/pom.xml > > > > >> > > > > > >> > Create a shaded artifact that provides the old com.datatorrent.* > > > > >> classes as > > > > >> > of release 3.7. Users can include that artifact if they don't > want > > > to > > > > >> > change import statements. At the same time they have an > incentive > > to > > > > >> switch > > > > >> > to the relocated classes to take advantage of bug fixes and new > > > > >> > functionality. > > > > >> > > > > > >> > I will work on the first PR that does the relocate. In the > > meantime, > > > > we > > > > >> can > > > > >> > finalize what backward compatibility support we want to provide > > and > > > > how. > > > > >> > > > > > >> > Thanks, > > > > >> > Thomas > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni < > > > > >> pra...@datatorrent.com> > > > > >> > wrote: > > > > >> > > > > > >> > > How about coming up with a list of @Evolving operators that we > > > would > > > > >> like > > > > >> > > to see in the eventual stable list and move those along with > the > > > not > > > > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the > > > rest > > > > as > > > > >> > they > > > > >> > > are. Then have a follow up JIRA for the rest to be moved over > to > > > > >> contrib > > > > >> > > and be deprecated. > > > > >> > > > > > > >> > > Thanks > > > > >> > > > > > > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise < > > > > >> thomas.we...@gmail.com> > > > > >> > > wrote: > > > > >> > > > > > > >> > > > We need to keep the discussion here on topic, if other > things > > > are > > > > >> piled > > > > >> > > on > > > > >> > > > top then nothing gets done. > > > > >> > > > > > > > >> > > > Most operators today are not stable, they are @Evolving. So > > > > perhaps > > > > >> it > > > > >> > is > > > > >> > > > necessary to have a separate discussion about that, outside > of > > > > this > > > > >> > > thread. > > > > >> > > > My guess is that there are only a few operators that we > could > > > > >> declare > > > > >> > > > stable. Specifically, under contrib the closest one might > have > > > > been > > > > >> > > Kafka, > > > > >> > > > but that is already superseded by the newer versions. > > > > >> > > > > > > > >> > > > Thomas > > > > >> > > > > > > > >> > > > > > > > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni < > > > > >> > > pra...@datatorrent.com> > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > We had a discussion a while back, agreed to relegate > > > non-stable > > > > >> and > > > > >> > > > > experimental operators to contrib and also added this to > > > > >> contribution > > > > >> > > > > guidelines. We aexecuted on this and cleaned up the repo > by > > > > moving > > > > >> > > > > operators deemed non-suitable for production use at that > > time > > > to > > > > >> > > contrib. > > > > >> > > > > So I wouldn't say the operators that are at the top level > > > today > > > > or > > > > >> > the > > > > >> > > > ones > > > > >> > > > > in library are 0.x.x quality. Granted, we may need to do > one > > > > more > > > > >> > pass > > > > >> > > as > > > > >> > > > > some of the operators may no longer be considered the > right > > > > >> > > > implementations > > > > >> > > > > with the advent of the windowed operator. > > > > >> > > > > > > > > >> > > > > Since contrib used to be the place that housed operators > > that > > > > >> > required > > > > >> > > > > third party libraries, there are some production quality > > > > >> operators in > > > > >> > > > there > > > > >> > > > > that need to be pulled out into top level modules. > > > > >> > > > > > > > > >> > > > > I think we are in agreement that for operators that we > > > consider > > > > >> > stable, > > > > >> > > > we > > > > >> > > > > should provide a b/w stub. I would add that we strongly > > > consider > > > > >> > > creating > > > > >> > > > > the org.apache.apex counterparts of any stable operators > > that > > > > are > > > > >> in > > > > >> > > > > contrib out in top level modules and have the > > com.datatorrent > > > > >> stubs > > > > >> > in > > > > >> > > > > contrib. > > > > >> > > > > > > > > >> > > > > For the operators not considered stable, I would prefer we > > > > either > > > > >> > leave > > > > >> > > > the > > > > >> > > > > current package path as is or if we change the package > path > > > then > > > > >> > create > > > > >> > > > the > > > > >> > > > > b/w stub as I am not sure how widely they are in use (so, > in > > > > >> essence, > > > > >> > > > > preserve semantic versioning). It would be good if there > is > > a > > > > >> > followup > > > > >> > > > JIRA > > > > >> > > > > that takes a look at what other operators can be moved to > > > > contrib > > > > >> > with > > > > >> > > > the > > > > >> > > > > advent of the newer frameworks and understanding. > > > > >> > > > > > > > > >> > > > > Thanks > > > > >> > > > > > > > > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise < > > t...@apache.org > > > > > > > > >> > wrote: > > > > >> > > > > > > > > >> > > > > > Most of the operators evolve, as is quite clearly > > indicated > > > by > > > > >> > > > @Evolving > > > > >> > > > > > annotations. So while perhaps 0.x.x would be a more > > > > appropriate > > > > >> > > version > > > > >> > > > > > number, I don't think you can change that. > > > > >> > > > > > > > > > >> > > > > > Thomas > > > > >> > > > > > > > > > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov < > > > > >> > v.ro...@datatorrent.com > > > > >> > > > > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > > If entire library is not stable, its version should be > > > 0.x.x > > > > >> and > > > > >> > > > there > > > > >> > > > > > > should not be any semantic versioning enabled or > > implied. > > > It > > > > >> > > evolves. > > > > >> > > > > If > > > > >> > > > > > > some operators are stable as 3.8.x version suggests, > the > > > > >> library > > > > >> > > > should > > > > >> > > > > > > follow semantic versioning and it is not OK to make a > > > major > > > > >> > change > > > > >> > > > that > > > > >> > > > > > > breaks semantic versioning across the entire library. > It > > > is > > > > >> not a > > > > >> > > > > finding > > > > >> > > > > > > an excuse not to make a change. For me semantic > > versioning > > > > and > > > > >> > > > backward > > > > >> > > > > > > compatibility is more important than a proper package > > > name. > > > > >> > > > > > > > > > > >> > > > > > > Thank you, > > > > >> > > > > > > > > > > >> > > > > > > Vlad > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote: > > > > >> > > > > > > > > > > >> > > > > > >> Semantic versioning makes sense when you have a > stable > > > > >> baseline. > > > > >> > > As > > > > >> > > > > long > > > > >> > > > > > >> as > > > > >> > > > > > >> frequent fixes need to be made to address basic > issues, > > > it > > > > >> makes > > > > >> > > no > > > > >> > > > > > sense > > > > >> > > > > > >> to declare operators stable, which is why they are > > marked > > > > >> > > evolving. > > > > >> > > > > > >> > > > > >> > > > > > >> Instead of finding reasons to avoid changes that > should > > > > have > > > > >> > been > > > > >> > > > > made a > > > > >> > > > > > >> long time ago, why not discuss what a "stable > operator" > > > is > > > > >> and > > > > >> > > which > > > > >> > > > > > ones > > > > >> > > > > > >> deserve to be in that category. Those are the ones > that > > > > will > > > > >> get > > > > >> > > the > > > > >> > > > > > >> backward compatibility stub. > > > > >> > > > > > >> > > > > >> > > > > > >> Thanks, > > > > >> > > > > > >> Thomas > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov < > > > > >> > > > v.ro...@datatorrent.com> > > > > >> > > > > > >> wrote: > > > > >> > > > > > >> > > > > >> > > > > > >> My preference is to agree that the next Malhar > release > > is > > > > >> 4.0.0 > > > > >> > > and > > > > >> > > > > make > > > > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise > > why > > > to > > > > >> keep > > > > >> > > > > > semantic > > > > >> > > > > > >>> versioning checks on Malhar in case there is no > > version > > > > >> > > > > compatibility. > > > > >> > > > > > >>> > > > > >> > > > > > >>> Thank you, > > > > >> > > > > > >>> > > > > >> > > > > > >>> Vlad > > > > >> > > > > > >>> > > > > >> > > > > > >>> > > > > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote: > > > > >> > > > > > >>> > > > > >> > > > > > >>> next release is 3.8.0 > > > > >> > > > > > >>>> > > > > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov < > > > > >> > > > > v.ro...@datatorrent.com> > > > > >> > > > > > >>>> wrote: > > > > >> > > > > > >>>> > > > > >> > > > > > >>>> next release - 3.9.0 or 4.0.0? > > > > >> > > > > > >>>> > > > > >> > > > > > >>>>> Thank you, > > > > >> > > > > > >>>>> > > > > >> > > > > > >>>>> Vlad > > > > >> > > > > > >>>>> > > > > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote: > > > > >> > > > > > >>>>> > > > > >> > > > > > >>>>> It is time to resurrect this thread and get going > > with > > > > the > > > > >> > > work. > > > > >> > > > > > >>>>> > > > > >> > > > > > >>>>>> For the next release, I will sign up to do the > > > package > > > > >> move > > > > >> > in > > > > >> > > > > > Malhar: > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> https://issues.apache.org/ > > > jira/browse/APEXMALHAR-2517 > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> In general this will be straightforward; most > > classes > > > > in > > > > >> > > Malhar > > > > >> > > > > are > > > > >> > > > > > >>>>>> marked > > > > >> > > > > > >>>>>> evolving and it is trivial for users to change > > import > > > > >> > > > statements. > > > > >> > > > > > >>>>>> However, > > > > >> > > > > > >>>>>> I would suggest to discuss if there are selected > > > > >> operators > > > > >> > > that > > > > >> > > > > are > > > > >> > > > > > >>>>>> worth > > > > >> > > > > > >>>>>> keeping a backward compatibility stub in the old > > > > >> location. > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> Here is my plan: > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* > > within > > > > one > > > > >> > PR - > > > > >> > > > > this > > > > >> > > > > > is > > > > >> > > > > > >>>>>> all > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> IDE automated work > > > > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in > > > > >> separate > > > > >> > PR > > > > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved > > > classes > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> Thanks, > > > > >> > > > > > >>>>>> Thomas > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni < > > > > >> > > > > > >>>>>> pra...@datatorrent.com > > > > >> > > > > > >>>>>> wrote: > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> Inline > > > > >> > > > > > >>>>>> > > > > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise < > > > > >> > > t...@apache.org> > > > > >> > > > > > >>>>>>> wrote: > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> --> > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod > Immaneni < > > > > >> > > > > > >>>>>>>> pra...@datatorrent.com > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> wrote: > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we > > do > > > > >> this as > > > > >> > > > part > > > > >> > > > > of > > > > >> > > > > > >>>>>>>> the > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith > > > modules > > > > >> into > > > > >> > > > baby > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> packages > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> and would also prefer deprecating the existing > > > > >> operators > > > > >> > in > > > > >> > > > > place. > > > > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for > > > > >> > > malhar-contrib > > > > >> > > > > and > > > > >> > > > > > >>>>>>>> given > > > > >> > > > > > >>>>>>>> the overall state of that module I think it is > OK > > > to > > > > >> defer > > > > >> > > > > package > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> renaming > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> there. I do however prefer to see the package > > rename > > > > >> > > addressed > > > > >> > > > > for > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> other > > > > >> > > > > > >>>>>>>> modules, especially for the main library > module. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> Should we consider breaking the library into > > > smaller > > > > >> > modules > > > > >> > > > as > > > > >> > > > > > >>>>>>>> well, > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> the > > > > >> > > > > > >>>>>>> file/block operators for example probably can be > > in > > > > >> their > > > > >> > own > > > > >> > > > > > module > > > > >> > > > > > >>>>>>> from > > > > >> > > > > > >>>>>>> just an organizational perspective. > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> This > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> will help us achieve two things. First, the > user > > > will > > > > >> see > > > > >> > > all > > > > >> > > > > the > > > > >> > > > > > >>>>>>>>> new > > > > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it > > > twice > > > > >> (with > > > > >> > > > > package > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> rename > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> and dependency changes) and second it will > allow > > > for > > > > a > > > > >> > > > smoother > > > > >> > > > > > >>>>>>>> transition > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> as the existing code will still work in a > > > deprecated > > > > >> > state. > > > > >> > > It > > > > >> > > > > > will > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>>> also > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For > > new > > > > >> > > operators, > > > > >> > > > > we > > > > >> > > > > > >>>>>>>> can > > > > >> > > > > > >>>>>>>> go > > > > >> > > > > > >>>>>>>> with the new package path but we need to ensure > > > they > > > > >> will > > > > >> > > get > > > > >> > > > > > moved > > > > >> > > > > > >>>>>>>> into > > > > >> > > > > > >>>>>>>> the baby packages as well. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> I think existing operators should be renamed so > > > that > > > > >> git > > > > >> > > > history > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> remains. A > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> possible solution for backward compatibility > could > > > be > > > > to > > > > >> > > > > > subsequently > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> add > > > > >> > > > > > >>>>>>>> empty subclasses in the previous location (for > > > > existing > > > > >> > > > concrete > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> operators > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> that we know are actually in use) to simplify > > > > migration > > > > >> for > > > > >> > > > > users. > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> Yes we can do that. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps > > are > > > > >> > typically > > > > >> > > > > used > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> wholesale > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> and the interface is typically manual > > interaction. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, > > > like > > > > >> the > > > > >> > > > > launcher > > > > >> > > > > > >>>>>>>>> api > > > > >> > > > > > >>>>>>>>> we > > > > >> > > > > > >>>>>>>>> added recently for example, we can go with new > > > > package > > > > >> > path > > > > >> > > > but > > > > >> > > > > > if > > > > >> > > > > > >>>>>>>>> we > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> are > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> making incremental additions to existing > > > > >> functionality, I > > > > >> > > feel > > > > >> > > > > it > > > > >> > > > > > is > > > > >> > > > > > >>>>>>>> better > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer > we > > > keep > > > > >> the > > > > >> > > > > package > > > > >> > > > > > of > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>>> the > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> implementation classes consistent with api, for > > > > >> > > > > understandability > > > > >> > > > > > >>>>>>>> and > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> readability of the code. So, for example, we > > don't > > > > >> change > > > > >> > > > > package > > > > >> > > > > > >>>>>>>>> path > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> of > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. > It > > > is > > > > >> > > > subjective, > > > > >> > > > > > but > > > > >> > > > > > >>>>>>>> it > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> will be good if we can also do the same with > > > classes > > > > >> > closely > > > > >> > > > > > related > > > > >> > > > > > >>>>>>>>> to > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> the > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can > > moving > > > > >> these > > > > >> > > on a > > > > >> > > > > > >>>>>>>> package > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>>> by > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> package basis, like everything in > > > > >> > > com.datatorrent.stram.engine > > > > >> > > > > > could > > > > >> > > > > > >>>>>>>> be > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> moved. For completely internal components like > > > buffer > > > > >> > > server, > > > > >> > > > we > > > > >> > > > > > can > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> move > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api > > and > > > > >> > classes, > > > > >> > > > when > > > > >> > > > > > we > > > > >> > > > > > >>>>>>>> go > > > > >> > > > > > >>>>>>>> to > > > > >> > > > > > >>>>>>>> next major release but would like to see if we > > can > > > > >> find a > > > > >> > > way > > > > >> > > > to > > > > >> > > > > > >>>>>>>> support > > > > >> > > > > > >>>>>>>> existing api for one more major release in > > > deprecated > > > > >> > mode. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> The point of the major release is to enable > > > backward > > > > >> > > > > incompatible > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> changes > > > > >> > > > > > >>>>>>>> and I don't think it is realistic to support > the > > > > >> existing > > > > >> > > API > > > > >> > > > > for > > > > >> > > > > > >>>>>>>> another > > > > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as > > most > > > > >> > existing > > > > >> > > > > > >>>>>>>> application > > > > >> > > > > > >>>>>>>> code refers to operators, attributes and the > > > > >> application > > > > >> > > > > > interface. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> Perhaps > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> it is possible to keep those around as interface > > > > >> extensions > > > > >> > > to > > > > >> > > > > help > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> migration. Custom operators may need to be > > migrated > > > > to > > > > >> > > reflect > > > > >> > > > > API > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> changes, > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> and I would consider that a reasonable task for > > > > operator > > > > >> > > > > developers > > > > >> > > > > > >>>>>>> as > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> part > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>> of a major upgrade. > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>>> It would be good if we can keep them as > > deprecated > > > > >> > interface > > > > >> > > > > > >>>>>>>> extensions > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> for > > > > >> > > > > > >>>>>>> one release to provide a smoother transition. > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> API and implementation in engine are kept > separate > > > > >> > > > intentionally. > > > > >> > > > > > >>>>>>> They > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> reside in different packages today, so I don't > > see a > > > > >> > problem > > > > >> > > > > > renaming > > > > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even > > when > > > > the > > > > >> API > > > > >> > > > > cannot > > > > >> > > > > > be > > > > >> > > > > > >>>>>>>> touched right away. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> They are different packages but sharing a > common > > > > prefix > > > > >> > with > > > > >> > > > api > > > > >> > > > > > >>>>>>>> will > > > > >> > > > > > >>>>>>>> be > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of > > > > >> > readability. > > > > >> > > > Not > > > > >> > > > > a > > > > >> > > > > > >>>>>>> big > > > > >> > > > > > >>>>>>> deal > > > > >> > > > > > >>>>>>> and can be changed. > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> Thanks > > > > >> > > > > > >>>>>>> > > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise < > > > > >> > > t...@apache.org> > > > > >> > > > > > >>>>>>>> wrote: > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>>> Hi, > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I > > think > > > it > > > > >> > > > warrants a > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> broader > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> discussion. > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> At the time of incubation, the decision was to > > > defer > > > > >> > change > > > > >> > > of > > > > >> > > > > > Java > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> packages from com.datatorrent to > > org.apache.apex > > > > till > > > > >> > next > > > > >> > > > > major > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> release > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> to > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> ensure backward compatibility for users. > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some > confusion, > > as > > > > >> > > > contributors > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> continue > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> to > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> add new code under legacy packages. > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for > > using > > > > Apex > > > > >> > > > continue > > > > >> > > > > > to > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> refer > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> to > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after > > > > >> > graduation. > > > > >> > > > More > > > > >> > > > > > and > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> more > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> user code is being built on top of something > > that > > > > >> needs > > > > >> > to > > > > >> > > > > > change, > > > > >> > > > > > >>>>>>>>> the > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> can > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> is being kicked down the road and users will > face > > > > more > > > > >> > > changes > > > > >> > > > > > >>>>>>>>> later. > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> I would like to propose the following: > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under > > > > >> > org.apache.apex > > > > >> > > > > > packages > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> 2. Not all code is under backward > compatibility > > > > >> > > restriction > > > > >> > > > > and > > > > >> > > > > > in > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> those > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> cases we can rename the packages right away. > > > > Examples: > > > > >> > > buffer > > > > >> > > > > > >>>>>>>>> server, > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators > can > > be > > > > >> > changed. > > > > >> > > > For > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> operators > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> we > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a > > > major > > > > >> > > release > > > > >> > > > as > > > > >> > > > > > >>>>>>>>> most > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> of > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the > > > ability > > > > >> to > > > > >> > > > > continue > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> using > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due > to > > > > >> engine > > > > >> > > > > backward > > > > >> > > > > > >>>>>>>> > > > > >> > > > > > >>>>>>>> compatibility guarantee. > > > > >> > > > > > >>>>>>>>> > > > > >> > > > > > >>>>>>>>>> Thanks, > > > > >> > > > > > >>>>>>>>>> Thomas > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > >>>>>>>>>> > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
