I think if there is a will to allow older apps to continue to benefit from improvements to operators, without changes, it can be done along with the proposed move, especially since we are not taking about all operators.
However, since that is not the case, I am fine with starting a 4.0 release line as long as we don't seal the future of 3.x now. What I would prefer is to create a release-3 branch and not a release-3.8 branch now and let the future determine if 3.8 will be the last 3.x minor release. Thanks On Fri, Aug 18, 2017 at 5:04 PM Thomas Weise <t...@apache.org> wrote: > The proposed change does NOT require changes to existing apps. It follows > what you see for example in the JDK where older classes are retired without > making breaking major release changes. > > Even though not required technically, it looks like the preference is to > make this change with a 4.0 release. I will therefore propose to change the > master branch to 4.0-SNAPSHOT and make this and other changes deemed > worthwhile. > > To me it is more important that to newcomers the codebase does not look > like an outdated legacy situation and therefore if I have to make a choice > than pretty much any version number will do for that. So far I have seen > very little interest and initiative in addressing maintenance work like the > packages, checkstyle, CI stability, documentation etc. So it would be great > if people interested would step up and contribute. > > Here is the next proposal how to proceed: > > - create release-3.8 branch for last 3.8 release right away > - update master to 4.0-SNAPSHOT as part of the open PR #664 > - identify further cleanup work for master > - release 4.0 > > Thanks, > Thomas > > > > > > > > > > On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <a...@datatorrent.com> wrote: > > > This following pull request should be taken up in 4.0.0. See my comments > in > > https://github.com/apache/apex-malhar/pull/664 > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com% > > 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg= > > AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ> > > https://github.com/apache/apex-malhar/pull/662 > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com% > > 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg= > > AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg> > > > > This merge should not be done without a consensus. This will require code > > changes to existing apps. > > > > Thks, > > Amol > > > > > > > > E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* > > > > www.datatorrent.com > > > > > > On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <t...@apache.org> wrote: > > > > > Hi, > > > > > > I opened following PRs for the package change: > > > > > > https://github.com/apache/apex-malhar/pull/662 > > > > > > Moves all classes with history retained (hence 2 commits). Also > contains > > > checkstyle and other mechanical changes. > > > > > > https://github.com/apache/apex-malhar/pull/664 > > > > > > Adds backward compatibility jar. > > > > > > Once above PRs are merged the new artifact can be deployed and > introduced > > > as dependency in malhar-library. > > > > > > Please review. > > > > > > Thanks, > > > Thomas > > > > > > > > > > > > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <t...@apache.org> wrote: > > > > > > > My original list of work items contained the b/w compatibility > aspect, > > I > > > > don't think there should be any confusion of whether it will be > covered > > > > here or not. > > > > > > > > The proposed shading will provide the old classes and they will be > > frozen > > > > as of release 3.7. That's the same as making a copy of the code and > > never > > > > again making changes to the original classes. This cannot be > > accomplished > > > > by using the older 3.7 release in your project because you cannot > use 2 > > > > different versions of Malhar in parallel unless you apply shading. > > > > > > > > The shaded artifact will only expose the com.datatorrent classes, and > > > they > > > > will be self-contained as the rest of the classes that they may > depend > > on > > > > are shaded. The shaded artifact does not evolve, there are not more > > > changes > > > > to com.datatorrent classes after they are relocated in master. > > > > > > > > Thanks, > > > > Thomas > > > > > > > > > > > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni < > > pra...@datatorrent.com > > > > > > > > wrote: > > > > > > > >> I don't think we can limit the topic strictly to relocation without > > > having > > > >> a good b/w compatibility story or at least one that goes far enough. > > > >> > > > >> The shading idea sounds interesting. Why not let the shaded version > > move > > > >> forward with each release till we hit a major release. If it is > going > > to > > > >> remain pegged at 3.7.0, why shade in the first place as the regular > > > 3.7.0 > > > >> release would do the same job and it would be same as the loss of > b/w > > > >> compatibility with newer releases. > > > >> > > > >> Thanks > > > >> > > > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <t...@apache.org> > wrote: > > > >> > > > >> > Discussing what in the future might become stable needs to be a > > > separate > > > >> > thread, it will be a much bigger discussion. > > > >> > > > > >> > The topic here is to relocate the packages. With a few exceptions > > > >> > relocation won't affect the semantic versioning. Semantic > versioning > > > is > > > >> > essentially not effective for Malhar because almost everything is > > > >> @Evolving > > > >> > (and there are reasons for that.. -> separate topic) > > > >> > > > > >> > I don't really like the idea of creating bw compatibility stubs > for > > > the > > > >> > follow up PR. It creates even more clutter in the source tree than > > > there > > > >> > already is and so here is an alternative suggestion: > > > >> > > > > >> > https://github.com/tweise/apex-malhar/blob/malhar37- > > > >> > compat/shaded-malhar37/pom.xml > > > >> > > > > >> > Create a shaded artifact that provides the old com.datatorrent.* > > > >> classes as > > > >> > of release 3.7. Users can include that artifact if they don't want > > to > > > >> > change import statements. At the same time they have an incentive > to > > > >> switch > > > >> > to the relocated classes to take advantage of bug fixes and new > > > >> > functionality. > > > >> > > > > >> > I will work on the first PR that does the relocate. In the > meantime, > > > we > > > >> can > > > >> > finalize what backward compatibility support we want to provide > and > > > how. > > > >> > > > > >> > Thanks, > > > >> > Thomas > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni < > > > >> pra...@datatorrent.com> > > > >> > wrote: > > > >> > > > > >> > > How about coming up with a list of @Evolving operators that we > > would > > > >> like > > > >> > > to see in the eventual stable list and move those along with the > > not > > > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the > > rest > > > as > > > >> > they > > > >> > > are. Then have a follow up JIRA for the rest to be moved over to > > > >> contrib > > > >> > > and be deprecated. > > > >> > > > > > >> > > Thanks > > > >> > > > > > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise < > > > >> thomas.we...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > > > We need to keep the discussion here on topic, if other things > > are > > > >> piled > > > >> > > on > > > >> > > > top then nothing gets done. > > > >> > > > > > > >> > > > Most operators today are not stable, they are @Evolving. So > > > perhaps > > > >> it > > > >> > is > > > >> > > > necessary to have a separate discussion about that, outside of > > > this > > > >> > > thread. > > > >> > > > My guess is that there are only a few operators that we could > > > >> declare > > > >> > > > stable. Specifically, under contrib the closest one might have > > > been > > > >> > > Kafka, > > > >> > > > but that is already superseded by the newer versions. > > > >> > > > > > > >> > > > Thomas > > > >> > > > > > > >> > > > > > > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni < > > > >> > > pra...@datatorrent.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > We had a discussion a while back, agreed to relegate > > non-stable > > > >> and > > > >> > > > > experimental operators to contrib and also added this to > > > >> contribution > > > >> > > > > guidelines. We aexecuted on this and cleaned up the repo by > > > moving > > > >> > > > > operators deemed non-suitable for production use at that > time > > to > > > >> > > contrib. > > > >> > > > > So I wouldn't say the operators that are at the top level > > today > > > or > > > >> > the > > > >> > > > ones > > > >> > > > > in library are 0.x.x quality. Granted, we may need to do one > > > more > > > >> > pass > > > >> > > as > > > >> > > > > some of the operators may no longer be considered the right > > > >> > > > implementations > > > >> > > > > with the advent of the windowed operator. > > > >> > > > > > > > >> > > > > Since contrib used to be the place that housed operators > that > > > >> > required > > > >> > > > > third party libraries, there are some production quality > > > >> operators in > > > >> > > > there > > > >> > > > > that need to be pulled out into top level modules. > > > >> > > > > > > > >> > > > > I think we are in agreement that for operators that we > > consider > > > >> > stable, > > > >> > > > we > > > >> > > > > should provide a b/w stub. I would add that we strongly > > consider > > > >> > > creating > > > >> > > > > the org.apache.apex counterparts of any stable operators > that > > > are > > > >> in > > > >> > > > > contrib out in top level modules and have the > com.datatorrent > > > >> stubs > > > >> > in > > > >> > > > > contrib. > > > >> > > > > > > > >> > > > > For the operators not considered stable, I would prefer we > > > either > > > >> > leave > > > >> > > > the > > > >> > > > > current package path as is or if we change the package path > > then > > > >> > create > > > >> > > > the > > > >> > > > > b/w stub as I am not sure how widely they are in use (so, in > > > >> essence, > > > >> > > > > preserve semantic versioning). It would be good if there is > a > > > >> > followup > > > >> > > > JIRA > > > >> > > > > that takes a look at what other operators can be moved to > > > contrib > > > >> > with > > > >> > > > the > > > >> > > > > advent of the newer frameworks and understanding. > > > >> > > > > > > > >> > > > > Thanks > > > >> > > > > > > > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise < > t...@apache.org > > > > > > >> > wrote: > > > >> > > > > > > > >> > > > > > Most of the operators evolve, as is quite clearly > indicated > > by > > > >> > > > @Evolving > > > >> > > > > > annotations. So while perhaps 0.x.x would be a more > > > appropriate > > > >> > > version > > > >> > > > > > number, I don't think you can change that. > > > >> > > > > > > > > >> > > > > > Thomas > > > >> > > > > > > > > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov < > > > >> > v.ro...@datatorrent.com > > > >> > > > > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > If entire library is not stable, its version should be > > 0.x.x > > > >> and > > > >> > > > there > > > >> > > > > > > should not be any semantic versioning enabled or > implied. > > It > > > >> > > evolves. > > > >> > > > > If > > > >> > > > > > > some operators are stable as 3.8.x version suggests, the > > > >> library > > > >> > > > should > > > >> > > > > > > follow semantic versioning and it is not OK to make a > > major > > > >> > change > > > >> > > > that > > > >> > > > > > > breaks semantic versioning across the entire library. It > > is > > > >> not a > > > >> > > > > finding > > > >> > > > > > > an excuse not to make a change. For me semantic > versioning > > > and > > > >> > > > backward > > > >> > > > > > > compatibility is more important than a proper package > > name. > > > >> > > > > > > > > > >> > > > > > > Thank you, > > > >> > > > > > > > > > >> > > > > > > Vlad > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote: > > > >> > > > > > > > > > >> > > > > > >> Semantic versioning makes sense when you have a stable > > > >> baseline. > > > >> > > As > > > >> > > > > long > > > >> > > > > > >> as > > > >> > > > > > >> frequent fixes need to be made to address basic issues, > > it > > > >> makes > > > >> > > no > > > >> > > > > > sense > > > >> > > > > > >> to declare operators stable, which is why they are > marked > > > >> > > evolving. > > > >> > > > > > >> > > > >> > > > > > >> Instead of finding reasons to avoid changes that should > > > have > > > >> > been > > > >> > > > > made a > > > >> > > > > > >> long time ago, why not discuss what a "stable operator" > > is > > > >> and > > > >> > > which > > > >> > > > > > ones > > > >> > > > > > >> deserve to be in that category. Those are the ones that > > > will > > > >> get > > > >> > > the > > > >> > > > > > >> backward compatibility stub. > > > >> > > > > > >> > > > >> > > > > > >> Thanks, > > > >> > > > > > >> Thomas > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov < > > > >> > > > v.ro...@datatorrent.com> > > > >> > > > > > >> wrote: > > > >> > > > > > >> > > > >> > > > > > >> My preference is to agree that the next Malhar release > is > > > >> 4.0.0 > > > >> > > and > > > >> > > > > make > > > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise > why > > to > > > >> keep > > > >> > > > > > semantic > > > >> > > > > > >>> versioning checks on Malhar in case there is no > version > > > >> > > > > compatibility. > > > >> > > > > > >>> > > > >> > > > > > >>> Thank you, > > > >> > > > > > >>> > > > >> > > > > > >>> Vlad > > > >> > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote: > > > >> > > > > > >>> > > > >> > > > > > >>> next release is 3.8.0 > > > >> > > > > > >>>> > > > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov < > > > >> > > > > v.ro...@datatorrent.com> > > > >> > > > > > >>>> wrote: > > > >> > > > > > >>>> > > > >> > > > > > >>>> next release - 3.9.0 or 4.0.0? > > > >> > > > > > >>>> > > > >> > > > > > >>>>> Thank you, > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> Vlad > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote: > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> It is time to resurrect this thread and get going > with > > > the > > > >> > > work. > > > >> > > > > > >>>>> > > > >> > > > > > >>>>>> For the next release, I will sign up to do the > > package > > > >> move > > > >> > in > > > >> > > > > > Malhar: > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> https://issues.apache.org/ > > jira/browse/APEXMALHAR-2517 > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> In general this will be straightforward; most > classes > > > in > > > >> > > Malhar > > > >> > > > > are > > > >> > > > > > >>>>>> marked > > > >> > > > > > >>>>>> evolving and it is trivial for users to change > import > > > >> > > > statements. > > > >> > > > > > >>>>>> However, > > > >> > > > > > >>>>>> I would suggest to discuss if there are selected > > > >> operators > > > >> > > that > > > >> > > > > are > > > >> > > > > > >>>>>> worth > > > >> > > > > > >>>>>> keeping a backward compatibility stub in the old > > > >> location. > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Here is my plan: > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* > within > > > one > > > >> > PR - > > > >> > > > > this > > > >> > > > > > is > > > >> > > > > > >>>>>> all > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> IDE automated work > > > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in > > > >> separate > > > >> > PR > > > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved > > classes > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Thanks, > > > >> > > > > > >>>>>> Thomas > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni < > > > >> > > > > > >>>>>> pra...@datatorrent.com > > > >> > > > > > >>>>>> wrote: > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Inline > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise < > > > >> > > t...@apache.org> > > > >> > > > > > >>>>>>> wrote: > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> --> > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni < > > > >> > > > > > >>>>>>>> pra...@datatorrent.com > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> wrote: > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we > do > > > >> this as > > > >> > > > part > > > >> > > > > of > > > >> > > > > > >>>>>>>> the > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith > > modules > > > >> into > > > >> > > > baby > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> packages > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> and would also prefer deprecating the existing > > > >> operators > > > >> > in > > > >> > > > > place. > > > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for > > > >> > > malhar-contrib > > > >> > > > > and > > > >> > > > > > >>>>>>>> given > > > >> > > > > > >>>>>>>> the overall state of that module I think it is OK > > to > > > >> defer > > > >> > > > > package > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> renaming > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> there. I do however prefer to see the package > rename > > > >> > > addressed > > > >> > > > > for > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> other > > > >> > > > > > >>>>>>>> modules, especially for the main library module. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> Should we consider breaking the library into > > smaller > > > >> > modules > > > >> > > > as > > > >> > > > > > >>>>>>>> well, > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> the > > > >> > > > > > >>>>>>> file/block operators for example probably can be > in > > > >> their > > > >> > own > > > >> > > > > > module > > > >> > > > > > >>>>>>> from > > > >> > > > > > >>>>>>> just an organizational perspective. > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> This > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> will help us achieve two things. First, the user > > will > > > >> see > > > >> > > all > > > >> > > > > the > > > >> > > > > > >>>>>>>>> new > > > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it > > twice > > > >> (with > > > >> > > > > package > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> rename > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> and dependency changes) and second it will allow > > for > > > a > > > >> > > > smoother > > > >> > > > > > >>>>>>>> transition > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> as the existing code will still work in a > > deprecated > > > >> > state. > > > >> > > It > > > >> > > > > > will > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>>> also > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For > new > > > >> > > operators, > > > >> > > > > we > > > >> > > > > > >>>>>>>> can > > > >> > > > > > >>>>>>>> go > > > >> > > > > > >>>>>>>> with the new package path but we need to ensure > > they > > > >> will > > > >> > > get > > > >> > > > > > moved > > > >> > > > > > >>>>>>>> into > > > >> > > > > > >>>>>>>> the baby packages as well. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> I think existing operators should be renamed so > > that > > > >> git > > > >> > > > history > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> remains. A > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> possible solution for backward compatibility could > > be > > > to > > > >> > > > > > subsequently > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> add > > > >> > > > > > >>>>>>>> empty subclasses in the previous location (for > > > existing > > > >> > > > concrete > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> operators > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> that we know are actually in use) to simplify > > > migration > > > >> for > > > >> > > > > users. > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> Yes we can do that. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps > are > > > >> > typically > > > >> > > > > used > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> wholesale > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> and the interface is typically manual > interaction. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, > > like > > > >> the > > > >> > > > > launcher > > > >> > > > > > >>>>>>>>> api > > > >> > > > > > >>>>>>>>> we > > > >> > > > > > >>>>>>>>> added recently for example, we can go with new > > > package > > > >> > path > > > >> > > > but > > > >> > > > > > if > > > >> > > > > > >>>>>>>>> we > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> are > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> making incremental additions to existing > > > >> functionality, I > > > >> > > feel > > > >> > > > > it > > > >> > > > > > is > > > >> > > > > > >>>>>>>> better > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer we > > keep > > > >> the > > > >> > > > > package > > > >> > > > > > of > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>>> the > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> implementation classes consistent with api, for > > > >> > > > > understandability > > > >> > > > > > >>>>>>>> and > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> readability of the code. So, for example, we > don't > > > >> change > > > >> > > > > package > > > >> > > > > > >>>>>>>>> path > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> of > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It > > is > > > >> > > > subjective, > > > >> > > > > > but > > > >> > > > > > >>>>>>>> it > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> will be good if we can also do the same with > > classes > > > >> > closely > > > >> > > > > > related > > > >> > > > > > >>>>>>>>> to > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> the > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can > moving > > > >> these > > > >> > > on a > > > >> > > > > > >>>>>>>> package > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>>> by > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> package basis, like everything in > > > >> > > com.datatorrent.stram.engine > > > >> > > > > > could > > > >> > > > > > >>>>>>>> be > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> moved. For completely internal components like > > buffer > > > >> > > server, > > > >> > > > we > > > >> > > > > > can > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> move > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api > and > > > >> > classes, > > > >> > > > when > > > >> > > > > > we > > > >> > > > > > >>>>>>>> go > > > >> > > > > > >>>>>>>> to > > > >> > > > > > >>>>>>>> next major release but would like to see if we > can > > > >> find a > > > >> > > way > > > >> > > > to > > > >> > > > > > >>>>>>>> support > > > >> > > > > > >>>>>>>> existing api for one more major release in > > deprecated > > > >> > mode. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> The point of the major release is to enable > > backward > > > >> > > > > incompatible > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> changes > > > >> > > > > > >>>>>>>> and I don't think it is realistic to support the > > > >> existing > > > >> > > API > > > >> > > > > for > > > >> > > > > > >>>>>>>> another > > > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as > most > > > >> > existing > > > >> > > > > > >>>>>>>> application > > > >> > > > > > >>>>>>>> code refers to operators, attributes and the > > > >> application > > > >> > > > > > interface. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> Perhaps > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> it is possible to keep those around as interface > > > >> extensions > > > >> > > to > > > >> > > > > help > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> migration. Custom operators may need to be > migrated > > > to > > > >> > > reflect > > > >> > > > > API > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> changes, > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> and I would consider that a reasonable task for > > > operator > > > >> > > > > developers > > > >> > > > > > >>>>>>> as > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> part > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>> of a major upgrade. > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>>> It would be good if we can keep them as > deprecated > > > >> > interface > > > >> > > > > > >>>>>>>> extensions > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> for > > > >> > > > > > >>>>>>> one release to provide a smoother transition. > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> API and implementation in engine are kept separate > > > >> > > > intentionally. > > > >> > > > > > >>>>>>> They > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> reside in different packages today, so I don't > see a > > > >> > problem > > > >> > > > > > renaming > > > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even > when > > > the > > > >> API > > > >> > > > > cannot > > > >> > > > > > be > > > >> > > > > > >>>>>>>> touched right away. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> They are different packages but sharing a common > > > prefix > > > >> > with > > > >> > > > api > > > >> > > > > > >>>>>>>> will > > > >> > > > > > >>>>>>>> be > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of > > > >> > readability. > > > >> > > > Not > > > >> > > > > a > > > >> > > > > > >>>>>>> big > > > >> > > > > > >>>>>>> deal > > > >> > > > > > >>>>>>> and can be changed. > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> Thanks > > > >> > > > > > >>>>>>> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise < > > > >> > > t...@apache.org> > > > >> > > > > > >>>>>>>> wrote: > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>>> Hi, > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I > think > > it > > > >> > > > warrants a > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> broader > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> discussion. > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> At the time of incubation, the decision was to > > defer > > > >> > change > > > >> > > of > > > >> > > > > > Java > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> packages from com.datatorrent to > org.apache.apex > > > till > > > >> > next > > > >> > > > > major > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> release > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> to > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> ensure backward compatibility for users. > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, > as > > > >> > > > contributors > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> continue > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> to > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> add new code under legacy packages. > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for > using > > > Apex > > > >> > > > continue > > > >> > > > > > to > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> refer > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> to > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after > > > >> > graduation. > > > >> > > > More > > > >> > > > > > and > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> more > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> user code is being built on top of something > that > > > >> needs > > > >> > to > > > >> > > > > > change, > > > >> > > > > > >>>>>>>>> the > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> can > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> is being kicked down the road and users will face > > > more > > > >> > > changes > > > >> > > > > > >>>>>>>>> later. > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> I would like to propose the following: > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under > > > >> > org.apache.apex > > > >> > > > > > packages > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility > > > >> > > restriction > > > >> > > > > and > > > >> > > > > > in > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> those > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> cases we can rename the packages right away. > > > Examples: > > > >> > > buffer > > > >> > > > > > >>>>>>>>> server, > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can > be > > > >> > changed. > > > >> > > > For > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> operators > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> we > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a > > major > > > >> > > release > > > >> > > > as > > > >> > > > > > >>>>>>>>> most > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> of > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the > > ability > > > >> to > > > >> > > > > continue > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> using > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to > > > >> engine > > > >> > > > > backward > > > >> > > > > > >>>>>>>> > > > >> > > > > > >>>>>>>> compatibility guarantee. > > > >> > > > > > >>>>>>>>> > > > >> > > > > > >>>>>>>>>> Thanks, > > > >> > > > > > >>>>>>>>>> Thomas > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > >>>>>>>>>> > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
