Discussing what in the future might become stable needs to be a separate thread, it will be a much bigger discussion.
The topic here is to relocate the packages. With a few exceptions relocation won't affect the semantic versioning. Semantic versioning is essentially not effective for Malhar because almost everything is @Evolving (and there are reasons for that.. -> separate topic) I don't really like the idea of creating bw compatibility stubs for the follow up PR. It creates even more clutter in the source tree than there already is and so here is an alternative suggestion: https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml Create a shaded artifact that provides the old com.datatorrent.* classes as of release 3.7. Users can include that artifact if they don't want to change import statements. At the same time they have an incentive to switch to the relocated classes to take advantage of bug fixes and new functionality. I will work on the first PR that does the relocate. In the meantime, we can finalize what backward compatibility support we want to provide and how. Thanks, Thomas On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pra...@datatorrent.com> wrote: > How about coming up with a list of @Evolving operators that we would like > to see in the eventual stable list and move those along with the not > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as they > are. Then have a follow up JIRA for the rest to be moved over to contrib > and be deprecated. > > Thanks > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <thomas.we...@gmail.com> > wrote: > > > We need to keep the discussion here on topic, if other things are piled > on > > top then nothing gets done. > > > > Most operators today are not stable, they are @Evolving. So perhaps it is > > necessary to have a separate discussion about that, outside of this > thread. > > My guess is that there are only a few operators that we could declare > > stable. Specifically, under contrib the closest one might have been > Kafka, > > but that is already superseded by the newer versions. > > > > Thomas > > > > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni < > pra...@datatorrent.com> > > wrote: > > > > > We had a discussion a while back, agreed to relegate non-stable and > > > experimental operators to contrib and also added this to contribution > > > guidelines. We aexecuted on this and cleaned up the repo by moving > > > operators deemed non-suitable for production use at that time to > contrib. > > > So I wouldn't say the operators that are at the top level today or the > > ones > > > in library are 0.x.x quality. Granted, we may need to do one more pass > as > > > some of the operators may no longer be considered the right > > implementations > > > with the advent of the windowed operator. > > > > > > Since contrib used to be the place that housed operators that required > > > third party libraries, there are some production quality operators in > > there > > > that need to be pulled out into top level modules. > > > > > > I think we are in agreement that for operators that we consider stable, > > we > > > should provide a b/w stub. I would add that we strongly consider > creating > > > the org.apache.apex counterparts of any stable operators that are in > > > contrib out in top level modules and have the com.datatorrent stubs in > > > contrib. > > > > > > For the operators not considered stable, I would prefer we either leave > > the > > > current package path as is or if we change the package path then create > > the > > > b/w stub as I am not sure how widely they are in use (so, in essence, > > > preserve semantic versioning). It would be good if there is a followup > > JIRA > > > that takes a look at what other operators can be moved to contrib with > > the > > > advent of the newer frameworks and understanding. > > > > > > Thanks > > > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <t...@apache.org> wrote: > > > > > > > Most of the operators evolve, as is quite clearly indicated by > > @Evolving > > > > annotations. So while perhaps 0.x.x would be a more appropriate > version > > > > number, I don't think you can change that. > > > > > > > > Thomas > > > > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.ro...@datatorrent.com > > > > > > wrote: > > > > > > > > > If entire library is not stable, its version should be 0.x.x and > > there > > > > > should not be any semantic versioning enabled or implied. It > evolves. > > > If > > > > > some operators are stable as 3.8.x version suggests, the library > > should > > > > > follow semantic versioning and it is not OK to make a major change > > that > > > > > breaks semantic versioning across the entire library. It is not a > > > finding > > > > > an excuse not to make a change. For me semantic versioning and > > backward > > > > > compatibility is more important than a proper package name. > > > > > > > > > > Thank you, > > > > > > > > > > Vlad > > > > > > > > > > > > > > > On 7/14/17 09:11, Thomas Weise wrote: > > > > > > > > > >> Semantic versioning makes sense when you have a stable baseline. > As > > > long > > > > >> as > > > > >> frequent fixes need to be made to address basic issues, it makes > no > > > > sense > > > > >> to declare operators stable, which is why they are marked > evolving. > > > > >> > > > > >> Instead of finding reasons to avoid changes that should have been > > > made a > > > > >> long time ago, why not discuss what a "stable operator" is and > which > > > > ones > > > > >> deserve to be in that category. Those are the ones that will get > the > > > > >> backward compatibility stub. > > > > >> > > > > >> Thanks, > > > > >> Thomas > > > > >> > > > > >> > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov < > > v.ro...@datatorrent.com> > > > > >> wrote: > > > > >> > > > > >> My preference is to agree that the next Malhar release is 4.0.0 > and > > > make > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep > > > > semantic > > > > >>> versioning checks on Malhar in case there is no version > > > compatibility. > > > > >>> > > > > >>> Thank you, > > > > >>> > > > > >>> Vlad > > > > >>> > > > > >>> > > > > >>> On 7/14/17 08:11, Thomas Weise wrote: > > > > >>> > > > > >>> next release is 3.8.0 > > > > >>>> > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov < > > > v.ro...@datatorrent.com> > > > > >>>> wrote: > > > > >>>> > > > > >>>> next release - 3.9.0 or 4.0.0? > > > > >>>> > > > > >>>>> Thank you, > > > > >>>>> > > > > >>>>> Vlad > > > > >>>>> > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote: > > > > >>>>> > > > > >>>>> It is time to resurrect this thread and get going with the > work. > > > > >>>>> > > > > >>>>>> For the next release, I will sign up to do the package move in > > > > Malhar: > > > > >>>>>> > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517 > > > > >>>>>> > > > > >>>>>> In general this will be straightforward; most classes in > Malhar > > > are > > > > >>>>>> marked > > > > >>>>>> evolving and it is trivial for users to change import > > statements. > > > > >>>>>> However, > > > > >>>>>> I would suggest to discuss if there are selected operators > that > > > are > > > > >>>>>> worth > > > > >>>>>> keeping a backward compatibility stub in the old location. > > > > >>>>>> > > > > >>>>>> Here is my plan: > > > > >>>>>> > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - > > > this > > > > is > > > > >>>>>> all > > > > >>>>>> > > > > >>>>>> IDE automated work > > > > >>>>>> 2. Add backward compatibility classes, if, any in separate PR > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes > > > > >>>>>> > > > > >>>>>> Thanks, > > > > >>>>>> Thomas > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni < > > > > >>>>>> pra...@datatorrent.com > > > > >>>>>> wrote: > > > > >>>>>> > > > > >>>>>> Inline > > > > >>>>>> > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise < > t...@apache.org> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>> --> > > > > >>>>>>> > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni < > > > > >>>>>>>> pra...@datatorrent.com > > > > >>>>>>>> > > > > >>>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>> For malhar, for existing operators, I prefer we do this as > > part > > > of > > > > >>>>>>>> the > > > > >>>>>>>> > > > > >>>>>>>> planned refactoring for breaking the monolith modules into > > baby > > > > >>>>>>>>> > > > > >>>>>>>>> packages > > > > >>>>>>>>> > > > > >>>>>>>> and would also prefer deprecating the existing operators in > > > place. > > > > >>>>>>>> Refactor into smaller modules was discussed for > malhar-contrib > > > and > > > > >>>>>>>> given > > > > >>>>>>>> the overall state of that module I think it is OK to defer > > > package > > > > >>>>>>>> > > > > >>>>>>>> renaming > > > > >>>>>>>> > > > > >>>>>>> there. I do however prefer to see the package rename > addressed > > > for > > > > >>>>>>> > > > > >>>>>>>> other > > > > >>>>>>>> modules, especially for the main library module. > > > > >>>>>>>> > > > > >>>>>>>> Should we consider breaking the library into smaller modules > > as > > > > >>>>>>>> well, > > > > >>>>>>>> > > > > >>>>>>>> the > > > > >>>>>>> file/block operators for example probably can be in their own > > > > module > > > > >>>>>>> from > > > > >>>>>>> just an organizational perspective. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> This > > > > >>>>>>> > > > > >>>>>>>> will help us achieve two things. First, the user will see > all > > > the > > > > >>>>>>>>> new > > > > >>>>>>>>> changes at once as opposed to dealing with it twice (with > > > package > > > > >>>>>>>>> > > > > >>>>>>>>> rename > > > > >>>>>>>>> > > > > >>>>>>>> and dependency changes) and second it will allow for a > > smoother > > > > >>>>>>>> transition > > > > >>>>>>>> > > > > >>>>>>>> as the existing code will still work in a deprecated state. > It > > > > will > > > > >>>>>>>> > > > > >>>>>>>>> also > > > > >>>>>>>>> > > > > >>>>>>>> give a more consistent structure to malhar. For new > operators, > > > we > > > > >>>>>>>> can > > > > >>>>>>>> go > > > > >>>>>>>> with the new package path but we need to ensure they will > get > > > > moved > > > > >>>>>>>> into > > > > >>>>>>>> the baby packages as well. > > > > >>>>>>>> > > > > >>>>>>>> I think existing operators should be renamed so that git > > history > > > > >>>>>>>>> > > > > >>>>>>>>> remains. A > > > > >>>>>>>> > > > > >>>>>>> possible solution for backward compatibility could be to > > > > subsequently > > > > >>>>>>> > > > > >>>>>>>> add > > > > >>>>>>>> empty subclasses in the previous location (for existing > > concrete > > > > >>>>>>>> > > > > >>>>>>>> operators > > > > >>>>>>>> > > > > >>>>>>> that we know are actually in use) to simplify migration for > > > users. > > > > >>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> Yes we can do that. > > > > >>>>>>>> > > > > >>>>>>>> For demos, we can modify the paths as the apps are typically > > > used > > > > >>>>>>> > > > > >>>>>>> wholesale > > > > >>>>>>>> > > > > >>>>>>>> and the interface is typically manual interaction. > > > > >>>>>>>> > > > > >>>>>>>>> For core, if we are adding new api subsystems, like the > > > launcher > > > > >>>>>>>>> api > > > > >>>>>>>>> we > > > > >>>>>>>>> added recently for example, we can go with new package path > > but > > > > if > > > > >>>>>>>>> we > > > > >>>>>>>>> > > > > >>>>>>>>> are > > > > >>>>>>>>> > > > > >>>>>>>> making incremental additions to existing functionality, I > feel > > > it > > > > is > > > > >>>>>>>> better > > > > >>>>>>>> > > > > >>>>>>>> to keep it in the same package. I also prefer we keep the > > > package > > > > of > > > > >>>>>>>> > > > > >>>>>>>>> the > > > > >>>>>>>>> > > > > >>>>>>>> implementation classes consistent with api, for > > > understandability > > > > >>>>>>>> and > > > > >>>>>>>> > > > > >>>>>>>> readability of the code. So, for example, we don't change > > > package > > > > >>>>>>>>> path > > > > >>>>>>>>> > > > > >>>>>>>>> of > > > > >>>>>>>>> > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is > > subjective, > > > > but > > > > >>>>>>>> it > > > > >>>>>>>> > > > > >>>>>>>> will be good if we can also do the same with classes closely > > > > related > > > > >>>>>>>>> to > > > > >>>>>>>>> > > > > >>>>>>>>> the > > > > >>>>>>>>> > > > > >>>>>>>> implementation classes as well. Maybe we can moving these > on a > > > > >>>>>>>> package > > > > >>>>>>>> > > > > >>>>>>>>> by > > > > >>>>>>>>> > > > > >>>>>>>> package basis, like everything in > com.datatorrent.stram.engine > > > > could > > > > >>>>>>>> be > > > > >>>>>>>> > > > > >>>>>>>> moved. For completely internal components like buffer > server, > > we > > > > can > > > > >>>>>>>>> > > > > >>>>>>>>> move > > > > >>>>>>>>> > > > > >>>>>>>> them wholesale. We can consider moving all api and classes, > > when > > > > we > > > > >>>>>>>> go > > > > >>>>>>>> to > > > > >>>>>>>> next major release but would like to see if we can find a > way > > to > > > > >>>>>>>> support > > > > >>>>>>>> existing api for one more major release in deprecated mode. > > > > >>>>>>>> > > > > >>>>>>>> The point of the major release is to enable backward > > > incompatible > > > > >>>>>>>>> > > > > >>>>>>>>> changes > > > > >>>>>>>> and I don't think it is realistic to support the existing > API > > > for > > > > >>>>>>>> another > > > > >>>>>>>> major release. IMO it is also not necessary as most existing > > > > >>>>>>>> application > > > > >>>>>>>> code refers to operators, attributes and the application > > > > interface. > > > > >>>>>>>> > > > > >>>>>>>> Perhaps > > > > >>>>>>>> > > > > >>>>>>> it is possible to keep those around as interface extensions > to > > > help > > > > >>>>>>> > > > > >>>>>>>> migration. Custom operators may need to be migrated to > reflect > > > API > > > > >>>>>>>> > > > > >>>>>>>> changes, > > > > >>>>>>>> > > > > >>>>>>> and I would consider that a reasonable task for operator > > > developers > > > > >>>>>>> as > > > > >>>>>>> > > > > >>>>>>>> part > > > > >>>>>>>> > > > > >>>>>>> of a major upgrade. > > > > >>>>>>> > > > > >>>>>>>> It would be good if we can keep them as deprecated interface > > > > >>>>>>>> extensions > > > > >>>>>>>> > > > > >>>>>>>> for > > > > >>>>>>> one release to provide a smoother transition. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> API and implementation in engine are kept separate > > intentionally. > > > > >>>>>>> They > > > > >>>>>>> > > > > >>>>>>> reside in different packages today, so I don't see a problem > > > > renaming > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API > > > cannot > > > > be > > > > >>>>>>>> touched right away. > > > > >>>>>>>> > > > > >>>>>>>> They are different packages but sharing a common prefix with > > api > > > > >>>>>>>> will > > > > >>>>>>>> be > > > > >>>>>>>> > > > > >>>>>>>> helpful to someone new to codebase in terms of readability. > > Not > > > a > > > > >>>>>>> big > > > > >>>>>>> deal > > > > >>>>>>> and can be changed. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> Thanks > > > > >>>>>>> > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise < > t...@apache.org> > > > > >>>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> Hi, > > > > >>>>>>>>> > > > > >>>>>>>>> This topic has come up on several PRs and I think it > > warrants a > > > > >>>>>>>>>> > > > > >>>>>>>>>> broader > > > > >>>>>>>>>> > > > > >>>>>>>>> discussion. > > > > >>>>>>>> > > > > >>>>>>>> At the time of incubation, the decision was to defer change > of > > > > Java > > > > >>>>>>>>> > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next > > > major > > > > >>>>>>>>>> > > > > >>>>>>>>>> release > > > > >>>>>>>>>> > > > > >>>>>>>>> to > > > > >>>>>>>>> > > > > >>>>>>>>> ensure backward compatibility for users. > > > > >>>>>>>>> > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as > > contributors > > > > >>>>>>>>>> > > > > >>>>>>>>>> continue > > > > >>>>>>>>>> > > > > >>>>>>>>> to > > > > >>>>>>>> > > > > >>>>>>>> add new code under legacy packages. > > > > >>>>>>>>> > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex > > continue > > > > to > > > > >>>>>>>>>> > > > > >>>>>>>>>> refer > > > > >>>>>>>>>> > > > > >>>>>>>>> to > > > > >>>>>>>> > > > > >>>>>>>> com.datatorrent packages, nearly one year after graduation. > > More > > > > and > > > > >>>>>>>>> > > > > >>>>>>>>>> more > > > > >>>>>>>>>> > > > > >>>>>>>>> user code is being built on top of something that needs to > > > > change, > > > > >>>>>>>>> the > > > > >>>>>>>>> > > > > >>>>>>>>> can > > > > >>>>>>>> > > > > >>>>>>>> is being kicked down the road and users will face more > changes > > > > >>>>>>>>> later. > > > > >>>>>>>>> > > > > >>>>>>>>>> I would like to propose the following: > > > > >>>>>>>>>> > > > > >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex > > > > packages > > > > >>>>>>>>>> > > > > >>>>>>>>>> 2. Not all code is under backward compatibility > restriction > > > and > > > > in > > > > >>>>>>>>>> > > > > >>>>>>>>>> those > > > > >>>>>>>>>> > > > > >>>>>>>>> cases we can rename the packages right away. Examples: > buffer > > > > >>>>>>>>> server, > > > > >>>>>>>>> > > > > >>>>>>>>> engine, demos/examples, benchmarks > > > > >>>>>>>>>> > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be changed. > > For > > > > >>>>>>>>>> > > > > >>>>>>>>>> operators > > > > >>>>>>>>>> > > > > >>>>>>>>> we > > > > >>>>>>>>> > > > > >>>>>>>>> have a bit more freedom to do changes before a major > release > > as > > > > >>>>>>>>> most > > > > >>>>>>>>> > > > > >>>>>>>>>> of > > > > >>>>>>>>>> > > > > >>>>>>>>> them are marked @Evolving and users have the ability to > > > continue > > > > >>>>>>>> > > > > >>>>>>>> using > > > > >>>>>>>>> > > > > >>>>>>>>> prior version of Malhar with newer engine due to engine > > > backward > > > > >>>>>>>> > > > > >>>>>>>> compatibility guarantee. > > > > >>>>>>>>> > > > > >>>>>>>>>> Thanks, > > > > >>>>>>>>>> Thomas > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > > > > > > > > > > > >
