Why is there a urgency, why cant this go into 4.0 Malhar with possibly other breaking changes? On Sat, Jul 15, 2017 at 7:57 AM Thomas Weise <t...@apache.org> wrote:
> Discussing what in the future might become stable needs to be a separate > thread, it will be a much bigger discussion. > > The topic here is to relocate the packages. With a few exceptions > relocation won't affect the semantic versioning. Semantic versioning is > essentially not effective for Malhar because almost everything is @Evolving > (and there are reasons for that.. -> separate topic) > > I don't really like the idea of creating bw compatibility stubs for the > follow up PR. It creates even more clutter in the source tree than there > already is and so here is an alternative suggestion: > > > https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml > > Create a shaded artifact that provides the old com.datatorrent.* classes as > of release 3.7. Users can include that artifact if they don't want to > change import statements. At the same time they have an incentive to switch > to the relocated classes to take advantage of bug fixes and new > functionality. > > I will work on the first PR that does the relocate. In the meantime, we can > finalize what backward compatibility support we want to provide and how. > > Thanks, > Thomas > > > > > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pra...@datatorrent.com> > wrote: > > > How about coming up with a list of @Evolving operators that we would like > > to see in the eventual stable list and move those along with the not > > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as > they > > are. Then have a follow up JIRA for the rest to be moved over to contrib > > and be deprecated. > > > > Thanks > > > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <thomas.we...@gmail.com> > > wrote: > > > > > We need to keep the discussion here on topic, if other things are piled > > on > > > top then nothing gets done. > > > > > > Most operators today are not stable, they are @Evolving. So perhaps it > is > > > necessary to have a separate discussion about that, outside of this > > thread. > > > My guess is that there are only a few operators that we could declare > > > stable. Specifically, under contrib the closest one might have been > > Kafka, > > > but that is already superseded by the newer versions. > > > > > > Thomas > > > > > > > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni < > > pra...@datatorrent.com> > > > wrote: > > > > > > > We had a discussion a while back, agreed to relegate non-stable and > > > > experimental operators to contrib and also added this to contribution > > > > guidelines. We aexecuted on this and cleaned up the repo by moving > > > > operators deemed non-suitable for production use at that time to > > contrib. > > > > So I wouldn't say the operators that are at the top level today or > the > > > ones > > > > in library are 0.x.x quality. Granted, we may need to do one more > pass > > as > > > > some of the operators may no longer be considered the right > > > implementations > > > > with the advent of the windowed operator. > > > > > > > > Since contrib used to be the place that housed operators that > required > > > > third party libraries, there are some production quality operators in > > > there > > > > that need to be pulled out into top level modules. > > > > > > > > I think we are in agreement that for operators that we consider > stable, > > > we > > > > should provide a b/w stub. I would add that we strongly consider > > creating > > > > the org.apache.apex counterparts of any stable operators that are in > > > > contrib out in top level modules and have the com.datatorrent stubs > in > > > > contrib. > > > > > > > > For the operators not considered stable, I would prefer we either > leave > > > the > > > > current package path as is or if we change the package path then > create > > > the > > > > b/w stub as I am not sure how widely they are in use (so, in essence, > > > > preserve semantic versioning). It would be good if there is a > followup > > > JIRA > > > > that takes a look at what other operators can be moved to contrib > with > > > the > > > > advent of the newer frameworks and understanding. > > > > > > > > Thanks > > > > > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <t...@apache.org> > wrote: > > > > > > > > > Most of the operators evolve, as is quite clearly indicated by > > > @Evolving > > > > > annotations. So while perhaps 0.x.x would be a more appropriate > > version > > > > > number, I don't think you can change that. > > > > > > > > > > Thomas > > > > > > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov < > v.ro...@datatorrent.com > > > > > > > > wrote: > > > > > > > > > > > If entire library is not stable, its version should be 0.x.x and > > > there > > > > > > should not be any semantic versioning enabled or implied. It > > evolves. > > > > If > > > > > > some operators are stable as 3.8.x version suggests, the library > > > should > > > > > > follow semantic versioning and it is not OK to make a major > change > > > that > > > > > > breaks semantic versioning across the entire library. It is not a > > > > finding > > > > > > an excuse not to make a change. For me semantic versioning and > > > backward > > > > > > compatibility is more important than a proper package name. > > > > > > > > > > > > Thank you, > > > > > > > > > > > > Vlad > > > > > > > > > > > > > > > > > > On 7/14/17 09:11, Thomas Weise wrote: > > > > > > > > > > > >> Semantic versioning makes sense when you have a stable baseline. > > As > > > > long > > > > > >> as > > > > > >> frequent fixes need to be made to address basic issues, it makes > > no > > > > > sense > > > > > >> to declare operators stable, which is why they are marked > > evolving. > > > > > >> > > > > > >> Instead of finding reasons to avoid changes that should have > been > > > > made a > > > > > >> long time ago, why not discuss what a "stable operator" is and > > which > > > > > ones > > > > > >> deserve to be in that category. Those are the ones that will get > > the > > > > > >> backward compatibility stub. > > > > > >> > > > > > >> Thanks, > > > > > >> Thomas > > > > > >> > > > > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov < > > > v.ro...@datatorrent.com> > > > > > >> wrote: > > > > > >> > > > > > >> My preference is to agree that the next Malhar release is 4.0.0 > > and > > > > make > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep > > > > > semantic > > > > > >>> versioning checks on Malhar in case there is no version > > > > compatibility. > > > > > >>> > > > > > >>> Thank you, > > > > > >>> > > > > > >>> Vlad > > > > > >>> > > > > > >>> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote: > > > > > >>> > > > > > >>> next release is 3.8.0 > > > > > >>>> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov < > > > > v.ro...@datatorrent.com> > > > > > >>>> wrote: > > > > > >>>> > > > > > >>>> next release - 3.9.0 or 4.0.0? > > > > > >>>> > > > > > >>>>> Thank you, > > > > > >>>>> > > > > > >>>>> Vlad > > > > > >>>>> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote: > > > > > >>>>> > > > > > >>>>> It is time to resurrect this thread and get going with the > > work. > > > > > >>>>> > > > > > >>>>>> For the next release, I will sign up to do the package move > in > > > > > Malhar: > > > > > >>>>>> > > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517 > > > > > >>>>>> > > > > > >>>>>> In general this will be straightforward; most classes in > > Malhar > > > > are > > > > > >>>>>> marked > > > > > >>>>>> evolving and it is trivial for users to change import > > > statements. > > > > > >>>>>> However, > > > > > >>>>>> I would suggest to discuss if there are selected operators > > that > > > > are > > > > > >>>>>> worth > > > > > >>>>>> keeping a backward compatibility stub in the old location. > > > > > >>>>>> > > > > > >>>>>> Here is my plan: > > > > > >>>>>> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one > PR - > > > > this > > > > > is > > > > > >>>>>> all > > > > > >>>>>> > > > > > >>>>>> IDE automated work > > > > > >>>>>> 2. Add backward compatibility classes, if, any in separate > PR > > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes > > > > > >>>>>> > > > > > >>>>>> Thanks, > > > > > >>>>>> Thomas > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni < > > > > > >>>>>> pra...@datatorrent.com > > > > > >>>>>> wrote: > > > > > >>>>>> > > > > > >>>>>> Inline > > > > > >>>>>> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise < > > t...@apache.org> > > > > > >>>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>> --> > > > > > >>>>>>> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni < > > > > > >>>>>>>> pra...@datatorrent.com > > > > > >>>>>>>> > > > > > >>>>>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we do this as > > > part > > > > of > > > > > >>>>>>>> the > > > > > >>>>>>>> > > > > > >>>>>>>> planned refactoring for breaking the monolith modules into > > > baby > > > > > >>>>>>>>> > > > > > >>>>>>>>> packages > > > > > >>>>>>>>> > > > > > >>>>>>>> and would also prefer deprecating the existing operators > in > > > > place. > > > > > >>>>>>>> Refactor into smaller modules was discussed for > > malhar-contrib > > > > and > > > > > >>>>>>>> given > > > > > >>>>>>>> the overall state of that module I think it is OK to defer > > > > package > > > > > >>>>>>>> > > > > > >>>>>>>> renaming > > > > > >>>>>>>> > > > > > >>>>>>> there. I do however prefer to see the package rename > > addressed > > > > for > > > > > >>>>>>> > > > > > >>>>>>>> other > > > > > >>>>>>>> modules, especially for the main library module. > > > > > >>>>>>>> > > > > > >>>>>>>> Should we consider breaking the library into smaller > modules > > > as > > > > > >>>>>>>> well, > > > > > >>>>>>>> > > > > > >>>>>>>> the > > > > > >>>>>>> file/block operators for example probably can be in their > own > > > > > module > > > > > >>>>>>> from > > > > > >>>>>>> just an organizational perspective. > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> This > > > > > >>>>>>> > > > > > >>>>>>>> will help us achieve two things. First, the user will see > > all > > > > the > > > > > >>>>>>>>> new > > > > > >>>>>>>>> changes at once as opposed to dealing with it twice (with > > > > package > > > > > >>>>>>>>> > > > > > >>>>>>>>> rename > > > > > >>>>>>>>> > > > > > >>>>>>>> and dependency changes) and second it will allow for a > > > smoother > > > > > >>>>>>>> transition > > > > > >>>>>>>> > > > > > >>>>>>>> as the existing code will still work in a deprecated > state. > > It > > > > > will > > > > > >>>>>>>> > > > > > >>>>>>>>> also > > > > > >>>>>>>>> > > > > > >>>>>>>> give a more consistent structure to malhar. For new > > operators, > > > > we > > > > > >>>>>>>> can > > > > > >>>>>>>> go > > > > > >>>>>>>> with the new package path but we need to ensure they will > > get > > > > > moved > > > > > >>>>>>>> into > > > > > >>>>>>>> the baby packages as well. > > > > > >>>>>>>> > > > > > >>>>>>>> I think existing operators should be renamed so that git > > > history > > > > > >>>>>>>>> > > > > > >>>>>>>>> remains. A > > > > > >>>>>>>> > > > > > >>>>>>> possible solution for backward compatibility could be to > > > > > subsequently > > > > > >>>>>>> > > > > > >>>>>>>> add > > > > > >>>>>>>> empty subclasses in the previous location (for existing > > > concrete > > > > > >>>>>>>> > > > > > >>>>>>>> operators > > > > > >>>>>>>> > > > > > >>>>>>> that we know are actually in use) to simplify migration for > > > > users. > > > > > >>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> Yes we can do that. > > > > > >>>>>>>> > > > > > >>>>>>>> For demos, we can modify the paths as the apps are > typically > > > > used > > > > > >>>>>>> > > > > > >>>>>>> wholesale > > > > > >>>>>>>> > > > > > >>>>>>>> and the interface is typically manual interaction. > > > > > >>>>>>>> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, like the > > > > launcher > > > > > >>>>>>>>> api > > > > > >>>>>>>>> we > > > > > >>>>>>>>> added recently for example, we can go with new package > path > > > but > > > > > if > > > > > >>>>>>>>> we > > > > > >>>>>>>>> > > > > > >>>>>>>>> are > > > > > >>>>>>>>> > > > > > >>>>>>>> making incremental additions to existing functionality, I > > feel > > > > it > > > > > is > > > > > >>>>>>>> better > > > > > >>>>>>>> > > > > > >>>>>>>> to keep it in the same package. I also prefer we keep the > > > > package > > > > > of > > > > > >>>>>>>> > > > > > >>>>>>>>> the > > > > > >>>>>>>>> > > > > > >>>>>>>> implementation classes consistent with api, for > > > > understandability > > > > > >>>>>>>> and > > > > > >>>>>>>> > > > > > >>>>>>>> readability of the code. So, for example, we don't change > > > > package > > > > > >>>>>>>>> path > > > > > >>>>>>>>> > > > > > >>>>>>>>> of > > > > > >>>>>>>>> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is > > > subjective, > > > > > but > > > > > >>>>>>>> it > > > > > >>>>>>>> > > > > > >>>>>>>> will be good if we can also do the same with classes > closely > > > > > related > > > > > >>>>>>>>> to > > > > > >>>>>>>>> > > > > > >>>>>>>>> the > > > > > >>>>>>>>> > > > > > >>>>>>>> implementation classes as well. Maybe we can moving these > > on a > > > > > >>>>>>>> package > > > > > >>>>>>>> > > > > > >>>>>>>>> by > > > > > >>>>>>>>> > > > > > >>>>>>>> package basis, like everything in > > com.datatorrent.stram.engine > > > > > could > > > > > >>>>>>>> be > > > > > >>>>>>>> > > > > > >>>>>>>> moved. For completely internal components like buffer > > server, > > > we > > > > > can > > > > > >>>>>>>>> > > > > > >>>>>>>>> move > > > > > >>>>>>>>> > > > > > >>>>>>>> them wholesale. We can consider moving all api and > classes, > > > when > > > > > we > > > > > >>>>>>>> go > > > > > >>>>>>>> to > > > > > >>>>>>>> next major release but would like to see if we can find a > > way > > > to > > > > > >>>>>>>> support > > > > > >>>>>>>> existing api for one more major release in deprecated > mode. > > > > > >>>>>>>> > > > > > >>>>>>>> The point of the major release is to enable backward > > > > incompatible > > > > > >>>>>>>>> > > > > > >>>>>>>>> changes > > > > > >>>>>>>> and I don't think it is realistic to support the existing > > API > > > > for > > > > > >>>>>>>> another > > > > > >>>>>>>> major release. IMO it is also not necessary as most > existing > > > > > >>>>>>>> application > > > > > >>>>>>>> code refers to operators, attributes and the application > > > > > interface. > > > > > >>>>>>>> > > > > > >>>>>>>> Perhaps > > > > > >>>>>>>> > > > > > >>>>>>> it is possible to keep those around as interface extensions > > to > > > > help > > > > > >>>>>>> > > > > > >>>>>>>> migration. Custom operators may need to be migrated to > > reflect > > > > API > > > > > >>>>>>>> > > > > > >>>>>>>> changes, > > > > > >>>>>>>> > > > > > >>>>>>> and I would consider that a reasonable task for operator > > > > developers > > > > > >>>>>>> as > > > > > >>>>>>> > > > > > >>>>>>>> part > > > > > >>>>>>>> > > > > > >>>>>>> of a major upgrade. > > > > > >>>>>>> > > > > > >>>>>>>> It would be good if we can keep them as deprecated > interface > > > > > >>>>>>>> extensions > > > > > >>>>>>>> > > > > > >>>>>>>> for > > > > > >>>>>>> one release to provide a smoother transition. > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> API and implementation in engine are kept separate > > > intentionally. > > > > > >>>>>>> They > > > > > >>>>>>> > > > > > >>>>>>> reside in different packages today, so I don't see a > problem > > > > > renaming > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API > > > > cannot > > > > > be > > > > > >>>>>>>> touched right away. > > > > > >>>>>>>> > > > > > >>>>>>>> They are different packages but sharing a common prefix > with > > > api > > > > > >>>>>>>> will > > > > > >>>>>>>> be > > > > > >>>>>>>> > > > > > >>>>>>>> helpful to someone new to codebase in terms of > readability. > > > Not > > > > a > > > > > >>>>>>> big > > > > > >>>>>>> deal > > > > > >>>>>>> and can be changed. > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> Thanks > > > > > >>>>>>> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise < > > t...@apache.org> > > > > > >>>>>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> Hi, > > > > > >>>>>>>>> > > > > > >>>>>>>>> This topic has come up on several PRs and I think it > > > warrants a > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> broader > > > > > >>>>>>>>>> > > > > > >>>>>>>>> discussion. > > > > > >>>>>>>> > > > > > >>>>>>>> At the time of incubation, the decision was to defer > change > > of > > > > > Java > > > > > >>>>>>>>> > > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till > next > > > > major > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> release > > > > > >>>>>>>>>> > > > > > >>>>>>>>> to > > > > > >>>>>>>>> > > > > > >>>>>>>>> ensure backward compatibility for users. > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as > > > contributors > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> continue > > > > > >>>>>>>>>> > > > > > >>>>>>>>> to > > > > > >>>>>>>> > > > > > >>>>>>>> add new code under legacy packages. > > > > > >>>>>>>>> > > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex > > > continue > > > > > to > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> refer > > > > > >>>>>>>>>> > > > > > >>>>>>>>> to > > > > > >>>>>>>> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after > graduation. > > > More > > > > > and > > > > > >>>>>>>>> > > > > > >>>>>>>>>> more > > > > > >>>>>>>>>> > > > > > >>>>>>>>> user code is being built on top of something that needs > to > > > > > change, > > > > > >>>>>>>>> the > > > > > >>>>>>>>> > > > > > >>>>>>>>> can > > > > > >>>>>>>> > > > > > >>>>>>>> is being kicked down the road and users will face more > > changes > > > > > >>>>>>>>> later. > > > > > >>>>>>>>> > > > > > >>>>>>>>>> I would like to propose the following: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> 1. All new code has to be submitted under > org.apache.apex > > > > > packages > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility > > restriction > > > > and > > > > > in > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> those > > > > > >>>>>>>>>> > > > > > >>>>>>>>> cases we can rename the packages right away. Examples: > > buffer > > > > > >>>>>>>>> server, > > > > > >>>>>>>>> > > > > > >>>>>>>>> engine, demos/examples, benchmarks > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be > changed. > > > For > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> operators > > > > > >>>>>>>>>> > > > > > >>>>>>>>> we > > > > > >>>>>>>>> > > > > > >>>>>>>>> have a bit more freedom to do changes before a major > > release > > > as > > > > > >>>>>>>>> most > > > > > >>>>>>>>> > > > > > >>>>>>>>>> of > > > > > >>>>>>>>>> > > > > > >>>>>>>>> them are marked @Evolving and users have the ability to > > > > continue > > > > > >>>>>>>> > > > > > >>>>>>>> using > > > > > >>>>>>>>> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to engine > > > > backward > > > > > >>>>>>>> > > > > > >>>>>>>> compatibility guarantee. > > > > > >>>>>>>>> > > > > > >>>>>>>>>> Thanks, > > > > > >>>>>>>>>> Thomas > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > > > > >
