Most of the operators evolve, as is quite clearly indicated by @Evolving annotations. So while perhaps 0.x.x would be a more appropriate version number, I don't think you can change that.
Thomas On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.ro...@datatorrent.com> wrote: > If entire library is not stable, its version should be 0.x.x and there > should not be any semantic versioning enabled or implied. It evolves. If > some operators are stable as 3.8.x version suggests, the library should > follow semantic versioning and it is not OK to make a major change that > breaks semantic versioning across the entire library. It is not a finding > an excuse not to make a change. For me semantic versioning and backward > compatibility is more important than a proper package name. > > Thank you, > > Vlad > > > On 7/14/17 09:11, Thomas Weise wrote: > >> Semantic versioning makes sense when you have a stable baseline. As long >> as >> frequent fixes need to be made to address basic issues, it makes no sense >> to declare operators stable, which is why they are marked evolving. >> >> Instead of finding reasons to avoid changes that should have been made a >> long time ago, why not discuss what a "stable operator" is and which ones >> deserve to be in that category. Those are the ones that will get the >> backward compatibility stub. >> >> Thanks, >> Thomas >> >> >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v.ro...@datatorrent.com> >> wrote: >> >> My preference is to agree that the next Malhar release is 4.0.0 and make >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic >>> versioning checks on Malhar in case there is no version compatibility. >>> >>> Thank you, >>> >>> Vlad >>> >>> >>> On 7/14/17 08:11, Thomas Weise wrote: >>> >>> next release is 3.8.0 >>>> >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.ro...@datatorrent.com> >>>> wrote: >>>> >>>> next release - 3.9.0 or 4.0.0? >>>> >>>>> Thank you, >>>>> >>>>> Vlad >>>>> >>>>> On 7/13/17 22:25, Thomas Weise wrote: >>>>> >>>>> It is time to resurrect this thread and get going with the work. >>>>> >>>>>> For the next release, I will sign up to do the package move in Malhar: >>>>>> >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517 >>>>>> >>>>>> In general this will be straightforward; most classes in Malhar are >>>>>> marked >>>>>> evolving and it is trivial for users to change import statements. >>>>>> However, >>>>>> I would suggest to discuss if there are selected operators that are >>>>>> worth >>>>>> keeping a backward compatibility stub in the old location. >>>>>> >>>>>> Here is my plan: >>>>>> >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is >>>>>> all >>>>>> >>>>>> IDE automated work >>>>>> 2. Add backward compatibility classes, if, any in separate PR >>>>>> 3. Create PR for Megh library to reflect moved classes >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni < >>>>>> pra...@datatorrent.com >>>>>> wrote: >>>>>> >>>>>> Inline >>>>>> >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <t...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> --> >>>>>>> >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni < >>>>>>>> pra...@datatorrent.com >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> For malhar, for existing operators, I prefer we do this as part of >>>>>>>> the >>>>>>>> >>>>>>>> planned refactoring for breaking the monolith modules into baby >>>>>>>>> >>>>>>>>> packages >>>>>>>>> >>>>>>>> and would also prefer deprecating the existing operators in place. >>>>>>>> Refactor into smaller modules was discussed for malhar-contrib and >>>>>>>> given >>>>>>>> the overall state of that module I think it is OK to defer package >>>>>>>> >>>>>>>> renaming >>>>>>>> >>>>>>> there. I do however prefer to see the package rename addressed for >>>>>>> >>>>>>>> other >>>>>>>> modules, especially for the main library module. >>>>>>>> >>>>>>>> Should we consider breaking the library into smaller modules as >>>>>>>> well, >>>>>>>> >>>>>>>> the >>>>>>> file/block operators for example probably can be in their own module >>>>>>> from >>>>>>> just an organizational perspective. >>>>>>> >>>>>>> >>>>>>> >>>>>>> This >>>>>>> >>>>>>>> will help us achieve two things. First, the user will see all the >>>>>>>>> new >>>>>>>>> changes at once as opposed to dealing with it twice (with package >>>>>>>>> >>>>>>>>> rename >>>>>>>>> >>>>>>>> and dependency changes) and second it will allow for a smoother >>>>>>>> transition >>>>>>>> >>>>>>>> as the existing code will still work in a deprecated state. It will >>>>>>>> >>>>>>>>> also >>>>>>>>> >>>>>>>> give a more consistent structure to malhar. For new operators, we >>>>>>>> can >>>>>>>> go >>>>>>>> with the new package path but we need to ensure they will get moved >>>>>>>> into >>>>>>>> the baby packages as well. >>>>>>>> >>>>>>>> I think existing operators should be renamed so that git history >>>>>>>>> >>>>>>>>> remains. A >>>>>>>> >>>>>>> possible solution for backward compatibility could be to subsequently >>>>>>> >>>>>>>> add >>>>>>>> empty subclasses in the previous location (for existing concrete >>>>>>>> >>>>>>>> operators >>>>>>>> >>>>>>> that we know are actually in use) to simplify migration for users. >>>>>>> >>>>>>>> >>>>>>>> Yes we can do that. >>>>>>>> >>>>>>>> For demos, we can modify the paths as the apps are typically used >>>>>>> >>>>>>> wholesale >>>>>>>> >>>>>>>> and the interface is typically manual interaction. >>>>>>>> >>>>>>>>> For core, if we are adding new api subsystems, like the launcher >>>>>>>>> api >>>>>>>>> we >>>>>>>>> added recently for example, we can go with new package path but if >>>>>>>>> we >>>>>>>>> >>>>>>>>> are >>>>>>>>> >>>>>>>> making incremental additions to existing functionality, I feel it is >>>>>>>> better >>>>>>>> >>>>>>>> to keep it in the same package. I also prefer we keep the package of >>>>>>>> >>>>>>>>> the >>>>>>>>> >>>>>>>> implementation classes consistent with api, for understandability >>>>>>>> and >>>>>>>> >>>>>>>> readability of the code. So, for example, we don't change package >>>>>>>>> path >>>>>>>>> >>>>>>>>> of >>>>>>>>> >>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but >>>>>>>> it >>>>>>>> >>>>>>>> will be good if we can also do the same with classes closely related >>>>>>>>> to >>>>>>>>> >>>>>>>>> the >>>>>>>>> >>>>>>>> implementation classes as well. Maybe we can moving these on a >>>>>>>> package >>>>>>>> >>>>>>>>> by >>>>>>>>> >>>>>>>> package basis, like everything in com.datatorrent.stram.engine could >>>>>>>> be >>>>>>>> >>>>>>>> moved. For completely internal components like buffer server, we can >>>>>>>>> >>>>>>>>> move >>>>>>>>> >>>>>>>> them wholesale. We can consider moving all api and classes, when we >>>>>>>> go >>>>>>>> to >>>>>>>> next major release but would like to see if we can find a way to >>>>>>>> support >>>>>>>> existing api for one more major release in deprecated mode. >>>>>>>> >>>>>>>> The point of the major release is to enable backward incompatible >>>>>>>>> >>>>>>>>> changes >>>>>>>> and I don't think it is realistic to support the existing API for >>>>>>>> another >>>>>>>> major release. IMO it is also not necessary as most existing >>>>>>>> application >>>>>>>> code refers to operators, attributes and the application interface. >>>>>>>> >>>>>>>> Perhaps >>>>>>>> >>>>>>> it is possible to keep those around as interface extensions to help >>>>>>> >>>>>>>> migration. Custom operators may need to be migrated to reflect API >>>>>>>> >>>>>>>> changes, >>>>>>>> >>>>>>> and I would consider that a reasonable task for operator developers >>>>>>> as >>>>>>> >>>>>>>> part >>>>>>>> >>>>>>> of a major upgrade. >>>>>>> >>>>>>>> It would be good if we can keep them as deprecated interface >>>>>>>> extensions >>>>>>>> >>>>>>>> for >>>>>>> one release to provide a smoother transition. >>>>>>> >>>>>>> >>>>>>> API and implementation in engine are kept separate intentionally. >>>>>>> They >>>>>>> >>>>>>> reside in different packages today, so I don't see a problem renaming >>>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot be >>>>>>>> touched right away. >>>>>>>> >>>>>>>> They are different packages but sharing a common prefix with api >>>>>>>> will >>>>>>>> be >>>>>>>> >>>>>>>> helpful to someone new to codebase in terms of readability. Not a >>>>>>> big >>>>>>> deal >>>>>>> and can be changed. >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <t...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> This topic has come up on several PRs and I think it warrants a >>>>>>>>>> >>>>>>>>>> broader >>>>>>>>>> >>>>>>>>> discussion. >>>>>>>> >>>>>>>> At the time of incubation, the decision was to defer change of Java >>>>>>>>> >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next major >>>>>>>>>> >>>>>>>>>> release >>>>>>>>>> >>>>>>>>> to >>>>>>>>> >>>>>>>>> ensure backward compatibility for users. >>>>>>>>> >>>>>>>>>> Unfortunately that has lead to some confusion, as contributors >>>>>>>>>> >>>>>>>>>> continue >>>>>>>>>> >>>>>>>>> to >>>>>>>> >>>>>>>> add new code under legacy packages. >>>>>>>>> >>>>>>>>>> It is also a wider issue that examples for using Apex continue to >>>>>>>>>> >>>>>>>>>> refer >>>>>>>>>> >>>>>>>>> to >>>>>>>> >>>>>>>> com.datatorrent packages, nearly one year after graduation. More and >>>>>>>>> >>>>>>>>>> more >>>>>>>>>> >>>>>>>>> user code is being built on top of something that needs to change, >>>>>>>>> the >>>>>>>>> >>>>>>>>> can >>>>>>>> >>>>>>>> is being kicked down the road and users will face more changes >>>>>>>>> later. >>>>>>>>> >>>>>>>>>> I would like to propose the following: >>>>>>>>>> >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex packages >>>>>>>>>> >>>>>>>>>> 2. Not all code is under backward compatibility restriction and in >>>>>>>>>> >>>>>>>>>> those >>>>>>>>>> >>>>>>>>> cases we can rename the packages right away. Examples: buffer >>>>>>>>> server, >>>>>>>>> >>>>>>>>> engine, demos/examples, benchmarks >>>>>>>>>> >>>>>>>>>> 3. Discuss when the core API and operators can be changed. For >>>>>>>>>> >>>>>>>>>> operators >>>>>>>>>> >>>>>>>>> we >>>>>>>>> >>>>>>>>> have a bit more freedom to do changes before a major release as >>>>>>>>> most >>>>>>>>> >>>>>>>>>> of >>>>>>>>>> >>>>>>>>> them are marked @Evolving and users have the ability to continue >>>>>>>> >>>>>>>> using >>>>>>>>> >>>>>>>>> prior version of Malhar with newer engine due to engine backward >>>>>>>> >>>>>>>> compatibility guarantee. >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Thomas >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >
