next release is 3.8.0 On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.ro...@datatorrent.com> wrote:
> next release - 3.9.0 or 4.0.0? > > Thank you, > > Vlad > > On 7/13/17 22:25, Thomas Weise wrote: > >> It is time to resurrect this thread and get going with the work. >> >> For the next release, I will sign up to do the package move in Malhar: >> >> https://issues.apache.org/jira/browse/APEXMALHAR-2517 >> >> In general this will be straightforward; most classes in Malhar are marked >> evolving and it is trivial for users to change import statements. However, >> I would suggest to discuss if there are selected operators that are worth >> keeping a backward compatibility stub in the old location. >> >> Here is my plan: >> >> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all >> >> IDE automated work >> 2. Add backward compatibility classes, if, any in separate PR >> 3. Create PR for Megh library to reflect moved classes >> >> Thanks, >> Thomas >> >> >> >> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pra...@datatorrent.com> >> wrote: >> >> Inline >>> >>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <t...@apache.org> wrote: >>> >>> --> >>>> >>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni < >>>> pra...@datatorrent.com >>>> >>>> wrote: >>>> >>>> For malhar, for existing operators, I prefer we do this as part of the >>>>> planned refactoring for breaking the monolith modules into baby >>>>> >>>> packages >>> >>>> and would also prefer deprecating the existing operators in place. >>>>> >>>> >>>> Refactor into smaller modules was discussed for malhar-contrib and given >>>> the overall state of that module I think it is OK to defer package >>>> >>> renaming >>> >>>> there. I do however prefer to see the package rename addressed for other >>>> modules, especially for the main library module. >>>> >>>> Should we consider breaking the library into smaller modules as well, >>> the >>> file/block operators for example probably can be in their own module from >>> just an organizational perspective. >>> >>> >>> >>>> This >>>>> will help us achieve two things. First, the user will see all the new >>>>> changes at once as opposed to dealing with it twice (with package >>>>> >>>> rename >>> >>>> and dependency changes) and second it will allow for a smoother >>>>> >>>> transition >>>> >>>>> as the existing code will still work in a deprecated state. It will >>>>> >>>> also >>> >>>> give a more consistent structure to malhar. For new operators, we can >>>>> >>>> go >>> >>>> with the new package path but we need to ensure they will get moved >>>>> >>>> into >>> >>>> the baby packages as well. >>>>> >>>>> I think existing operators should be renamed so that git history >>>> >>> remains. A >>> >>>> possible solution for backward compatibility could be to subsequently >>>> add >>>> empty subclasses in the previous location (for existing concrete >>>> >>> operators >>> >>>> that we know are actually in use) to simplify migration for users. >>>> >>>> >>>> Yes we can do that. >>> >>> >>> For demos, we can modify the paths as the apps are typically used >>>>> >>>> wholesale >>>> >>>>> and the interface is typically manual interaction. >>>>> >>>>> For core, if we are adding new api subsystems, like the launcher api we >>>>> added recently for example, we can go with new package path but if we >>>>> >>>> are >>> >>>> making incremental additions to existing functionality, I feel it is >>>>> >>>> better >>>> >>>>> to keep it in the same package. I also prefer we keep the package of >>>>> >>>> the >>> >>>> implementation classes consistent with api, for understandability and >>>>> readability of the code. So, for example, we don't change package path >>>>> >>>> of >>> >>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it >>>>> will be good if we can also do the same with classes closely related to >>>>> >>>> the >>>> >>>>> implementation classes as well. Maybe we can moving these on a package >>>>> >>>> by >>> >>>> package basis, like everything in com.datatorrent.stram.engine could be >>>>> moved. For completely internal components like buffer server, we can >>>>> >>>> move >>> >>>> them wholesale. We can consider moving all api and classes, when we go >>>>> >>>> to >>> >>>> next major release but would like to see if we can find a way to >>>>> >>>> support >>> >>>> existing api for one more major release in deprecated mode. >>>>> >>>>> >>>>> The point of the major release is to enable backward incompatible >>>> changes >>>> and I don't think it is realistic to support the existing API for >>>> another >>>> major release. IMO it is also not necessary as most existing application >>>> code refers to operators, attributes and the application interface. >>>> >>> Perhaps >>> >>>> it is possible to keep those around as interface extensions to help >>>> migration. Custom operators may need to be migrated to reflect API >>>> >>> changes, >>> >>>> and I would consider that a reasonable task for operator developers as >>>> >>> part >>> >>>> of a major upgrade. >>>> >>>> It would be good if we can keep them as deprecated interface extensions >>> for >>> one release to provide a smoother transition. >>> >>> >>> API and implementation in engine are kept separate intentionally. They >>>> reside in different packages today, so I don't see a problem renaming >>>> com.datatorrent.stram.engine as you say, even when the API cannot be >>>> touched right away. >>>> >>>> They are different packages but sharing a common prefix with api will be >>> helpful to someone new to codebase in terms of readability. Not a big >>> deal >>> and can be changed. >>> >>> >>> Thanks >>>> >>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <t...@apache.org> wrote: >>>>> >>>>> Hi, >>>>>> >>>>>> This topic has come up on several PRs and I think it warrants a >>>>>> >>>>> broader >>> >>>> discussion. >>>>>> >>>>>> At the time of incubation, the decision was to defer change of Java >>>>>> packages from com.datatorrent to org.apache.apex till next major >>>>>> >>>>> release >>>> >>>>> to >>>>> >>>>>> ensure backward compatibility for users. >>>>>> >>>>>> Unfortunately that has lead to some confusion, as contributors >>>>>> >>>>> continue >>> >>>> to >>>>> >>>>>> add new code under legacy packages. >>>>>> >>>>>> It is also a wider issue that examples for using Apex continue to >>>>>> >>>>> refer >>> >>>> to >>>>> >>>>>> com.datatorrent packages, nearly one year after graduation. More and >>>>>> >>>>> more >>>> >>>>> user code is being built on top of something that needs to change, >>>>>> >>>>> the >>> >>>> can >>>>> >>>>>> is being kicked down the road and users will face more changes later. >>>>>> >>>>>> I would like to propose the following: >>>>>> >>>>>> 1. All new code has to be submitted under org.apache.apex packages >>>>>> >>>>>> 2. Not all code is under backward compatibility restriction and in >>>>>> >>>>> those >>>> >>>>> cases we can rename the packages right away. Examples: buffer server, >>>>>> engine, demos/examples, benchmarks >>>>>> >>>>>> 3. Discuss when the core API and operators can be changed. For >>>>>> >>>>> operators >>>> >>>>> we >>>>> >>>>>> have a bit more freedom to do changes before a major release as most >>>>>> >>>>> of >>> >>>> them are marked @Evolving and users have the ability to continue >>>>>> >>>>> using >>> >>>> prior version of Malhar with newer engine due to engine backward >>>>>> compatibility guarantee. >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> >
