Inline On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <t...@apache.org> wrote:
> --> > > On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <pra...@datatorrent.com> > wrote: > > > For malhar, for existing operators, I prefer we do this as part of the > > planned refactoring for breaking the monolith modules into baby packages > > and would also prefer deprecating the existing operators in place. > > > Refactor into smaller modules was discussed for malhar-contrib and given > the overall state of that module I think it is OK to defer package renaming > there. I do however prefer to see the package rename addressed for other > modules, especially for the main library module. > Should we consider breaking the library into smaller modules as well, the file/block operators for example probably can be in their own module from just an organizational perspective. > > > > This > > will help us achieve two things. First, the user will see all the new > > changes at once as opposed to dealing with it twice (with package rename > > and dependency changes) and second it will allow for a smoother > transition > > as the existing code will still work in a deprecated state. It will also > > give a more consistent structure to malhar. For new operators, we can go > > with the new package path but we need to ensure they will get moved into > > the baby packages as well. > > > > I think existing operators should be renamed so that git history remains. A > possible solution for backward compatibility could be to subsequently add > empty subclasses in the previous location (for existing concrete operators > that we know are actually in use) to simplify migration for users. > > Yes we can do that. > > > For demos, we can modify the paths as the apps are typically used > wholesale > > and the interface is typically manual interaction. > > > > For core, if we are adding new api subsystems, like the launcher api we > > added recently for example, we can go with new package path but if we are > > making incremental additions to existing functionality, I feel it is > better > > to keep it in the same package. I also prefer we keep the package of the > > implementation classes consistent with api, for understandability and > > readability of the code. So, for example, we don't change package path of > > LogicalPlan as it is an implementation of DAG. It is subjective, but it > > will be good if we can also do the same with classes closely related to > the > > implementation classes as well. Maybe we can moving these on a package by > > package basis, like everything in com.datatorrent.stram.engine could be > > moved. For completely internal components like buffer server, we can move > > them wholesale. We can consider moving all api and classes, when we go to > > next major release but would like to see if we can find a way to support > > existing api for one more major release in deprecated mode. > > > > > The point of the major release is to enable backward incompatible changes > and I don't think it is realistic to support the existing API for another > major release. IMO it is also not necessary as most existing application > code refers to operators, attributes and the application interface. Perhaps > it is possible to keep those around as interface extensions to help > migration. Custom operators may need to be migrated to reflect API changes, > and I would consider that a reasonable task for operator developers as part > of a major upgrade. > It would be good if we can keep them as deprecated interface extensions for one release to provide a smoother transition. > > API and implementation in engine are kept separate intentionally. They > reside in different packages today, so I don't see a problem renaming > com.datatorrent.stram.engine as you say, even when the API cannot be > touched right away. > They are different packages but sharing a common prefix with api will be helpful to someone new to codebase in terms of readability. Not a big deal and can be changed. > > Thanks > > > > On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <t...@apache.org> wrote: > > > > > Hi, > > > > > > This topic has come up on several PRs and I think it warrants a broader > > > discussion. > > > > > > At the time of incubation, the decision was to defer change of Java > > > packages from com.datatorrent to org.apache.apex till next major > release > > to > > > ensure backward compatibility for users. > > > > > > Unfortunately that has lead to some confusion, as contributors continue > > to > > > add new code under legacy packages. > > > > > > It is also a wider issue that examples for using Apex continue to refer > > to > > > com.datatorrent packages, nearly one year after graduation. More and > more > > > user code is being built on top of something that needs to change, the > > can > > > is being kicked down the road and users will face more changes later. > > > > > > I would like to propose the following: > > > > > > 1. All new code has to be submitted under org.apache.apex packages > > > > > > 2. Not all code is under backward compatibility restriction and in > those > > > cases we can rename the packages right away. Examples: buffer server, > > > engine, demos/examples, benchmarks > > > > > > 3. Discuss when the core API and operators can be changed. For > operators > > we > > > have a bit more freedom to do changes before a major release as most of > > > them are marked @Evolving and users have the ability to continue using > > > prior version of Malhar with newer engine due to engine backward > > > compatibility guarantee. > > > > > > Thanks, > > > Thomas > > > > > >
