On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <t...@apache.org> wrote:
-->
On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
pra...@datatorrent.com
wrote:
For malhar, for existing operators, I prefer we do this as part of the
planned refactoring for breaking the monolith modules into baby
packages
and would also prefer deprecating the existing operators in place.
Refactor into smaller modules was discussed for malhar-contrib and
given
the overall state of that module I think it is OK to defer package
renaming
there. I do however prefer to see the package rename addressed for
other
modules, especially for the main library module.
Should we consider breaking the library into smaller modules as well,
the
file/block operators for example probably can be in their own module
from
just an organizational perspective.
This
will help us achieve two things. First, the user will see all the new
changes at once as opposed to dealing with it twice (with package
rename
and dependency changes) and second it will allow for a smoother
transition
as the existing code will still work in a deprecated state. It will
also
give a more consistent structure to malhar. For new operators, we can
go
with the new package path but we need to ensure they will get moved
into
the baby packages as well.
I think existing operators should be renamed so that git history
remains. A
possible solution for backward compatibility could be to subsequently
add
empty subclasses in the previous location (for existing concrete
operators
that we know are actually in use) to simplify migration for users.
Yes we can do that.
For demos, we can modify the paths as the apps are typically used
wholesale
and the interface is typically manual interaction.
For core, if we are adding new api subsystems, like the launcher api
we
added recently for example, we can go with new package path but if we
are
making incremental additions to existing functionality, I feel it is
better
to keep it in the same package. I also prefer we keep the package of
the
implementation classes consistent with api, for understandability and
readability of the code. So, for example, we don't change package
path
of
LogicalPlan as it is an implementation of DAG. It is subjective, but
it
will be good if we can also do the same with classes closely related
to
the
implementation classes as well. Maybe we can moving these on a package
by
package basis, like everything in com.datatorrent.stram.engine could
be
moved. For completely internal components like buffer server, we can
move
them wholesale. We can consider moving all api and classes, when we go
to
next major release but would like to see if we can find a way to
support
existing api for one more major release in deprecated mode.
The point of the major release is to enable backward incompatible
changes
and I don't think it is realistic to support the existing API for
another
major release. IMO it is also not necessary as most existing
application
code refers to operators, attributes and the application interface.
Perhaps
it is possible to keep those around as interface extensions to help
migration. Custom operators may need to be migrated to reflect API
changes,
and I would consider that a reasonable task for operator developers as
part
of a major upgrade.
It would be good if we can keep them as deprecated interface
extensions
for
one release to provide a smoother transition.
API and implementation in engine are kept separate intentionally. They
reside in different packages today, so I don't see a problem renaming
com.datatorrent.stram.engine as you say, even when the API cannot be
touched right away.
They are different packages but sharing a common prefix with api will
be
helpful to someone new to codebase in terms of readability. Not a big
deal
and can be changed.
Thanks
On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <t...@apache.org> wrote:
Hi,
This topic has come up on several PRs and I think it warrants a
broader
discussion.
At the time of incubation, the decision was to defer change of Java
packages from com.datatorrent to org.apache.apex till next major
release
to
ensure backward compatibility for users.
Unfortunately that has lead to some confusion, as contributors
continue
to
add new code under legacy packages.
It is also a wider issue that examples for using Apex continue to
refer
to
com.datatorrent packages, nearly one year after graduation. More and
more
user code is being built on top of something that needs to change,
the
can
is being kicked down the road and users will face more changes later.
I would like to propose the following:
1. All new code has to be submitted under org.apache.apex packages
2. Not all code is under backward compatibility restriction and in
those
cases we can rename the packages right away. Examples: buffer server,
engine, demos/examples, benchmarks
3. Discuss when the core API and operators can be changed. For
operators
we
have a bit more freedom to do changes before a major release as most
of
them are marked @Evolving and users have the ability to continue
using
prior version of Malhar with newer engine due to engine backward
compatibility guarantee.
Thanks,
Thomas
