GitHub user prateekm opened a pull request:
https://github.com/apache/samza/pull/552
SAMZA-1745: Remove all usages of StreamSpec and ApplicationRunner from the
operator spec and impl layers.
This PR is a pre-requisite for adding support for user-provided
SystemDescriptors and StreamDescriptors to the High Level API.
It removes all usages of StreamSpec and ApplicationRunner from the
OperatorSpec and OperatorImpl layers. DAG specification (StreamGraphSpec,
OperatorSpecs) now only relies on logical streamIds (and in future, will use
the user-provided StreamDescriptors). DAG execution (i.e., StreamOperatorTask,
OperatorImpls) now only relies on logical streamIds and their corresponding
SystemStreams, which are obtained using StreamConfig in OperatorImplGraph.
After this change, StreamSpec can be thought of as the API between
StreamManager and SystemAdmins for creating and validating streams. Ideally
ExecutionPlanner shouldn't rely on StreamSpec either, but it currently does so
extensively, so I'll leave that refactor for later.
Additional changes:
1. ApplicationRunner is no longer responsible for creating/returning
StreamSpec instances. Instances can be created directly using the StreamSpec
constructors, or by using one of the util methods in the new StreamUtil class.
2. StreamSpec class no longer tracks the isBroadcast and isBounded status
for streams.
The former was being used for communicating broadcast status from the
StreamGraphSpec to the planner so that it could write the broadcast input
configurations. This is now done using a separate Set of broadcast streamIds in
StreamGraphSpec.
The latter was being set by the ApplicationRunner based on a config, and
then passed to the planner so that it could write the bounded input configs.
This was redundant, so I removed it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prateekm/samza stream-spec-cleanup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/552.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #552
----
commit 8e6fc2dac5af11f0748fdcd32175c2ae95418fc7
Author: prateekm <prateekm@...>
Date: 2018-06-09T22:01:10Z
Remove all usages of StreamSpec and ApplicationRunner from the operator
spec and impl layers.
----
---
