GitHub user kennknowles opened a pull request: https://github.com/apache/beam/pull/3924
[BEAM-2576] Split Beam portability framework into construction, job, and execution modules Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- The Beam portability framework has a lot of simple pieces that can be modularized more usefully than "Runner API" and "Fn API". Here are some semantic categories that should remain stable, and make sense to depend on separately: - Modeling of a Pipeline and pieces of it - Standardized transform payloads - primitives - non-primitives - execution-time only transforms - Standardized UDFs - Job submission and management (Job API) - Artifact upload and retrieval (Artifact API) (could separate upload and retrieval) - Model of bundle processing instructions (ProcessBundleDescriptor) - Standardized execution-only instruction payloads (like data plane read instructions) - Setting up the SDK harness (Provision API) - Control plane for bundle processing (FnControl) - Multi-purpose data streaming (FnData) - Logging - ... more later, maybe ... This PR is a first move in this direction, organizing the components according to when in a pipeline's lifecycle they are used. - **Construction**: building a pipeline and specifying transforms. Basically just data structures, not APIs. Code doing this work should only depend on this module. - **Job**: submission and management of a pipeline, including artifacts, etc. Construction code should not depend on this module. - **Execution**: SDK harness management and UDF invocation. Only runner internals and SDK harnesses should depend on this module. What this PR does is move the existing protos into the right maven modules, but does not split them yet. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/beam beam-model Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3924 ---- commit 4ff34d419d8551c874294d9cb2a20ae69ae80530 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T19:09:35Z Add model/construction to Maven structure and gen_protos.py commit da9184d7de59c43c08f29cd2072eac917b012911 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T17:46:22Z Move beam_runner_api.proto to model/construction commit d3368c0f761d41baa0d607991b6c3471853c44f9 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T17:47:49Z Add model/job to Maven structure and gen_protos.py commit 856be477e07ea1c1acbebc43ca3ba2b04bf77ce7 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T18:02:14Z Move standard_window_fns.proto to model/construction commit 118faaba61ed2724f27f6b77770725fcb2f062c9 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T18:04:22Z Move beam_artifact_api.proto to model/job commit a72ecb869d969e8f1dc7fef7b0ec254a5bb13b15 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T18:05:35Z Move beam_job_api.proto to model/job commit 1989720e4b4b316697b33600abd4e2bdb7c46d2b Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T18:07:30Z Remove empty runner-api module from Maven structure and gen_protos.py commit 208a6f48bfc5d8b4196fb74f7d5a54abc665565f Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T17:48:11Z Add model/execution to Maven structure and gen_protos.py commit d7c24d16db6c14be2211eed4e296e504fe396b92 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T18:25:40Z Move beam_provision_api.proto to model/execution commit 986b1a0ef02a25c3d22d217f22139b72ebde2f48 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T19:04:42Z Move beam_fn_api.proto to model/execution commit a5114b0d9f1daeec673f23dfe459c8bc98b652c9 Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T19:06:43Z Move standard_coders.yaml to model/execution commit 9ac1d0a5fb521db4de99dec1d2746e82996c3e2c Author: Kenneth Knowles <k...@google.com> Date: 2017-09-29T19:07:21Z Remove empty fn-api and sdks/common from Maven structure and gen_protos.py ---- ---