GitHub user kennknowles opened a pull request: https://github.com/apache/beam/pull/3750
[BEAM-2430] DoFnRunner for processing a bundle over the Fn API Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- R: @aljoscha this is not super polished, and much of the complexity I thought I would encapsulate really fits better outside the DoFnRunner. But I thought reviewing this and getting it in might be a good way to proceed and share understanding of the Fn API. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/beam FnApiDoFnRunner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3750.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3750 ---- commit eb18585969f67089de62b20dc5c3b2722d7e566a Author: Kenneth Knowles <k...@google.com> Date: 2017-08-23T13:20:33Z Basic Java wrapper for Fn API data plane commit f71101685265c53ecee36d3aadec037cc792f0b4 Author: Kenneth Knowles <k...@google.com> Date: 2017-07-26T14:42:03Z FnApiControlClient, de-inverts Fn API control plane The Fn API control plane has an inverted client/server relationship in order to support firewall rules where the runner is forbidden from connecting out to the SDK harness. This Java helper provides an API with the more conventional polarity. It also associates streamed gRPC requests with responses to support simple future-based programming. commit 858f219ce3a6df64b8344472022e01141ecf130b Author: Kenneth Knowles <k...@google.com> Date: 2017-07-26T14:44:41Z Service for pooling incoming Fn API control plane connections commit c3c882df800926b46c07bdeb867b5639bcde00f1 Author: Kenneth Knowles <k...@google.com> Date: 2017-07-26T14:45:37Z Higher-level SdkHarnessClient This adds a fairly thin wrapper on FnApiSdkHarnessClient, encapsulating the fact that all request and response types are injected into a disjoint union and sent over the same low-level RPC. commit a947ccb752f4200cbc8a98f85129115e45577973 Author: Kenneth Knowles <k...@google.com> Date: 2017-07-26T14:48:48Z Add SdkHarnessDoFnRunner This encapsulates processing a bundle over the Fn API. A Beam runner is responsible for: - Setting up a FnApiControlClientPoolService to listen for incoming FnApiSdkHarnessClient connections - Wrapping those connections in the higher-level SdkHarnessClient - Building the ProcessBundleDescriptor (instruction graph) to be executed - Establishing data plane endpoints referenced by the ProcessBundleDescriptor - Registering the data plane endpoints and ProcessBundleDescriptor with the SdkHarnessClient This class is responsible for: - Registering each bundle with the SDK harness - Streaming the elements of each bundle to the SDK harness over the data plane ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---