robertwb commented on code in PR #27632:
URL: https://github.com/apache/beam/pull/27632#discussion_r1290783117
##########
website/www/site/content/en/contribute/runner-guide.md:
##########
@@ -813,83 +723,47 @@ message PCollection {
### `Coder` proto
This is a very interesting proto. A coder is a parameterized function that may
-only be understood by a particular SDK, hence an `SdkFunctionSpec`, but also
+only be understood by a particular SDK, hence an `FunctionSpec`, but also
may have component coders that fully define it. For example, a `ListCoder` is
only a meta-format, while `ListCoder(VarIntCoder)` is a fully specified format.
{{< highlight class="no-toggle" >}}
message Coder {
- SdkFunctionSpec spec;
+ FunctionSpec spec;
repeated string component_coder_ids;
}
{{< /highlight >}}
-## The Runner API RPCs
+There are a large number of
+[standard
coders](https://github.com/apache/beam/blob/release-2.49.0/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L829)
+understood by most, if not all,
+SDKs. Using these allows for cross-language transforms.
-While your language's SDK will probably insulate you from touching the Runner
-API protos directly, you may need to implement adapters for your runner, to
-expose it to another language. So this section covers proto that you will
-possibly interact with quite directly.
+## The Jobs API RPCs
-The specific manner in which the existing runner method calls will be expressed
-as RPCs is not implemented as proto yet. This RPC layer is to enable, for
-example, building a pipeline using the Python SDK and launching it on a runner
-that is written in Java. It is expected that a small Python shim will
-communicate with a Java process or service hosting the Runner API.
+[Overview](https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE/edit#slide=id.g42e4c9aad6_1_3722)
+[Spec](https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_job_api.proto)
+
+While your language's SDK will may insulate you from touching the Runner
+API protos directly, you may need to implement adapters for your runner, to
+expose it to another language.
+This allows a Python SDK to invoke a Java runner or vice versa.
+A typical implementation of this can be found in
+[local_job_service.py](https://github.com/apache/beam/blob/release-2.48.0/sdks/python/apache_beam/runners/portability/local_job_service.py)
+which is used directly to front several Python-implemented runners.
The RPCs themselves will necessarily follow the existing APIs of PipelineRunner
and PipelineResult, but altered to be the minimal backend channel, versus a
rich and convenient API.
-### `PipelineRunner.run(Pipeline)` RPC
-
-This will take the same form, but `PipelineOptions` will have to be serialized
-to JSON (or a proto `Struct`) and passed along.
+A key piece of this is the
+(Artifacts
API)[https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/org/apache/beam/model/job_management/v1/beam_artifact_api.proto],
+which allows a Runner to fetch and deploy binary artifacts (such as jars,
+pypi packages, etc.) that are listed as dependencies in the various
environments,
+and may have various representations. This is invoked after a pipeline
+is submitted, but before it is executed. The SDK submitting a pipeline acts
+as an artifact server to the runner receiving the request, and in turn the
+runner then acts as an artifact server to the workers (environments) hosting
+the users UDFs.
-{{< highlight class="no-toggle" >}}
-message RunPipelineRequest {
- Pipeline pipeline;
- Struct pipeline_options;
-}
-{{< /highlight >}}
-
-{{< highlight class="no-toggle" >}}
-message RunPipelineResponse {
- bytes pipeline_id;
-
- // TODO: protocol for rejecting pipelines that cannot be executed
- // by this runner. May just be REJECTED job state with error message.
-
- // totally opaque to the SDK; for the shim to interpret
- Any contents;
-}
-{{< /highlight >}}
-
-### `PipelineResult` aka "Job API"
-
-The two core pieces of functionality in this API today are getting the state of
-a job and canceling the job. It is very much likely to evolve, for example to
-be generalized to support draining a job (stop reading input and let watermarks
-go to infinity). Today, verifying our test framework benefits (but does not
-depend upon wholly) querying metrics over this channel.
-
-{{< highlight class="no-toggle" >}}
-message CancelPipelineRequest {
- bytes pipeline_id;
- ...
-}
-
-message GetStateRequest {
- bytes pipeline_id;
- ...
-}
-
-message GetStateResponse {
- JobState state;
- ...
-}
-
-enum JobState {
- ...
-}
{{< /highlight >}}
Review Comment:
Nice catch! Thanks. (Sure would be nice if it pointed us at the bad
line--this brings back memories of hunting down g++ template errors...)
Fingers crossed...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
