This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9487927adce [#32562] Incorporate Prism into the Beam Website. (#32563)
9487927adce is described below
commit 9487927adce935db60caf4ee428c7d0aab654b1a
Author: Robert Burke <lostl...@users.noreply.github.com>
AuthorDate: Mon Sep 30 11:46:40 2024 -0700
[#32562] Incorporate Prism into the Beam Website. (#32563)
---
.github/autolabeler.yml | 7 +-
README.md | 1 +
website/www/site/content/en/blog/beam-2.59.0.md | 2 +-
.../documentation/pipelines/test-your-pipeline.md | 4 +-
.../content/en/documentation/runners/direct.md | 1 +
.../site/content/en/documentation/runners/prism.md | 145 +++++++++++++++++++++
website/www/site/content/en/roadmap/_index.md | 4 +-
.../www/site/content/en/roadmap/prism-runner.md | 35 +++++
.../www/site/data/en/documentation_runners.yaml | 3 +
.../layouts/partials/section-menu/en/roadmap.html | 1 +
.../layouts/partials/section-menu/en/runners.html | 1 +
11 files changed, 196 insertions(+), 8 deletions(-)
diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml
index a4ce41fce07..a1a4c3bf039 100644
--- a/.github/autolabeler.yml
+++ b/.github/autolabeler.yml
@@ -67,7 +67,7 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*",
"sdks/python/apache_beam/
"mqtt": ["sdks/java/io/mqtt/**/*"]
"parquet": ["sdks/java/io/parquet/**/*"]
"rabbitmq": ["sdks/java/io/rabbitmq/**/*"]
-"redis": ["sdks/java/io/redis/**/*"]
+"redis": ["sdks/java/io/redis/**/*"]
"solr": ["sdks/java/io/solr/**/*"]
"spanner": ["sdks/go/pkg/beam/io/spannerio/**/*",
"sdks/python/apache_beam/io/gcp/spanner.py",
"sdks/python/apache_beam/io/gcp/experimental/spannerio.py",
"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/**/*"]
"bigtable": ["sdks/go/pkg/beam/io/bigtableio/**/*",
"sdks/go/pkg/beam/io/xlang/bigtableio/**/*",
"sdks/python/apache_beam/io/gcp/bigtableio.py",
"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/**/*"]
@@ -76,9 +76,9 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*",
"sdks/python/apache_beam/
"thrift": ["sdks/java/io/thrift/**/*"]
"tika": ["sdks/java/io/tika/**/*"]
"xml": ["sdks/java/io/xml/**/*"]
-
+
# Runners
-"runners": ["runners/**/*", "sdks/go/pkg/beam/runners/**/*",
"sdks/python/runners/**/*", "sdks/typescript/src/apache_beam/runners/**/*"]
+"runners": ["runners/**/*", "sdks/go/pkg/beam/runners/**/*",
"sdks/python/apache_beam/runners/**/*",
"sdks/typescript/src/apache_beam/runners/**/*"]
"core": ["runners/core-construction-java/**/*", "runners/core-java/**/*"]
"dataflow": ["runners/google-cloud-dataflow-java/**/*",
"sdks/go/pkg/beam/runners/dataflow/**/*", "sdks/python/runners/dataflow/**/*"]
"direct": ["runners/direct-java/**/*", "sdks/go/pkg/beam/runners/direct/**/*",
"sdks/python/runners/direct/**/*"]
@@ -88,6 +88,7 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*",
"sdks/python/apache_beam/
"jet": ["runners/jet/**/*"]
"local": ["runners/local-java/**/*"]
"portability": ["runners/portability/**/*"]
+"prism": ["runners/prism/**/*", "sdks/go/pkg/beam/runners/prism/**/*",
"sdks/go/cmd/prism/**/*",
"sdks/python/apache_beam/runners/portability/prism_runner.py","sdks/python/apache_beam/runners/portability/prism_runner_test.py"]
"samza": ["runners/samza/**/*"]
"spark": ["runners/spark/**/*", "sdks/go/pkg/beam/runners/spark/**/*"]
"twister2": ["runners/twister2/**/*"]
diff --git a/README.md b/README.md
index 8a6db162206..24ab4963701 100644
--- a/README.md
+++ b/README.md
@@ -65,6 +65,7 @@ Have ideas for new SDKs or DSLs? See the [sdk-ideas
label](https://github.com/ap
Beam supports executing programs on multiple distributed processing backends
through PipelineRunners. Currently, the following PipelineRunners are available:
- The `DirectRunner` runs the pipeline on your local machine.
+- The `PrismRunner` runs the pipeline on your local machine using Beam
Portability.
- The `DataflowRunner` submits the pipeline to the [Google Cloud
Dataflow](http://cloud.google.com/dataflow/).
- The `FlinkRunner` runs the pipeline on an Apache Flink cluster. The code has
been donated from
[dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow)
and is now part of Beam.
- The `SparkRunner` runs the pipeline on an Apache Spark cluster.
diff --git a/website/www/site/content/en/blog/beam-2.59.0.md
b/website/www/site/content/en/blog/beam-2.59.0.md
index 68b712ae8fe..2720eaed738 100644
--- a/website/www/site/content/en/blog/beam-2.59.0.md
+++ b/website/www/site/content/en/blog/beam-2.59.0.md
@@ -30,7 +30,7 @@ For more information on changes in 2.59.0, check out the
[detailed release notes
## Highlights
* Added support for setting a configureable timeout when loading a model and
performing inference in the
[RunInference](https://beam.apache.org/documentation/ml/inference-overview/)
transform using
[with_exception_handling](https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.base.html#apache_beam.ml.inference.base.RunInference.with_exception_handling)
([#32137](https://github.com/apache/beam/issues/32137))
-* Initial experimental support for using Prism with the Java and Python SDKs
+* Initial experimental support for using
[Prism](/documentation/runners/prism/) with the Java and Python SDKs
* Prism is presently targeting local testing usage, or other small scale
execution.
* For Java, use 'PrismRunner', or 'TestPrismRunner' as an argument to the
`--runner` flag.
* For Python, use 'PrismRunner' as an argument to the `--runner` flag.
diff --git
a/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
b/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
index 05767fea484..82f5e33ed07 100644
--- a/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
+++ b/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
@@ -22,9 +22,9 @@ Testing your pipeline is a particularly important step in
developing an effectiv
Before running your pipeline on the runner of your choice, unit testing your
pipeline code locally is often the best way to identify and fix bugs in your
pipeline code. Unit testing your pipeline locally also allows you to use your
familiar/favorite local debugging tools.
-You can use [DirectRunner](/documentation/runners/direct), a local runner
helpful for testing and local development.
+You can use [DirectRunner](/documentation/runners/direct), or
[PrismRunner](/documentation/runners/prism). Both are local runners helpful for
testing and local development.
-After you test your pipeline using the `DirectRunner`, you can use the runner
of your choice to test on a small scale. For example, use the Flink runner with
a local or remote Flink cluster.
+After you test your pipeline locally, you can use the runner of your choice to
test on a small scale. For example, use the Flink runner with a local or remote
Flink cluster.
The Beam SDKs provide a number of ways to unit test your pipeline code, from
the lowest to the highest levels. From the lowest to the highest level, these
are:
diff --git a/website/www/site/content/en/documentation/runners/direct.md
b/website/www/site/content/en/documentation/runners/direct.md
index e18d4a902c1..82e7127d19c 100644
--- a/website/www/site/content/en/documentation/runners/direct.md
+++ b/website/www/site/content/en/documentation/runners/direct.md
@@ -32,6 +32,7 @@ Using the Direct Runner for testing and development helps
ensure that pipelines
Here are some resources with information about how to test your pipelines.
<ul>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your
Pipeline</a></li>
<!-- Java specific links -->
<li class="language-java"><a
href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache
Beam</a> talks about the use of Java classes <a
href="https://beam.apache.org/releases/javadoc/{{< param release_latest
>}}/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a
href="https://beam.apache.org/releases/javadoc/{{< param release_latest
>}}/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to
test your pipelines.</li>
<li class="language-java">The <a
href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache
Beam WordCount Walkthrough</a> contains an example of logging and testing a
pipeline with <a href="https://beam.apache.org/releases/javadoc/{{< param
release_latest
>}}/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a>.</li>
diff --git a/website/www/site/content/en/documentation/runners/prism.md
b/website/www/site/content/en/documentation/runners/prism.md
new file mode 100644
index 00000000000..a898a691812
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/prism.md
@@ -0,0 +1,145 @@
+---
+type: runners
+title: "Prism Runner"
+aliases: /learn/runners/prism/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Overview
+
+The Apache Beam Prism Runner can be used to execute Beam pipelines locally
using [Beam Portability](/roadmap/portability/).
+
+The Prism runner is suitable for small scale local testing and provides:
+
+* A statically compiled, single binary for simple deployment without
additional configuration.
+* A web UI when executing in stand alone mode.
+* A direct implementation of Beam execution semantics.
+* A streaming-first runtime that supports batch processing and data streaming
programs.
+* Fast, in-memory execution for to simplify SDK, Transform, and Pipeline
development.
+* Cross Language Transform support.
+
+Written in [Go](https://go.dev), it is the default runner for the [Go
SDK](/roadmap/go-sdk/), but can be used in other SDKs as well (see below).
+
+# Capabilities
+
+While Prism already supports a great deal of Beam features, it doesn't yet
support everything.
+Prism is under active development to close these gaps.
+
+With the exception of timer issues, use of unsupported features should fail
the pipeline at job submission time.
+
+In the [2.59.0 release](/blog/beam-2.59.0/), Prism passes most runner
validations tests with the exceptions of pipelines using the following features:
+
+OrderedListState, OnWindowExpiry (eg. GroupIntoBatches), CustomWindows,
MergingWindowFns, Trigger and WindowingStrategy associated features, Bundle
Finalization, Looping Timers, and some Coder related issues such as with Python
combiner packing, and Java Schema transforms, and heterogenous flatten coders.
+Processing Time timers do not yet have real time support.
+
+
+See the [Roadmap](/roadmap/prism-runner/) for how to find current progress.
+Specific feature support information will soon migrate to the [Runner
Capability Matrix](/documentation/runners/capability-matrix/).
+
+# Using the Prism Runner
+
+{{< language-switcher go java py >}}
+
+<span class="language-go">Prism is the default runner for the Go SDK and is
used automatically. Set the runner with the flag `--runner=PrismRunner`. </span>
+<span class="language-java">Set the runner to `PrismRunner`. </span>
+<span class="language-py">Set the runner to `PrismRunner`. </span>
+
+For other SDKs, Prism is included as an asset on [Beam Github
Releases](https://github.com/apache/beam/releases/tag/v{{< param release_latest
>}}) for download and stand alone use.
+
+Here are some resources with information about how to test your pipelines.
+<ul>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your
Pipeline</a></li>
+ <li>The <a
href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache
Beam WordCount Walkthrough</a> contains an example of logging and testing a
pipeline with asserts.
+ <!-- Java specific links -->
+ <li class="language-java"><a
href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache
Beam</a> talks about the use of Java classes <a
href="https://beam.apache.org/releases/javadoc/{{< param release_latest
>}}/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a
href="https://beam.apache.org/releases/javadoc/{{< param release_latest
>}}/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to
test your pipelines.</li>
+</ul>
+
+### Specify your dependency
+
+<span class="language-java">When using Java, you must specify your dependency
on the Direct Runner in your `pom.xml`.</span>
+{{< highlight java >}}
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-prism-java</artifactId>
+ <version>{{< param release_latest >}}</version>
+ <scope>runtime</scope>
+</dependency>
+{{< /highlight >}}
+
+<span class="language-py">This section is not applicable to the Beam SDK for
Python. Prism is built in.</span>
+<span class="language-go">This section is not applicable to the Beam SDK for
Go. Prism is built in.</span>
+
+Except for the Go SDK, Prism is included as an asset on [Beam Github
Releases](https://github.com/apache/beam/releases/tag/v{{< param release_latest
>}}) for automatic download, startup, and shutdown on SDKs.
+The binary is cached locally for subsequent executions.
+
+## Pipeline options for the Prism Runner
+
+Prism aims to have minimal configuration required, and does not currently
present user pipeline options.
+
+## Running Prism Standalone
+
+Prism can be executed as a stand alone binary and will present a basic UI for
listing jobs, and job status.
+This is an optional mode for Prism that is useful for demos or rapid iteration.
+It is not a requirement for using Prism in the Java or Python SDKs.
+
+This can be done in two ways, downloading an asset from the github release, or
building the binary locally with Go installed.
+
+In either case, Prism serves a JobManagement API endpoint, and a Webpage UI
locally.
+Jobs can be submitted using `--runner=PortableRunner --endpoint=<endpoint
address>` and monitored using the webpage UI.
+
+Example output from the Prism binary:
+
+```
+2024/09/30 09:56:42 INFO Serving JobManagement endpoint=localhost:8073
+2024/09/30 09:56:42 INFO Serving WebUI endpoint=http://localhost:8074
+```
+
+The binary has the following optional flags:
+
+* `--job_port` sets the port for the Job management server (defaults to 8073)
+* `--web_port` sets the port for the web ui (defaults to 8074)
+* `--serve_http` enables or disables the web ui (defaults to true)
+* `---idle_shutdown_timeout` sets a duration that Prism will wait for a new
job before automatically shutting itself down. Uses duration format like `10s`,
`5m`,`2h`. Defaults to not shutting down.
+
+### Download a release asset
+
+This approach doesn't require other dependencies or runtimes installed.
+This is recommended if you want to deploy Prism on some other machine.
+
+Navigate to the latest [Beam Release Github
page](https://github.com/apache/beam/releases/tag/v{{< param release_latest
>}}), scroll to the bottom, and download the correct asset for where you want
to run Prism.
+
+For example, if you want to execute Prism on a newer MacBook, you'd download
the `darwin-arm64` asset. For executing on many cloud machines, you'd download
the `linux-amd64` asset.
+
+This requires downloading the right asset for the machine Prism will run on,
such as your development machine.
+
+Simply unzip, and execute.
+
+### Build from the release with Go.
+
+This approach requires a [recent version of Go installed](https://go.dev/dl/).
+This is recommended if you only want to run Prism on your local machine.
+
+You can insall Prism with `go install`:
+
+```sh
+go install github.com/apache/beam/sdks/v2/go/cmd/prism@latest
+prism
+```
+
+Or simply build and execute the binary immeadiately using `go run`:
+
+```sh
+go run github.com/apache/beam/sdks/v2/go/cmd/prism@latest
+```
diff --git a/website/www/site/content/en/roadmap/_index.md
b/website/www/site/content/en/roadmap/_index.md
index 8074d5a710a..b845b64ffbb 100644
--- a/website/www/site/content/en/roadmap/_index.md
+++ b/website/www/site/content/en/roadmap/_index.md
@@ -32,8 +32,8 @@ Below are some highlights for the project as a whole.
Portability is the primary Beam vision: running pipelines authored with _any
SDK_
on _any runner_. This is a cross-cutting effort across Java, Python, and Go,
and every Beam runner. Portability is currently supported on the
-[Flink](/documentation/runners/flink/)
-and [Spark](/documentation/runners/spark/) runners.
+[Flink](/documentation/runners/flink/), [Spark](/documentation/runners/spark/)
+and [Prism](/documentation/runners/prism/) runners.
See the details on the [Portability Roadmap](/roadmap/portability/)
diff --git a/website/www/site/content/en/roadmap/prism-runner.md
b/website/www/site/content/en/roadmap/prism-runner.md
new file mode 100644
index 00000000000..7f741b94d2a
--- /dev/null
+++ b/website/www/site/content/en/roadmap/prism-runner.md
@@ -0,0 +1,35 @@
+---
+title: "Prism Runner Roadmap"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Apache Beam Prism Runner Roadmap
+
+The goal for the Prism runner is to provide a good default onboarding
experience for Apache Beam.
+
+* Prism should be able to execute any Beam pipeline that can execute on a
local machine.
+* Prism should be fast to start and execute pipelines.
+* Prism should be able to assist with the local testing and debugging of
pipelines.
+* Prism may develop into a robust, production ready runner for pipelines that
can execute locally.
+
+The detailed roadmap lives in an [umbrella tracking issue in
Github](https://github.com/apache/beam/issues/29650).
+
+Here are available resources:
+
+ - [Runner documentation](/documentation/runners/prism)
+ - Issues:
[prism](https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aprism)
+ - CLI [Code](https://github.com/apache/beam/tree/master/sdks/go/cmd/prism),
CLI Binaries are available as assets on [Github
Releases](https://github.com/apache/beam/releases/tag/v{{< param release_latest
>}}).
+ - Core
[Code](https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism)
+ - [Prism Internals Deep
Dive](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/README.md)
diff --git a/website/www/site/data/en/documentation_runners.yaml
b/website/www/site/data/en/documentation_runners.yaml
index 0d5901a4525..aa8480e3459 100644
--- a/website/www/site/data/en/documentation_runners.yaml
+++ b/website/www/site/data/en/documentation_runners.yaml
@@ -13,6 +13,9 @@
- name: { text: "DirectRunner:", link: /documentation/runners/direct/ }
icon: icons/documentation/runners/beam-icon.svg
description: Runs locally on your machine – great for developing, testing,
and debugging.
+- name: { text: "PrismRunner:", link: /documentation/runners/prism/ }
+ icon: icons/documentation/runners/beam-icon.svg
+ description: Runs locally on your machine – great for developing, testing,
and debugging.
- name: { text: "FlinkRunner:", link: /documentation/runners/flink/ }
icon: icons/documentation/runners/flink-icon.svg
description: Runs on <a target="_blank"
href="https://flink.apache.org";>Apache Flink</a>.
diff --git a/website/www/site/layouts/partials/section-menu/en/roadmap.html
b/website/www/site/layouts/partials/section-menu/en/roadmap.html
index 91624ac0912..8c8d14ecce2 100644
--- a/website/www/site/layouts/partials/section-menu/en/roadmap.html
+++ b/website/www/site/layouts/partials/section-menu/en/roadmap.html
@@ -29,6 +29,7 @@
<ul class="section-nav-list">
<li><a href="/roadmap/dataflow-runner/">Google Cloud Dataflow
Runner</a></li>
+ <li><a href="/roadmap/prism-runner/">Prism Runner</a></li>
<li><a href="/roadmap/flink-runner/">Flink Runner</a></li>
<li><a href="/roadmap/nemo-runner/">Nemo Runner</a></li>
<li><a href="/roadmap/spark-runner/">Spark Runner</a></li>
diff --git a/website/www/site/layouts/partials/section-menu/en/runners.html
b/website/www/site/layouts/partials/section-menu/en/runners.html
index 3d9d834b5bf..58576bb5f96 100644
--- a/website/www/site/layouts/partials/section-menu/en/runners.html
+++ b/website/www/site/layouts/partials/section-menu/en/runners.html
@@ -13,6 +13,7 @@
<li><span class="section-nav-list-main-title">Runners</span></li>
<li><a href="/documentation/runners/capability-matrix/">Capability
Matrix</a></li>
<li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+<li><a href="/documentation/runners/prism/">Prism Runner</a></li>
<li><a href="/documentation/runners/flink/">Apache Flink</a></li>
<li><a href="/documentation/runners/nemo/">Apache Nemo</a></li>
<li><a href="/documentation/runners/samza/">Apache Samza</a></li>
