This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new e90c3ba Publishing website 2019/07/17 00:33:55 at commit 4a7ba2d e90c3ba is described below commit e90c3ba51d553e065140ea711bfd11fe628df371 Author: jenkins <bui...@apache.org> AuthorDate: Wed Jul 17 00:33:55 2019 +0000 Publishing website 2019/07/17 00:33:55 at commit 4a7ba2d --- .../get-started/quickstart-py/index.html | 9 ++++-- .../get-started/wordcount-example/index.html | 9 ++++-- .../roadmap/portability/index.html | 37 ++++++++++++++-------- 3 files changed, 35 insertions(+), 20 deletions(-) diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html index 9979577..0f2f3f7 100644 --- a/website/generated-content/get-started/quickstart-py/index.html +++ b/website/generated-content/get-started/quickstart-py/index.html @@ -414,15 +414,18 @@ environment’s directories.</p> </code></pre> </div> -<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-flink for more information. </code></pre> </div> -<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/documentation/runners/flink/ for more information. </code></pre> </div> -<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Spark requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-spark for more information. </code></pre> </div> diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html index ed2cb21..73a8898 100644 --- a/website/generated-content/get-started/wordcount-example/index.html +++ b/website/generated-content/get-started/wordcount-example/index.html @@ -675,15 +675,18 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl </code></pre> </div> -<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-flink-local highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-flink for more information. </code></pre> </div> -<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-flink-cluster highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/documentation/runners/flink/ for more information. </code></pre> </div> -<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>This runner is not yet available for the Python SDK. +<div class="runner-spark highlighter-rouge"><pre class="highlight"><code>Currently, running wordcount.py on Spark requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-spark for more information. </code></pre> </div> diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index 7e0fb1b..01737d2 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -229,7 +229,8 @@ <li><a href="#issues">Issues</a></li> <li><a href="#status">Status</a> <ul> - <li><a href="#python-on-flink">Running Python wordcount on Flink or Spark</a></li> + <li><a href="#python-on-flink">Running Python wordcount on Flink</a></li> + <li><a href="#python-on-spark">Running Python wordcount on Spark</a></li> </ul> </li> </ul> @@ -389,33 +390,41 @@ their respective components.</p> <p>MVP, and FeatureCompletness nearly done (missing SDF, timers) for SDKs, Python ULR, and shared java runners library. -Flink is the first runner to fully leverage this, with focus moving to -Performance. +Currently, the Flink and Spark runners support portable pipeline execution. See the <a href="https://s.apache.org/apache-beam-portability-support-table">Portability support table</a> for details.</p> -<h3 id="python-on-flink">Running Python wordcount on Flink or Spark</h3> +<h3 id="python-on-flink">Running Python wordcount on Flink</h3> -<p>Currently, the Flink and Spark runners support portable pipeline execution. -To run a basic Python wordcount (in batch mode) with embedded Flink or Spark:</p> +<p>To run a basic Python wordcount (in batch mode) with embedded Flink:</p> <ol> <li>Run once to build the SDK harness container: <code class="highlighter-rouge">./gradlew :sdks:python:container:docker</code></li> - <li>Choose one: - <ul> - <li>Start the Flink portable JobService endpoint: <code class="highlighter-rouge">./gradlew :runners:flink:1.5:job-server:runShadow</code></li> - <li>Or start the Spark portable JobService endpoint: <code class="highlighter-rouge">./gradlew :runners:spark:job-server:runShadow</code></li> - </ul> - </li> - <li>Submit the wordcount pipeline to above endpoint: <code class="highlighter-rouge">./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK</code></li> + <li>Start the Flink portable JobService endpoint: <code class="highlighter-rouge">./gradlew :runners:flink:1.5:job-server:runShadow</code></li> + <li>In a new terminal, submit the wordcount pipeline to above endpoint: <code class="highlighter-rouge">./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK</code></li> </ol> -<p>To run the pipeline in streaming mode (currently only supported on Flink): <code class="highlighter-rouge">./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming</code></p> +<p>To run the pipeline in streaming mode: <code class="highlighter-rouge">./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming</code></p> <p>Please see the <a href="/documentation/runners/flink/">Flink Runner page</a> for more information on how to run portable pipelines on top of Flink.</p> +<h3 id="python-on-spark">Running Python wordcount on Spark</h3> + +<p>To run a basic Python wordcount (in batch mode) with embedded Spark:</p> + +<ol> + <li>Run once to build the SDK harness container: <code class="highlighter-rouge">./gradlew :sdks:python:container:docker</code></li> + <li>Start the Spark portable JobService endpoint: <code class="highlighter-rouge">./gradlew :runners:spark:job-server:runShadow</code></li> + <li>In a new terminal, submit the wordcount pipeline to above endpoint: <code class="highlighter-rouge">./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK</code></li> +</ol> + +<p>Python streaming mode is not yet supported on Spark.</p> + +<p>Please see the <a href="/documentation/runners/spark/">Spark Runner page</a> for more information on +how to run portable pipelines on top of Spark.</p> + </div> </div>