This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1a73185 Publishing website 2021/10/07 18:02:54 at commit 87ffd1f
1a73185 is described below
commit 1a73185a2be03a82bb19a08f63d4d09478837458
Author: jenkins <bui...@apache.org>
AuthorDate: Thu Oct 7 18:02:55 2021 +0000
Publishing website 2021/10/07 18:02:54 at commit 87ffd1f
---
website/generated-content/documentation/index.xml | 14 +++++++-------
.../documentation/programming-guide/index.html | 12 ++++++------
website/generated-content/sitemap.xml | 2 +-
3 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index e3aa7ac..ccdac3b 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -11132,18 +11132,18 @@ $ jar -jar /path/to/expansion_service.jar
&lt;PORT_NUMBER&gt;</code>&
</div>
</div>
<p>The expansion service is now ready to serve transforms on the specified
port.</p>
-<p>When creating SDK-specific wrappers for your transform, SDKs may provide
utilities that are readily available for easily starting up an expansion
service. For example, the Python SDK provides the utilities <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>JavaJarExpansionService
and BeamJarExpansionService</a> utility for starting up a Java expansion
service using a JAR file.</p>
+<p>When creating SDK-specific wrappers for your transform, SDKs may provide
utilities that are readily available for easily starting up an expansion
service. For example, the Python SDK provides the utilities <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService";><code>JavaJarExpansionService</code></a>
and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.tr
[...]
<p><strong>Including dependencies</strong></p>
<p>If your transform requires external libraries, you can include them by
adding them to the classpath of the expansion service. After they are included
in the classpath, they will be staged when your transform is expanded by the
expansion service.</p>
<p><strong>Writing SDK-specific wrappers</strong></p>
-<p>Your cross-language Java transform can be called through the lower-level
<a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>ExternalTransform</a>
class in a multi-language pipeline (as described in the next section);
however, if possible, you should create a SDK-specific wrapper written in the
programming language of the pipeline (such as Python) to access the transform
instead. This higher-level abstraction will make it easier [...]
+<p>Your cross-language Java transform can be called through the lower-level
<a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform";><code>ExternalTransform</code></a>
class in a multi-language pipeline (as described in the next section);
however, if possible, you should create a SDK-specific wrapper written in the
programming language of the pipeline (such as Python) to access the trans [...]
<p>To create an SDK wrapper for use in a Python pipeline, do the
following:</p>
<ol>
<li>
<p>Create a Python module for your cross-language transform(s).</p>
</li>
<li>
-<p>In the module, build the payload that should be used to initiate the
cross-language transform expansion request using one of the available <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>PayloadBuilder</a>
classes.</p>
+<p>In the module, build the payload that should be used to initiate the
cross-language transform expansion request using one of the available <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder";><code>PayloadBuilder</code></a>
classes.</p>
<p>The parameter names and types of the payload should map to parameter
names and types of the configuration POJO provided to the Java
<code>ExternalTransformBuilder</code>. Parameter types are mapped across
SDKs using a <a
href="https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/schema.proto";>Beam
schema</a>. Parameter names are mapped by simply converting Python
underscore-separated variable names to camel-case (Java standard).</p>
<p>In the following example, <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py";>kafka.py</a>
uses <code>NamedTupleBasedPayloadBuilder</code> to build the payload.
The parameters map to the Java <a
href="https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java";>KafkaIO.External.Configuration</a>
config object defined previously in the <strong>Implementing the interfac
[...]
<div class="snippet">
@@ -11161,7 +11161,7 @@ payload =
NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
</div>
</li>
<li>
-<p>Start an expansion service unless one is specified by the pipeline
creator. The Apache Beam Python SDK provides utilities
<code>JavaJarExpansionService</code> and
<code>BeamJarExpansionService</code> for easily starting up an expansion
service using a JAR file.. <code>JavaJarExpansionService</code> can be
used to startup an expansion service using path (a local path or a URL) to a
given JAR file. <code>BeamJarExpansionService</code> can be used for
easily st [...]
+<p>Start an expansion service unless one is specified by the pipeline
creator. The Apache Beam Python SDK provides utilities <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService";><code>JavaJarExpansionService</code></a>
and <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService";><
[...]
<p>For transforms released with Beam do the following:</p>
<ol>
<li>
@@ -11182,7 +11182,7 @@ payload =
NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
</ol>
</li>
<li>
-<p>Add a Python wrapper transform class that extends <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>ExternalTransform</a>.
Pass the payload and expansion service defined above as parameters to the
constructor of the <code>ExternalTransform</code> parent class.</p>
+<p>Add a Python wrapper transform class that extends <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform";><code>ExternalTransform</code></a>.
Pass the payload and expansion service defined above as parameters to the
constructor of the <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform";><
[...]
</li>
</ol>
<h4 id="1312-creating-cross-language-python-transforms">13.1.2. Creating
cross-language Python transforms</h4>
@@ -11322,7 +11322,7 @@ transforms from other languages; see more at <a
href="https://issues.apache.o
</li>
</ol>
<h4 id="1322-using-cross-language-transforms-in-a-python-pipeline">13.2.2.
Using cross-language transforms in a Python pipeline</h4>
-<p>If a Python-specific wrapper for a cross-language transform is
available, use that; otherwise, you have to use the lower-level <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>ExternalTransform</a>
class to access the transform.</p>
+<p>If a Python-specific wrapper for a cross-language transform is
available, use that; otherwise, you have to use the lower-level <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform";><code>ExternalTransform</code></a>
class to access the transform.</p>
<p><strong>Using an SDK wrapper</strong></p>
<p>To use a cross-language transform through an SDK wrapper, import the
module for the SDK wrapper and call it from your pipeline as shown in the
example:</p>
<div class="snippet">
@@ -11355,7 +11355,7 @@ expansion_service=&lt;Address of expansion
service&gt;))
<p>Make sure the transform you&rsquo;re trying to use is available and
can be used by the expansion service. For Java, make sure the builder and
registrar for the transform are available in the classpath of the expansion
service.</p>
</li>
<li>
-<p>Include <code>ExternalTransform</code> when instantiating your
pipeline. Reference the URN, Payload, and expansion service. You can use one of
the available <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>PayloadBuilder</a>
classes to build the payload for <code>ExternalTransform</code>.</p>
+<p>Include <code>ExternalTransform</code> when instantiating your
pipeline. Reference the URN, Payload, and expansion service. You can use one of
the available <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder";><code>PayloadBuilder</code></a>
classes to build the payload for <code>ExternalTransform</code>.</p>
<div class="snippet">
<div class="notebook-skip code-snippet without_switcher">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index b86067b..77d0002 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -4168,14 +4168,14 @@ public static class External implements
ExternalTransformRegistrar {
</code></pre></div></div></li></ol><p>After you have implemented the
<code>ExternalTransformBuilder</code> and
<code>ExternalTransformRegistrar</code> interfaces, your transform can be
registered and created successfully by the default Java expansion
service.</p><p><strong>Starting the expansion service</strong></p><p>An
expansion service can be used with multiple transforms in the same pipeline.
Java has a default expansion service included and available in the Apache Beam
Java SDK [...]
# Start the expansion service at the specified port.
-$ jar -jar /path/to/expansion_service.jar
<PORT_NUMBER></code></pre></div></div><p>The expansion service is now
ready to serve transforms on the specified port.</p><p>When creating
SDK-specific wrappers for your transform, SDKs may provide utilities that are
readily available for easily starting up an expansion service. For example, the
Python SDK provides the utilities <a
href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py>JavaJarExpansio
[...]
+$ jar -jar /path/to/expansion_service.jar
<PORT_NUMBER></code></pre></div></div><p>The expansion service is now
ready to serve transforms on the specified port.</p><p>When creating
SDK-specific wrappers for your transform, SDKs may provide utilities that are
readily available for easily starting up an expansion service. For example, the
Python SDK provides the utilities <a
href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transform
[...]
consumer_config: typing.Mapping[str, str]
topics: typing.List[str]
# Other properties omitted for clarity.
payload = NamedTupleBasedPayloadBuilder(ReadFromKafkaSchema(...))
- </code></pre></div></div></li><li><p>Start an expansion service unless one
is specified by the pipeline creator. The Apache Beam Python SDK provides
utilities <code>JavaJarExpansionService</code> and
<code>BeamJarExpansionService</code> for easily starting up an expansion
service using a JAR file.. <code>JavaJarExpansionService</code> can be used to
startup an expansion service using path (a local path or a URL) to a given JAR
file. <code>BeamJarExpansionService</code> can be used fo [...]
- </code></pre></div></div></li></ol></li><li><p>Add a Python wrapper
transform class that extends <a
href=https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py>ExternalTransform</a>.
Pass the payload and expansion service defined above as parameters to the
constructor of the <code>ExternalTransform</code> parent
class.</p></li></ol><h4
id=1312-creating-cross-language-python-transforms>13.1.2. Creating
cross-language Python transforms</h4><p>To m [...]
+ </code></pre></div></div></li><li><p>Start an expansion service unless one
is specified by the pipeline creator. The Apache Beam Python SDK provides
utilities <a
href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService><code>JavaJarExpansionService</code></a>
and <a
href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpan
[...]
+ </code></pre></div></div></li></ol></li><li><p>Add a Python wrapper
transform class that extends <a
href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform><code>ExternalTransform</code></a>.
Pass the payload and expansion service defined above as parameters to the
constructor of the <a
href=https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.e
[...]
</code></pre></div></div></li><li><p>For an existing Python transform,
create a new class to register the URN with the Python expansion
service.</p><div class=snippet><div class="notebook-skip code-snippet
without_switcher"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><pre><code>@ptransform.PTransform.register_urn(TEST_COMPK_URN,
None)
class CombinePerKeyTransform(ptransform.PTransform):
</code></pre></div></div></li><li><p>From within the class, define an
expand method that takes an input PCollection, runs the Python transform, and
then returns the output PCollection.</p><div class=snippet><div
class="notebook-skip code-snippet without_switcher"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><pre><code>def expand(self, pcoll):
@@ -4191,7 +4191,7 @@ def from_runner_api_parameter(
</code></pre></div></div></li></ol><p><strong>Starting the expansion
service</strong></p><p>An expansion service can be used with multiple
transforms in the same pipeline. Python has a default expansion service
included and available in the Apache Beam Python SDK for you to use with your
Python transforms. You are free to write your own expansion service, but that
is generally not needed, so it is not covered in this section.</p><p>Perform
the following steps to start up the default [...]
</code></pre></div></div></li><li><p>Import any modules that contain
transforms to be made available using the expansion service.</p><div
class=snippet><div class="notebook-skip code-snippet without_switcher"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>$
python -m apache_beam.runners.portability.expansion_service_test -p
$PORT_FOR_EXPANSION_SERVICE
</code></pre></div></div></li><li><p>This expansion service is not ready
to serve up transforms on the address
<code>localhost:$PORT_FOR_EXPANSION_SERVICE</code>.</p></li></ol><p><strong>Including
dependencies</strong></p><p>Currently Python external transforms are limited
to dependencies available in core Beam SDK Harness.</p><h4
id=1313-creating-cross-language-go-transforms>13.1.3. Creating cross-language
Go transforms</h4><p>Go currently does not support creating cross-language tr
[...]
-transforms from other languages; see more at <a
href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h3
id=use-x-lang-transforms>13.2. Using cross-language transforms</h3><p>Depending
on the SDK language of the pipeline, you can use a high-level SDK-wrapper
class, or a low-level transform class to access a cross-language
transform.</p><h4
id=1321-using-cross-language-transforms-in-a-java-pipeline>13.2.1. Using
cross-language transforms in a Java pipeline</h4><p>Current [...]
+transforms from other languages; see more at <a
href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h3
id=use-x-lang-transforms>13.2. Using cross-language transforms</h3><p>Depending
on the SDK language of the pipeline, you can use a high-level SDK-wrapper
class, or a low-level transform class to access a cross-language
transform.</p><h4
id=1321-using-cross-language-transforms-in-a-java-pipeline>13.2.1. Using
cross-language transforms in a Java pipeline</h4><p>Current [...]
kafka_records = (
pipeline
@@ -4203,7 +4203,7 @@ kafka_records = (
topics=[self.topic],
max_num_records=max_num_records,
expansion_service=<Address of expansion service>))
- </code></pre></div></div><p><strong>Using the ExternalTransform
class</strong></p><p>When an SDK-specific wrapper isn’t available, you
will have to access the cross-language transform through the
<code>ExternalTransform</code> class.</p><ol><li><p>Make sure you have any
runtime environment dependencies (like JRE) installed on your local machine.
See the expansion service section for more details.</p></li><li><p>Start up the
expansion service for the SDK that is in the language of [...]
+ </code></pre></div></div><p><strong>Using the ExternalTransform
class</strong></p><p>When an SDK-specific wrapper isn’t available, you
will have to access the cross-language transform through the
<code>ExternalTransform</code> class.</p><ol><li><p>Make sure you have any
runtime environment dependencies (like JRE) installed on your local machine.
See the expansion service section for more details.</p></li><li><p>Start up the
expansion service for the SDK that is in the language of [...]
res = (
p
| beam.Create(['a', 'b']).with_output_types(unicode)
@@ -4245,7 +4245,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr,
beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/10/04</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/10/07</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 4ad78bf..16d8a39 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
