This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4f6f75d Publishing website 2021/11/13 06:01:42 at commit fd00fc9
4f6f75d is described below
commit 4f6f75db23f7a49547d4680b39204c63f16126f1
Author: jenkins <bui...@apache.org>
AuthorDate: Sat Nov 13 06:01:43 2021 +0000
Publishing website 2021/11/13 06:01:42 at commit fd00fc9
---
website/generated-content/documentation/index.xml | 158 +++++++++++++++++++++
.../patterns/cross-language/index.html | 4 +-
.../documentation/programming-guide/index.html | 60 +++++++-
website/generated-content/sitemap.xml | 2 +-
4 files changed, 217 insertions(+), 7 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 75db27a..bfc9450 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -11379,6 +11379,115 @@ use case.</p>
<p>At runtime, the Beam runner will execute both Python and Java transforms
to execute your pipeline.</p>
<p>In this section, we will use <a
href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html";>KafkaIO.Read</a>
to illustrate how to create a cross-language transform for Java and a test
example for Python.</p>
<h4 id="1311-creating-cross-language-java-transforms">13.1.1. Creating
cross-language Java transforms</h4>
+<p>There are two ways to make Java transforms available to other
SDKs.</p>
+<ul>
+<li>Option 1: In some cases, you can use existing Java transforms from
other SDKs without writing any additional Java code.</li>
+<li>Option 2: You can use arbitrary Java Transforms from other SDKs by
adding a few Java classes.</li>
+</ul>
+<h5
id="13111-using-existing-java-transforms-from-other-sdks-without-writing-more-java-code">13.1.1.1
Using Existing Java Transforms from Other SDKs Without Writing more Java
Code</h5>
+<p>Starting with Beam 2.34.0, Python SDK users can use some Java transforms
without writing additional Java code. This can be useful in many cases. For
example,</p>
+<ul>
+<li>A developer not familiar with Java may need to use an existing Java
transform from a Python pipeline</li>
+<li>A developer may need to make an existing Java transform available to a
Python pipeline without writing/releasing more Java code</li>
+</ul>
+<blockquote>
+<p><strong>Note:</strong> This feature is currently only available
when using Java transforms from a Python pipeline.</p>
+</blockquote>
+<p>To be eligible for direct usage, the API of the Java transform has to
follow the following pattern.</p>
+<ul>
+<li>Requirement 1: The Java transform can be constructed using an available
public constructor or a public static method (a constructor method) in the same
Java class.</li>
+<li>Requirement 2: The Java transform can be configured using one or more
builder methods. Each builder method should be public and should return an
instance of the Java transform.</li>
+</ul>
+<p>See below for the structure of an example Java class that can be
directly used from the Python API.</p>
+<div class="snippet">
+<div class="notebook-skip code-snippet without_switcher">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<pre><code>public class JavaDataGenerator extends
PTransform&lt;PBegin, PCollection&lt;String&gt;&gt; {
+. . .
+// Following method satisfies the Requirement 1.
+// Note that you may also use a class constructor instead of a static method.
+public static JavaDataGenerator create(Integer size) {
+return new JavaDataGenerator(size);
+}
+static class JavaDataGeneratorConfig implements Serializable {
+public String prefix;
+public long length;
+public String suffix;
+. . .
+}
+// Following method conforms to the Requirement 2
+public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig
dataConfig) {
+return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+}
+. . .
+}</code></pre>
+</div>
+</div>
+<p>To use a Java class that conforms to the above requirement from a Python
SDK pipeline you may do the following.</p>
+<ul>
+<li>Step 1: create an allowlist file in the <em>yaml</em> format that
describes the Java transform classes and methods that will be directly accessed
from Python.</li>
+<li>Step 2: start an Expansion Service with the
<code>javaClassLookupAllowlistFile</code> option passing path to the
allowlist defined in Step 1 as the value.</li>
+<li>Step 3: Use the Python <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py";>JavaExternalTransform</a>
API to directly
+access Java transforms defined in the allowlist from the Python side.</li>
+</ul>
+<p>Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in
corresponding sections below.</p>
+<h5 id="step-1">Step 1</h5>
+<p>To use this Java transform from Python, you may define an allowlist file
in the <em>yaml</em> format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be
used from the Python side.</p>
+<p>Starting with Beam 2.35.0, you have the option to specify
<code>*</code> to the <code>javaClassLookupAllowlistFile</code>
option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the expansion
service may be accessed through the API.</p>
+<div class="snippet">
+<div class="notebook-skip code-snippet without_switcher">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<pre><code>version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+allowedConstructorMethods:
+- create
+allowedBuilderMethods:
+- withJavaDataGeneratorConfig</code></pre>
+</div>
+</div>
+<h5 id="step-2">Step 2</h5>
+<p>The allowlist is provided as an argument when starting up the Java
expansion service. For example, the expansion service can be started
+as a local Java process using the following command.</p>
+<div class="snippet">
+<div class="notebook-skip code-snippet without_switcher">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<pre><code>java -jar &lt;jar file&gt; &lt;port&gt;
--javaClassLookupAllowlistFile=&lt;path to the allowlist
file&gt;</code></pre>
+</div>
+</div>
+<p>Starting with Beam 2.35.0, Beam ``JavaExternalTransform<code>API will
automatically startup an expansion service with a given set of</code>jar`
file dependencies
+if an expansion service address was not provided.</p>
+<h5 id="step-3">Step 3</h5>
+<p>You can directly use the Java class from your Python pipeline using a
stub transform created using the <code>JavaExternalTransform</code> API.
This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to
configure the class.</p>
+<p>Constructor and method parameter types are mapped between Python and
Java using a Beam Schema. The Schema is auto-generated using the object types
+provided on the Python side. If the Java class constructor method or builder
method accepts any complex object types, make sure that the Beam Schema
+for these objects is registered and available for the Java expansion service.
If a schema has not been registered, the Java expansion service will
+try to register a schema using <a
href="https://beam.apache.org/documentation/programming-guide/#creating-schemas";>JavaFieldSchema</a>.
In Python arbitrary objects
+can be represented using <code>NamedTuple</code>s which will be
represented as Beam Rows in the Schema. See below for a Python stub transform
that represents the above
+mentioned Java transform.</p>
+<div class="snippet">
+<div class="notebook-skip code-snippet without_switcher">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<pre><code>JavaDataGeneratorConfig = typing.NamedTuple(
+&#39;JavaDataGeneratorConfig&#39;, [(&#39;prefix&#39;, str),
(&#39;length&#39;, int), (&#39;suffix&#39;, str)])
+data_config = JavaDataGeneratorConfig(prefix=&#39;start&#39;,
length=20, suffix=&#39;end&#39;)
+java_transform = JavaExternalTransform(
+&#39;my.beam.transforms.JavaDataGenerator&#39;,
expansion_service=&#39;localhost:&lt;port&gt;&#39;).create(numpy.int32(100)).withJavaDataGeneratorConfig(data_config)</code></pre>
+</div>
+</div>
+<p>This transform can be used in a Python pipeline along with other Python
transforms.</p>
+<h5
id="13112-full-api-for-making-existing-java-transforms-available-to-other-sdks">13.1.1.2
Full API for Making Existing Java Transforms Available to Other SDKs</h5>
<p>To make your Apache Beam Java SDK transform portable across SDK
languages, you must implement two interfaces: <a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java";>ExternalTransformBuilder</a>
and <a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java";>ExternalTransformRegistrar</a>.
The <code [...]
<p><strong>Implementing the interfaces</strong></p>
<ol>
@@ -11634,6 +11743,52 @@ return CombinePerKeyTransform()
<h4 id="1313-creating-cross-language-go-transforms">13.1.3. Creating
cross-language Go transforms</h4>
<p>Go currently does not support creating cross-language transforms, only
using cross-language
transforms from other languages; see more at <a
href="https://issues.apache.org/jira/browse/BEAM-9923";>BEAM-9923</a>.</p>
+<h4 id="1314-selecting-a-urn-for-cross-language-transforms">13.1.4.
Selecting a URN for Cross-language Transforms</h4>
+<p>Developing a cross-language transform involves defining a URN for
registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is
optional but it will ensure that your transform
+will not run into conflicts when registering in an expansion service along
with transforms developed by other developers.</p>
+<h5 id="schema">Schema</h5>
+<p>A URN should consist of the following components:</p>
+<ul>
+<li>ns-id: A namespace identifier. Default recommendation is
<code>beam:transform</code>.</li>
+<li>org-identifier: Identifies the organization where the transform was
defined. Transforms defined in Apache Beam use
<code>org.apache.beam</code> for this.</li>
+<li>functionality-identifier - Identifies the functionality of the
cross-language transform.</li>
+<li>version - a version number for the transform</li>
+</ul>
+<p>We provide the schema from the URN convention in <a
href="https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form";>augmented
Backus–Naur</a> form.
+Keywords in upper case are from the <a
href="https://datatracker.ietf.org/doc/html/rfc8141";>URN spec</a>.</p>
+<div class="snippet">
+<div class="notebook-skip code-snippet without_switcher">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<pre><code>transform-urn = ns-id “:” org-identifier “:”
functionality-identifier “:” version
+ns-id = (“beam” / NID) “:” “transform”
+id-char = ALPHA / DIGIT / &#34;-&#34; / &#34;.&#34; /
&#34;_&#34; / &#34;~&#34; ; A subset of characters allowed in a
URN
+org-identifier = 1*id-char
+functionality-identifier = 1*id-char
+version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’</code></pre>
+</div>
+</div>
+<h5 id="examples">Examples</h5>
+<p>Below we’ve given some example transform classes and corresponding URNs
to be used.</p>
+<ul>
+<li>A transform offered with Apache Beam that writes Parquet files.
+<ul>
+<li><code>beam:transform:org.apache.beam:parquet_write:v1</code></li>
+</ul>
+</li>
+<li>A transform offered with Apache Beam that reads from Kafka with
metadata.
+<ul>
+<li><code>beam:transform:org.apache.beam:kafka_read_with_metadata:v1</code></li>
+</ul>
+</li>
+<li>A transform developed by organization abc.org that reads from data
store MyDatastore.
+<ul>
+<li><code>beam:transform:org.abc:mydatastore_read:v1</code></li>
+</ul>
+</li>
+</ul>
<h3 id="use-x-lang-transforms">13.2. Using cross-language transforms</h3>
<p>Depending on the SDK language of the pipeline, you can use a high-level
SDK-wrapper class, or a low-level transform class to access a cross-language
transform.</p>
<h4 id="1321-using-cross-language-transforms-in-a-java-pipeline">13.2.1.
Using cross-language transforms in a Java pipeline</h4>
@@ -14469,6 +14624,9 @@ limitations under the License.
-->
<h1 id="cross-language-transforms">Cross-language transforms</h1>
<p>With the samples on this page we will demonstrate how to create and
leverage cross-language pipelines.</p>
+<blockquote>
+<p><strong>Note:</strong> Please see the <a
href="https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines";>Beam
Programming Guide</a> for full documentation on cross-language
transforms.</p>
+</blockquote>
<p>The goal of a cross-language pipeline is to incorporate transforms from
one SDK (e.g. the Python SDK) into a pipeline written using another SDK (e.g.
the Java SDK). This enables having already developed transforms (e.g. ML
transforms in Python) and libraries (e.g. the vast library of IOs in Java), and
strengths of certain languages at your disposal in whichever language you are
more comfortable authoring pipelines while vastly expanding your toolkit in
given language.</p>
<p>In this section we will cover a specific use-case: incorporating a
Python transform that does inference on a model but is part of a larger Java
pipeline. The section is broken down into 2 parts:</p>
<ol>
diff --git
a/website/generated-content/documentation/patterns/cross-language/index.html
b/website/generated-content/documentation/patterns/cross-language/index.html
index 4224a39..7aa7354 100644
--- a/website/generated-content/documentation/patterns/cross-language/index.html
+++ b/website/generated-content/documentation/patterns/cross-language/index.html
@@ -18,7 +18,7 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you
looking for?");}
function endSearch(){var
search=document.querySelector(".searchBar");search.classList.add("disappear");var
icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
+function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
<span class=kd>private</span> <span class=kd>static</span> <span
class=kd>final</span> <span class=n>String</span> <span class=n>URN</span>
<span class=o>=</span> <span
class=s>"beam:transforms:xlang:pythontransform"</span><span
class=o>;</span>
<span class=kd>private</span> <span class=kd>static</span> <span
class=n>String</span> <span class=n>expansionAddress</span><span
class=o>;</span>
@@ -96,7 +96,7 @@ function
openMenu(){addPlaceholder();blockScroll();}</script><div class="clearfi
<span class=n>logging</span><span class=o>.</span><span
class=n>getLogger</span><span class=p>()</span><span class=o>.</span><span
class=n>setLevel</span><span class=p>(</span><span class=n>logging</span><span
class=o>.</span><span class=n>INFO</span><span class=p>)</span>
<span class=n>main</span><span class=p>(</span><span class=n>sys</span><span
class=o>.</span><span class=n>argv</span><span
class=p>)</span></code></pre></div></div></div><h2
id=how-to-run-the-cross-language-pipeline>How to run the cross-language
pipeline?</h2><p>In this section, the steps to run a cross-language pipeline
are set out:</p><ol><li><p>Start the <strong>expansion service</strong> with
your Python transforms: <code>python expansion_service.py -p
9097</code></p></li><li><p>S [...]
<code>./gradlew :runners:spark:job-server:runShadow</code></li><li>Using the
pre-build Docker container:
-<code>docker run -net=host
apache/beam_spark_job_server</code></li></ul></li><li><p><strong>Run
pipeline</strong>: <code>mvn exec:java -Dexec.mainClass=CrossLanguagePipeline \
-Pportable-runner \ -Dexec.args=" \ --runner=PortableRunner \
--jobEndpoint=localhost:8099 \ --useExternal=true \
--expansionServiceURL=localhost:9097 \
--experiments=beam_fn_api"</code></p></li></ol><div class=feedback><p
class=update>Last updated on 2021/10/04</p><h3>Have you found everything you
were looking for [...]
+<code>docker run -net=host
apache/beam_spark_job_server</code></li></ul></li><li><p><strong>Run
pipeline</strong>: <code>mvn exec:java -Dexec.mainClass=CrossLanguagePipeline \
-Pportable-runner \ -Dexec.args=" \ --runner=PortableRunner \
--jobEndpoint=localhost:8099 \ --useExternal=true \
--expansionServiceURL=localhost:9097 \
--experiments=beam_fn_api"</code></p></li></ol><div class=feedback><p
class=update>Last updated on 2021/11/12</p><h3>Have you found everything you
were looking for [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index db188c3..44f84a5 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -18,7 +18,7 @@
function addPlaceholder(){$('input:text').attr('placeholder',"What are you
looking for?");}
function endSearch(){var
search=document.querySelector(".searchBar");search.classList.add("disappear");var
icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
-function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
+function openMenu(){addPlaceholder();blockScroll();}</script><div
class="clearfix container-main-content"><div class="section-nav closed"
data-offset-top=90 data-offset-bottom=500><span class="section-nav-back
glyphicon glyphicon-menu-left"></span><nav><ul class=section-nav-list
data-section-nav><li><span
class=section-nav-list-main-title>Documentation</span></li><li><a
href=/documentation>Using the Documentation</a></li><li
class=section-nav-item--collapsible><span class=section-nav-lis [...]
Beam SDKs to create data processing pipelines. It provides guidance for using
the Beam SDK classes to build and test your pipeline. It is not intended as an
exhaustive reference, but as a language-agnostic, high-level guide to
@@ -4127,7 +4127,51 @@ use case.</p><div class="language-java snippet"><div
class="notebook-skip code-s
<span class=c1># Register callback function for this bundle that performs
the side</span>
<span class=c1># effect.</span>
- <span class=n>bundle_finalizer</span><span class=o>.</span><span
class=n>register</span><span class=p>(</span><span
class=n>my_callback_func</span><span
class=p>)</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><spa [...]
+ <span class=n>bundle_finalizer</span><span class=o>.</span><span
class=n>register</span><span class=p>(</span><span
class=n>my_callback_func</span><span
class=p>)</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><spa [...]
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig
dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}</code></pre></div></div><p>To use a Java class that conforms to the above
requirement from a Python SDK pipeline you may do the
following.</p><ul><li>Step 1: create an allowlist file in the <em>yaml</em>
format that describes the Java transform classes and methods that will be
directly accessed from Python.</li><li>Step 2: start an Expansion Service with
the <code>javaClassLookupAllowlistFile</code> option passing path to the
allowlist defined in Step 1 as the value.</li><li>Step 3: Us [...]
+access Java transforms defined in the allowlist from the Python
side.</li></ul><p>Starting with Beam 2.35.0, Step 1 and 2 may be skipped as
described in corresponding sections below.</p><h5 id=step-1>Step 1</h5><p>To
use this Java transform from Python, you may define an allowlist file in the
<em>yaml</em> format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be
used from the Python side.</p><p>Starting with Beam 2.35.0, you have the option
to specify <code>*</code> to the <code>javaClassLookupAllowlistFile</code>
option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the expansion
service may be accessed through the API.</p><div class=snippet><div
class="notebook-skip code-snippet without_switcher"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><pre><code>version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig</code></pre></div></div><h5 id=step-2>Step
2</h5><p>The allowlist is provided as an argument when starting up the Java
expansion service. For example, the expansion service can be started
+as a local Java process using the following command.</p><div
class=snippet><div class="notebook-skip code-snippet without_switcher"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>java
-jar <jar file> <port> --javaClassLookupAllowlistFile=<path to
the allowlist file></code></pre></div></div><p>Starting with Beam 2.35.0,
Beam ``JavaExternalTransform<code>API will automatical [...]
+if an expansion service address was not provided.</p><h5 id=step-3>Step
3</h5><p>You can directly use the Java class from your Python pipeline using a
stub transform created using the <code>JavaExternalTransform</code> API. This
API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to
configure the class.</p><p>Constructor and method parameter types are mapped
between Python and Java using a Beam Schema. The Schema is auto-generated using
the object types
+provided on the Python side. If the Java class constructor method or builder
method accepts any complex object types, make sure that the Beam Schema
+for these objects is registered and available for the Java expansion service.
If a schema has not been registered, the Java expansion service will
+try to register a schema using <a
href=https://beam.apache.org/documentation/programming-guide/#creating-schemas>JavaFieldSchema</a>.
In Python arbitrary objects
+can be represented using <code>NamedTuple</code>s which will be represented as
Beam Rows in the Schema. See below for a Python stub transform that represents
the above
+mentioned Java transform.</p><div class=snippet><div class="notebook-skip
code-snippet without_switcher"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><pre><code>JavaDataGeneratorConfig =
typing.NamedTuple(
+'JavaDataGeneratorConfig', [('prefix', str),
('length', int), ('suffix', str)])
+data_config = JavaDataGeneratorConfig(prefix='start', length=20,
suffix='end')
+
+java_transform = JavaExternalTransform(
+'my.beam.transforms.JavaDataGenerator',
expansion_service='localhost:<port>').create(numpy.int32(100)).withJavaDataGeneratorConfig(data_config)</code></pre></div></div><p>This
transform can be used in a Python pipeline along with other Python
transforms.</p><h5
id=13112-full-api-for-making-existing-java-transforms-available-to-other-sdks>13.1.1.2
Full API for Making Existing Java Transforms Available to Other SDKs</h5><p>To
make your Apache Beam Java SDK transform p [...]
abstract static class Builder<K, V>
implements ExternalTransformBuilder<External.Configuration, PBegin,
PCollection<KV<K, V>>> {
abstract Builder<K, V> setConsumerConfig(Map<String, Object>
config);
@@ -4201,7 +4245,15 @@ def from_runner_api_parameter(
</code></pre></div></div></li></ol><p><strong>Starting the expansion
service</strong></p><p>An expansion service can be used with multiple
transforms in the same pipeline. Python has a default expansion service
included and available in the Apache Beam Python SDK for you to use with your
Python transforms. You are free to write your own expansion service, but that
is generally not needed, so it is not covered in this section.</p><p>Perform
the following steps to start up the default [...]
</code></pre></div></div></li><li><p>Import any modules that contain
transforms to be made available using the expansion service.</p><div
class=snippet><div class="notebook-skip code-snippet without_switcher"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><pre><code>$
python -m apache_beam.runners.portability.expansion_service_test -p
$PORT_FOR_EXPANSION_SERVICE
</code></pre></div></div></li><li><p>This expansion service is not ready
to serve up transforms on the address
<code>localhost:$PORT_FOR_EXPANSION_SERVICE</code>.</p></li></ol><p><strong>Including
dependencies</strong></p><p>Currently Python external transforms are limited
to dependencies available in core Beam SDK Harness.</p><h4
id=1313-creating-cross-language-go-transforms>13.1.3. Creating cross-language
Go transforms</h4><p>Go currently does not support creating cross-language tr
[...]
-transforms from other languages; see more at <a
href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h3
id=use-x-lang-transforms>13.2. Using cross-language transforms</h3><p>Depending
on the SDK language of the pipeline, you can use a high-level SDK-wrapper
class, or a low-level transform class to access a cross-language
transform.</p><h4
id=1321-using-cross-language-transforms-in-a-java-pipeline>13.2.1. Using
cross-language transforms in a Java pipeline</h4><p>Current [...]
+transforms from other languages; see more at <a
href=https://issues.apache.org/jira/browse/BEAM-9923>BEAM-9923</a>.</p><h4
id=1314-selecting-a-urn-for-cross-language-transforms>13.1.4. Selecting a URN
for Cross-language Transforms</h4><p>Developing a cross-language transform
involves defining a URN for registering the transform with an expansion
service. In this section
+we provide a convention for defining such URNs. Following this convention is
optional but it will ensure that your transform
+will not run into conflicts when registering in an expansion service along
with transforms developed by other developers.</p><h5 id=schema>Schema</h5><p>A
URN should consist of the following components:</p><ul><li>ns-id: A namespace
identifier. Default recommendation is
<code>beam:transform</code>.</li><li>org-identifier: Identifies the
organization where the transform was defined. Transforms defined in Apache Beam
use <code>org.apache.beam</code> for this.</li><li>functionality-identifi [...]
+Keywords in upper case are from the <a
href=https://datatracker.ietf.org/doc/html/rfc8141>URN spec</a>.</p><div
class=snippet><div class="notebook-skip code-snippet without_switcher"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><pre><code>transform-urn = ns-id “:”
org-identifier “:” functionality-identifier “:” version
+ns-id = (“beam” / NID) “:” “transform”
+id-char = ALPHA / DIGIT / "-" / "." / "_" /
"~" ; A subset of characters allowed in a URN
+org-identifier = 1*id-char
+functionality-identifier = 1*id-char
+version = “v” 1*(DIGIT / “.”) ; For example,
‘v1.2’</code></pre></div></div><h5 id=examples>Examples</h5><p>Below we’ve
given some example transform classes and corresponding URNs to be
used.</p><ul><li>A transform offered with Apache Beam that writes Parquet
files.<ul><li><code>beam:transform:org.apache.beam:parquet_write:v1</code></li></ul></li><li>A
transform offered with Apache Beam that reads from Kafka with
metadata.<ul><li><code>beam:transform:org.apache.beam:kafka_read_with_meta [...]
kafka_records = (
pipeline
@@ -4255,7 +4307,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr,
beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/11/10</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/11/12</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index f030b49..293aee1 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.34.0/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/blog/g
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.34.0/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-11-11T11:07:06-08:00</lastmod></url><url><loc>/blog/g
[...]
\ No newline at end of file
