This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d0596f6 Publishing website 2021/10/13 06:06:04 at commit 71d7213
d0596f6 is described below
commit d0596f6f645e53f75a02bf54291dd3807e323c46
Author: jenkins <bui...@apache.org>
AuthorDate: Wed Oct 13 06:06:04 2021 +0000
Publishing website 2021/10/13 06:06:04 at commit 71d7213
---
website/generated-content/documentation/index.xml | 121 ++++++++++++++++++++-
.../documentation/programming-guide/index.html | 36 +++---
website/generated-content/sitemap.xml | 2 +-
3 files changed, 138 insertions(+), 21 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index e6c7c3d..73601a9 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -9600,12 +9600,17 @@ state. Merging windows are not currently supported when
using state and timers.&
care must be taken to remember that the elements in input PCollection have no
guaranteed order and to ensure that the
program logic is resilient to this. Unit tests written using the DirectRunner
will shuffle the order of element
processing, and are recommended to test for correctness.</p>
-<p class="language-java">In Java DoFn declares states to be accessed by
creating final <code>StateSpec</code> member variables representing each
state. Each
+<p class="language-java">In Java, DoFn declares states to be accessed by
creating final <code>StateSpec</code> member variables representing each
state. Each
state must be named using the <code>StateId</code> annotation; this name
is unique to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple
state variables.</p>
-<p class="language-py">In Python DoFn declares states to be accessed by
creating <code>StateSpec</code> class member variables representing each
state. Each
+<p class="language-py">In Python, DoFn declares states to be accessed by
creating <code>StateSpec</code> class member variables representing each
state. Each
<code>StateSpec</code> is initialized with a name, this name is unique
to a ParDo in the graph and has no relation
to other nodes in the graph. A <code>DoFn</code> can declare multiple
state variables.</p>
+<span class="language-go">
+<blockquote>
+<p><strong>Note:</strong> The Beam SDK for Go does not yet support a
State and Timer API. See <a
href="https://issues.apache.org/jira/browse/BEAM-10660";>BEAM-10660</a> to
contribute.</p>
+</blockquote>
+</span>
<h3 id="types-of-state">11.1. Types of state</h3>
<p>Beam provides several types of state:</p>
<h4 id="valuestate">ValueState</h4>
@@ -9661,6 +9666,14 @@ accumulates the number of elements seen.</p>
<span class="o">|</span> <span class="s1">&#39;state
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">ReadModifyWriteStateDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="combiningstate">CombiningState</h4>
<p><code>CombiningState</code> allows you to create a state object
that is updated using a Beam combiner. For example, the previous
<code>ValueState</code> example could be rewritten to use
<code>CombiningState</code></p>
@@ -9692,6 +9705,14 @@ accumulates the number of elements seen.</p>
<span class="o">|</span> <span class="s1">&#39;Combine state
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">CombiningStateDofn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="bagstate">BagState</h4>
<p>A common use case for state is to accumulate multiple elements.
<code>BagState</code> allows for accumulating an unordered set
of elements. This allows for addition of elements to the collection without
requiring the reading of the entire
@@ -9737,6 +9758,14 @@ bags larger than available memory.</p>
<span class="o">|</span> <span class="s1">&#39;Bag state
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">BagStateDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="deferred-state-reads">11.2. Deferred state reads</h3>
<p>When a <code>DoFn</code> contains multiple state specifications,
reading each one in order can be slow. Calling the <code>read()</code>
function
on a state can cause the runner to perform a blocking read. Performing
multiple blocking reads in sequence adds latency
@@ -9771,6 +9800,14 @@ runner can prefetch all of the states necessary. For
example:</p>
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">This</span> <span
class="ow">is</span> <span class="ow">not</span> <span
class="n">supported</span> <span class="n">yet</span><span
class="p">,</span> <span class="n">see</span> <span
class="n">BEAM</span><span class="o">-</span><span
class="mf">11506.</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<p>If however there are code paths in which the states are not fetched,
then annotating with @AlwaysFetched will add
unnecessary fetching for those paths. In this case, the readLater method
allows the runner to know that the state will
be read in the future, allowing multiple state reads to be batched
together.</p>
@@ -9857,6 +9894,14 @@ allows for event-time aggregations.</p>
<span class="o">|</span> <span class="s1">&#39;EventTime timer
pardo&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">EventTimerDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="processing-time-timers">11.3.2. Processing-time timers</h4>
<p>Processing-time timers fire when the real wall-clock time passes. This
is often used to create larger batches of data
before processing. It can also be used to schedule events that should occur at
a specific time. Just like with
@@ -9905,6 +9950,14 @@ to the current time. In Java, the
<code>Timer.offset</code> and <code>T
<span class="o">|</span> <span class="s1">&#39;ProcessingTime
timer pardo&#39;</span> <span class="o">&gt;&gt;</span>
<span class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">ProcessingTimerDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="dynamic-timer-tags">11.3.3. Dynamic timer tags</h4>
<p>Beam also supports dynamically setting a timer tag using
<code>TimerMap</code> in the Java SDK. This allows for setting multiple
different timers
in a <code>DoFn</code> and allowing for the timer tags to be dynamically
chosen - e.g. based on data in the input elements. A
@@ -9962,6 +10015,14 @@ tag is an empty string if not specified.</p>
<span class="o">|</span> <span class="s1">&#39;ProcessingTime
timer pardo&#39;</span> <span class="o">&gt;&gt;</span>
<span class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">TimerDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h4 id="timer-output-timestamps">11.3.4. Timer output timestamps</h4>
<p>By default, event-time timers will hold the output watermark of the
<code>ParDo</code> to the timestamp of the timer. This means
that if a timer is set to 12pm, any windowed aggregations or event-time timers
later in the pipeline graph that finish
@@ -10071,6 +10132,14 @@ past the timestamp of the minimum element. The
following code demonstrates this.
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">Timer</span> <span
class="n">output</span> <span class="n">timestamps</span> <span
class="ow">is</span> <span class="ow">not</span> <span
class="n">yet</span> <span class="n">supported</span> <span
class="ow">in</span> <span class="n">Python</span> <span
class="n">SDK</span><span class="o">.</span> <span class="n [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="garbage-collecting-state">11.4. Garbage collecting state</h3>
<p>Per-key state needs to be garbage collected, or eventually the
increasing size of state may negatively impact
performance. There are two common strategies for garbage collecting
state.</p>
@@ -10116,6 +10185,14 @@ garbage-collection strategy.</p>
<span class="o">|</span> <span
class="s1">&#39;DoFn&#39;</span> <span
class="o">&gt;&gt;</span> <span class="n">beam</span><span
class="o">.</span><span class="n">ParDo</span><span
class="p">(</span><span class="n">StateDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<p>This <code>ParDo</code> stores state per day. Once the pipeline is
done processing data for a given day, all the state for that
day is garbage collected.</p>
<h5 id="using-timers-for-garbage-collection">11.4.1. <strong>Using
timers For garbage collection</strong></h5>
@@ -10191,6 +10268,14 @@ This can be done by updating a timer that garbage
collects state. For example<
<span class="o">|</span> <span class="s1">&#39;User
DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span
class="n">beam</span><span class="o">.</span><span
class="n">ParDo</span><span class="p">(</span><span
class="n">UserDoFn</span><span
class="p">()))</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10660.</span></code></pre></div>
+</div>
+</div>
<h3 id="state-timers-examples">11.5. State and timers examples</h3>
<p>Following are some example uses of state and timers</p>
<h4 id="joining-clicks-and-views">11.5.1. Joining clicks and views</h4>
@@ -10749,6 +10834,14 @@ resource utilization.</p>
<span class="k">return</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">11104.</span></code></pre></div>
+</div>
+</div>
<h3 id="runner-initiated-split">12.4. Runner-initiated split</h3>
<p>A runner at any time may attempt to split a restriction while it is
being processed. This allows the
runner to either pause processing of the restriction so that other work may be
done (common for
@@ -10961,6 +11054,14 @@ watermark estimator implementation. You can also
provide your own watermark esti
<span class="n">watermark_estimator</span><span
class="o">.</span><span class="n">current_watermark</span><span
class="p">()</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">11105.</span></code></pre></div>
+</div>
+</div>
<h3 id="truncating-during-drain">12.6. Truncating during drain</h3>
<p>Runners which support draining pipelines need the ability to drain SDFs;
otherwise, the
pipeline may never stop. By default, bounded restrictions process the
remainder of the restriction while
@@ -10997,6 +11098,14 @@ provider.</p>
<span class="k">return</span> <span
class="n">restriction</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">11106.</span></code></pre></div>
+</div>
+</div>
<h3 id="bundle-finalization">12.7. Bundle finalization</h3>
<p>Bundle finalization enables a <code>DoFn</code> to perform side
effects by registering a callback.
The callback is invoked once the runner has acknowledged that it has durably
persisted the output.
@@ -11033,6 +11142,14 @@ use case.</p>
<span class="n">bundle_finalizer</span><span
class="o">.</span><span class="n">register</span><span
class="p">(</span><span class="n">my_callback_func</span><span
class="p">)</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="nx">This</span> <span
class="nx">is</span> <span class="nx">not</span> <span
class="nx">supported</span> <span class="nx">yet</span><span
class="p">,</span> <span class="nx">see</span> <span
class="nx">BEAM</span><span class="o">-</span><span
class="mf">10976.</span></code></pre></div>
+</div>
+</div>
<h2 id="multi-language-pipelines">13. Multi-language pipelines</h2>
<p>Beam allows you to combine transforms written in any supported SDK
language (currently, Java and Python) and use them in one multi-language
pipeline. This capability makes it easy to provide new functionality
simultaneously in different Apache Beam SDKs through a single cross-language
transform. For example, the <a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py";>Apache
Kafka connector</a> and <a href="https://github.com/apache/beam/blo [...]
<p>Pipelines that use transforms from more than one SDK-language are known
as <em>multi-language pipelines</em>.</p>
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 81b826f..224974a 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -3090,11 +3090,11 @@ the stateful operator. For example, using a combiner to
preaggregate data, and t
state. Merging windows are not currently supported when using state and
timers.</p><p>Sometimes stateful processing is used to implement state-machine
style processing inside a <code>DoFn</code>. When doing this,
care must be taken to remember that the elements in input PCollection have no
guaranteed order and to ensure that the
program logic is resilient to this. Unit tests written using the DirectRunner
will shuffle the order of element
-processing, and are recommended to test for correctness.</p><p
class=language-java>In Java DoFn declares states to be accessed by creating
final <code>StateSpec</code> member variables representing each state. Each
+processing, and are recommended to test for correctness.</p><p
class=language-java>In Java, DoFn declares states to be accessed by creating
final <code>StateSpec</code> member variables representing each state. Each
state must be named using the <code>StateId</code> annotation; this name is
unique to a ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><p class=language-py>In Python DoFn declares states to be
accessed by creating <code>StateSpec</code> class member variables representing
each state. Each
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><p class=language-py>In Python, DoFn declares states to be
accessed by creating <code>StateSpec</code> class member variables representing
each state. Each
<code>StateSpec</code> is initialized with a name, this name is unique to a
ParDo in the graph and has no relation
-to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><h3 id=types-of-state>11.1. Types of state</h3><p>Beam provides
several types of state:</p><h4 id=valuestate>ValueState</h4><p>A ValueState is
a scalar state value. For each key in the input, a ValueState will store a
typed value that can be
+to other nodes in the graph. A <code>DoFn</code> can declare multiple state
variables.</p><span class=language-go><blockquote><p><strong>Note:</strong> The
Beam SDK for Go does not yet support a State and Timer API. See <a
href=https://issues.apache.org/jira/browse/BEAM-10660>BEAM-10660</a> to
contribute.</p></blockquote></span><h3 id=types-of-state>11.1. Types of
state</h3><p>Beam provides several types of state:</p><h4
id=valuestate>ValueState</h4><p>A ValueState is a scalar state valu [...]
read and modified inside the DoFn’s <code>@ProcessElement</code> or
<code>@OnTimer</code> methods. If the type of the ValueState has a coder
registered, then Beam will automatically infer the coder for the state value.
Otherwise, a coder can be explicitly
specified when creating the ValueState. For example, the following ParDo
creates a single state variable that
@@ -3122,7 +3122,7 @@ accumulates the number of elements seen.</p><p>Note:
<code>ValueState</code> is
<span class=n>state</span><span class=o>.</span><span
class=n>write</span><span class=p>(</span><span
class=n>current_value</span><span class=o>+</span><span class=mi>1</span><span
class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ReadModifyWriteStateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h4
id=combiningstate>CombiningState</h4><p><code>CombiningState</code> allows you
to create a state object that is updated using a Beam combiner. For example,
the previous
+ <span class=o>|</span> <span class=s1>'state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ReadModifyWriteStateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/co [...]
<code>ValueState</code> example could be rewritten to use
<code>CombiningState</code></p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span cl
[...]
<span class=n>perUser</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span
class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
<span class=nd>@StateId</span><span class=o>(</span><span
class=s>"state"</span><span class=o>)</span> <span
class=kd>private</span> <span class=kd>final</span> <span
class=n>StateSpec</span><span class=o><</span><span
class=n>CombiningState</span><span class=o><</span><span
class=n>Integer</span><span class=o>,</span> <span class=kt>int</span><span
class=o>[],</span> <span class=n>Integer</span><span class=o>>></span>
<span class=n>numElements</span> <span class=o> [...]
@@ -3138,7 +3138,7 @@ accumulates the number of elements seen.</p><p>Note:
<code>ValueState</code> is
<span class=n>state</span><span class=o>.</span><span
class=n>add</span><span class=p>(</span><span class=mi>1</span><span
class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Combine state
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>CombiningStateDofn</span><span
class=p>()))</span></code></pre></div></div></div><h4
id=bagstate>BagState</h4><p>A common use case for state is to accumulate
multiple elements. <code>BagState</code> allows for accumulating an unordered
set
+ <span class=o>|</span> <span class=s1>'Combine state
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>CombiningStateDofn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/ [...]
of elements. This allows for addition of elements to the collection without
requiring the reading of the entire
collection first, which is an efficiency gain. In addition, runners that
support paged reads can allow individual
bags larger than available memory.</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span
class=n>String</span><span class=o>,</span> <span cla [...]
@@ -3168,7 +3168,7 @@ bags larger than available memory.</p><div
class="language-java snippet"><div cl
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'Bag state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>BagStateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h3
id=deferred-state-reads>11.2. Deferred state reads</h3><p>When a
<code>DoFn</code> contains multiple state specifications, reading each one in
order can be slow. Calling the <code>read()</code> function
+ <span class=o>|</span> <span class=s1>'Bag state pardo'</span>
<span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>BagStateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon. [...]
on a state can cause the runner to perform a blocking read. Performing
multiple blocking reads in sequence adds latency
to element processing. If you know that a state will always be read, you can
annotate it as @AlwaysFetched, and then the
runner can prefetch all of the states necessary. For example:</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>PCollection</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span [...]
@@ -3185,7 +3185,7 @@ runner can prefetch all of the states necessary. For
example:</p><div class="lan
<span class=n>state2</span><span class=o>.</span><span
class=na>read</span><span class=o>();</span>
<span class=n>state3</span><span class=o>.</span><span
class=na>read</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=n>This</span> <span class=ow>is</span> <span class=ow>not</span> <span
class=n>supported</span> <span class=n>yet</span><span class=p>,</spa [...]
+<span class=o>}));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=n>This</span> <span class=ow>is</span> <span class=ow>not</span> <span
class=n>supported</span> <span class=n>yet</span><span class=p>,</spa [...]
unnecessary fetching for those paths. In this case, the readLater method
allows the runner to know that the state will
be read in the future, allowing multiple state reads to be batched
together.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span
class=n>Str [...]
<span class=n>perUser</span><span class=o>.</span><span
class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=k>new</span> <span class=n>DoFn</span><span class=o><</span><span
class=n>KV</span><span class=o><</span><span class=n>String</span><span
class=o>,</span> <span class=n>ValueT</span><span class=o>>,</span> <span
class=n>OutputT</span><span class=o>>()</span> <span class=o>{</span>
@@ -3249,7 +3249,7 @@ allows for event-time aggregations.</p><div
class="language-java snippet"><div c
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'EventTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>EventTimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h4
id=processing-time-timers>11.3.2. Processing-time timers</h4><p>Processing-time
timers fire when the real wall-clock time passes. This is often used to create
larger batches of data
+ <span class=o>|</span> <span class=s1>'EventTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>EventTimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/co [...]
before processing. It can also be used to schedule events that should occur at
a specific time. Just like with
event-time timers, processing-time timers are per key - each key has a
separate copy of the timer.</p><p>While processing-time timers can be set to an
absolute timestamp, it is very common to set them to an offset relative
to the current time. In Java, the <code>Timer.offset</code> and
<code>Timer.setRelative</code> methods can be used to accomplish this.</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>PCollection</span><span class=o><</span><span c
[...]
@@ -3283,7 +3283,7 @@ to the current time. In Java, the
<code>Timer.offset</code> and <code>Timer.setR
<span class=n>state</span><span class=o>.</span><span
class=n>clear</span><span class=p>()</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ProcessingTimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h4
id=dynamic-timer-tags>11.3.3. Dynamic timer tags</h4><p>Beam also supports
dynamically setting a timer tag using <code>TimerMap</code> in the Java SDK.
This allows for setting mu [...]
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>ProcessingTimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src= [...]
in a <code>DoFn</code> and allowing for the timer tags to be dynamically
chosen - e.g. based on data in the input elements. A
timer with a specific tag can only be set to a single timestamp, so setting
the timer again has the effect of
overwriting the previous expiration time for the timer with that tag. Each
<code>TimerMap</code> is identified with a timer family
@@ -3326,7 +3326,7 @@ tag is an empty string if not specified.</p><div
class="language-java snippet"><
<span class=k>yield</span> <span class=p>(</span><span
class=n>timer_tag</span><span class=p>,</span> <span
class=s1>'fired'</span><span class=p>)</span>
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>TimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h4
id=timer-output-timestamps>11.3.4. Timer output timestamps</h4><p>By default,
event-time timers will hold the output watermark of the <code>ParDo</code> to
the timestamp of the timer. This means
+ <span class=o>|</span> <span class=s1>'ProcessingTime timer
pardo'</span> <span class=o>>></span> <span class=n>beam</span><span
class=o>.</span><span class=n>ParDo</span><span class=p>(</span><span
class=n>TimerDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/co [...]
that if a timer is set to 12pm, any windowed aggregations or event-time timers
later in the pipeline graph that finish
after 12pm will not expire. The timestamp of the timer is also the default
output timestamp for the timer callback. This
means that any elements output from the onTimer method will have a timestamp
equal to the timestamp of the timer firing.
@@ -3413,7 +3413,7 @@ past the timestamp of the minimum element. The following
code demonstrates this.
<span class=c1>// Note that the timer has now fired.
</span><span class=c1></span> <span class=n>timerTimestamp</span><span
class=o>.</span><span class=na>clear</span><span class=o>();</span>
<span class=o>}</span>
-<span class=o>}));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=n>Timer</span> <span class=n>output</span> <span
class=n>timestamps</span> <span class=ow>is</span> <span class=ow>not</span>
<span class=n> [...]
+<span class=o>}));</span></code></pre></div></div></div><div
class="language-py snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-py data-lang=py><span
class=n>Timer</span> <span class=n>output</span> <span
class=n>timestamps</span> <span class=ow>is</span> <span class=ow>not</span>
<span class=n> [...]
performance. There are two common strategies for garbage collecting
state.</p><h5 id=using-windows-for-garbage-collection>11.4.1. <strong>Using
windows for garbage collection</strong></h5><p>All state and timers for a key
is scoped to the window it is in. This means that depending on the timestamp of
the
input element the ParDo will see different values for the state depending on
the window that element falls into. In
addition, once the input watermark passes the end of the window, the runner
should garbage collect all state for that
@@ -3439,7 +3439,7 @@ garbage-collection strategy.</p><p>For example, given the
following:</p><div cla
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
<span class=o>|</span> <span class=s1>'Windowing'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>WindowInto</span><span class=p>(</span><span
class=n>FixedWindows</span><span class=p>(</span><span class=mi>60</span> <span
class=o>*</span> <span class=mi>60</span> <span class=o>*</span> <span
class=mi>24</span><span class=p>))</span>
- <span class=o>|</span> <span class=s1>'DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><p>This <code>ParDo</code>
stores state per day. Once the pipeline is done processing data for a given
day, all the state for that
+ <span class=o>|</span> <span class=s1>'DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>StateDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div c [...]
day is garbage collected.</p><h5
id=using-timers-for-garbage-collection>11.4.1. <strong>Using timers For garbage
collection</strong></h5><p>In some cases, it is difficult to find a windowing
strategy that models the desired garbage-collection strategy. For
example, a common desire is to garbage collect state for a key once no
activity has been seen on the key for some time.
This can be done by updating a timer that garbage collects state. For
example</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>KV</span><span class=o><</span><span
class=n>St [...]
@@ -3505,7 +3505,7 @@ This can be done by updating a timer that garbage
collects state. For example</p
<span class=n>_</span> <span class=o>=</span> <span class=p>(</span><span
class=n>p</span> <span class=o>|</span> <span class=s1>'Read per
user'</span> <span class=o>>></span> <span
class=n>ReadPerUser</span><span class=p>()</span>
- <span class=o>|</span> <span class=s1>'User DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span
class=p>()))</span></code></pre></div></div></div><h3
id=state-timers-examples>11.5. State and timers examples</h3><p>Following are
some example uses of state and timers</p><h4
id=joining-clicks-and-views>11.5.1. Joining clicks and views</h4><p>In this
example [...]
+ <span class=o>|</span> <span class=s1>'User DoFn'</span> <span
class=o>>></span> <span class=n>beam</span><span class=o>.</span><span
class=n>ParDo</span><span class=p>(</span><span class=n>UserDoFn</span><span
class=p>()))</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><d [...]
a stream of views, representing suggested product links displayed to the user
on the home page, and a stream of
clicks, representing actual user clicks on these links. The goal of the
pipeline is to join click events with view
events, outputting a new joined event that contains information from both
events. Each link has a unique identifier
@@ -3906,7 +3906,7 @@ resource utilization.</p><div class="language-java
snippet"><div class="notebook
<span class=k>except</span> <span class=n>TimeoutError</span><span
class=p>:</span>
<span class=c1># Set a longer delay in case we are being
throttled.</span>
<span class=n>restriction_tracker</span><span class=o>.</span><span
class=n>defer_remainder</span><span class=p>(</span><span
class=n>timestamp</span><span class=o>.</span><span
class=n>Duration</span><span class=p>(</span><span class=n>seconds</span><span
class=o>=</span><span class=mi>60</span><span class=p>))</span>
- <span class=k>return</span></code></pre></div></div></div><h3
id=runner-initiated-split>12.4. Runner-initiated split</h3><p>A runner at any
time may attempt to split a restriction while it is being processed. This
allows the
+ <span class=k>return</span></code></pre></div></div></div><div
class="language-go snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-go data-lang=go><span
class=nx>This</span> <span class=nx>is</span> <span class=nx>not</span> <span
class=nx>supported</span> <span class=nx>yet</span><span c [...]
runner to either pause processing of the restriction so that other work may be
done (common for
unbounded restrictions to limit the amount of output and/or improve latency)
or split the restriction
into two pieces, increasing the available parallelism within the system.
Different runners (e.g.,
@@ -4080,7 +4080,7 @@ watermark estimator implementation. You can also provide
your own watermark esti
<span class=n>watermark_estimator</span><span class=o>=</span><span
class=n>beam</span><span class=o>.</span><span class=n>DoFn</span><span
class=o>.</span><span class=n>WatermarkEstimatorParam</span><span
class=p>(</span>
<span class=n>MyWatermarkEstimatorProvider</span><span
class=p>())):</span>
<span class=c1># The current watermark can be inspected.</span>
- <span class=n>watermark_estimator</span><span class=o>.</span><span
class=n>current_watermark</span><span
class=p>()</span></code></pre></div></div></div><h3
id=truncating-during-drain>12.6. Truncating during drain</h3><p>Runners which
support draining pipelines need the ability to drain SDFs; otherwise, the
+ <span class=n>watermark_estimator</span><span class=o>.</span><span
class=n>current_watermark</span><span
class=p>()</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=nx>This</span> <span
class=nx>is</span [...]
pipeline may never stop. By default, bounded restrictions process the
remainder of the restriction while
unbounded restrictions finish processing at the next SDF-initiated checkpoint
or runner-initiated split.
You are able to override this default behavior by defining the appropriate
method on the restriction
@@ -4098,7 +4098,7 @@ provider.</p><div class="language-java snippet"><div
class="notebook-skip code-s
<span class=k>if</span> <span class=s2>"optional"</span> <span
class=ow>in</span> <span class=n>file_name</span><span class=p>:</span>
<span class=c1># Skip optional files</span>
<span class=k>return</span> <span class=bp>None</span>
- <span class=k>return</span> <span
class=n>restriction</span></code></pre></div></div></div><h3
id=bundle-finalization>12.7. Bundle finalization</h3><p>Bundle finalization
enables a <code>DoFn</code> to perform side effects by registering a callback.
+ <span class=k>return</span> <span
class=n>restriction</span></code></pre></div></div></div><div
class="language-go snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-go data-lang=go><span
class=nx>This</span> <span class=nx>is</span> <span class=nx>not</span> <span
class=nx>supported</span> <sp [...]
The callback is invoked once the runner has acknowledged that it has durably
persisted the output.
For example, a message queue might need to acknowledge messages that it has
ingested into the pipeline.
Bundle finalization is not limited to SDFs but is called out here since this
is the primary
@@ -4117,7 +4117,7 @@ use case.</p><div class="language-java snippet"><div
class="notebook-skip code-s
<span class=c1># Register callback function for this bundle that performs
the side</span>
<span class=c1># effect.</span>
- <span class=n>bundle_finalizer</span><span class=o>.</span><span
class=n>register</span><span class=p>(</span><span
class=n>my_callback_func</span><span
class=p>)</span></code></pre></div></div></div><h2
id=multi-language-pipelines>13. Multi-language pipelines</h2><p>Beam allows you
to combine transforms written in any supported SDK language (currently, Java
and Python) and use them in one multi-language pipeline. This capability makes
it easy to provide new functionality simultaneou [...]
+ <span class=n>bundle_finalizer</span><span class=o>.</span><span
class=n>register</span><span class=p>(</span><span
class=n>my_callback_func</span><span
class=p>)</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><spa [...]
abstract static class Builder<K, V>
implements ExternalTransformBuilder<External.Configuration, PBegin,
PCollection<KV<K, V>>> {
abstract Builder<K, V> setConsumerConfig(Map<String, Object>
config);
@@ -4245,7 +4245,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr,
beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/10/07</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/10/12</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 217d5af..4e6db2c 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.33.0/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.33.0/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-10-11T18:22:03-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
