This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 39fc301 Publishing website 2021/10/02 06:02:45 at commit b9457b7
39fc301 is described below
commit 39fc301df36a239e676d921c95f0b09321b0775e
Author: jenkins <bui...@apache.org>
AuthorDate: Sat Oct 2 06:02:45 2021 +0000
Publishing website 2021/10/02 06:02:45 at commit b9457b7
---
website/generated-content/documentation/index.xml | 259 +++++++++++++++++++--
.../documentation/programming-guide/index.html | 211 ++++++++++++++---
website/generated-content/sitemap.xml | 2 +-
3 files changed, 425 insertions(+), 47 deletions(-)
diff --git a/website/generated-content/documentation/index.xml
b/website/generated-content/documentation/index.xml
index 9c698e5..56112c0 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -5898,7 +5898,7 @@ determined by the input data, or depend on a different
branch of your pipeline.&
<span class="kd">var</span> <span class="nx">cutOff</span>
<span class="kt">float64</span>
<span class="nx">ok</span> <span class="o">:=</span> <span
class="nf">lengthCutOffIter</span><span class="p">(</span><span
class="o">&amp;</span><span class="nx">cutOff</span><span
class="p">)</span>
<span class="k">if</span> <span class="p">!</span><span
class="nx">ok</span> <span class="p">{</span>
-<span class="k">return</span> <span class="nx">fmt</span><span
class="p">.</span><span class="nf">Errorf</span><span
class="p">(</span><span class="s">&#34;No length cutoff
provided.&#34;</span><span class="p">)</span>
+<span class="k">return</span> <span class="nx">fmt</span><span
class="p">.</span><span class="nf">Errorf</span><span
class="p">(</span><span class="s">&#34;no length cutoff
provided&#34;</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">if</span> <span class="nb">float64</span><span
class="p">(</span><span class="nb">len</span><span
class="p">(</span><span class="nx">word</span><span
class="p">))</span> <span class="p">&gt;</span> <span
class="nx">cutOff</span> <span class="p">{</span>
<span class="nf">emitAboveCutoff</span><span
class="p">(</span><span class="nx">word</span><span
class="p">)</span>
@@ -5948,7 +5948,7 @@ a different view of the side input each time.</p>
latest trigger firing. This is particularly useful if you use a side input with
a single global window and specify a trigger.</p>
<h3 id="additional-outputs">4.5. Additional outputs</h3>
-<p class="language-java language-python">While <code>ParDo</code>
always produces a main output <code>PCollection</code> (as the return
value
+<p class="language-java language-py">While <code>ParDo</code> always
produces a main output <code>PCollection</code> (as the return value
from <code>apply</code>), you can also have your
<code>ParDo</code> produce any number of additional
output <code>PCollection</code>s. If you choose to have multiple
outputs, your <code>ParDo</code>
returns all of the output <code>PCollection</code>s (including the main
output) bundled
@@ -6875,6 +6875,8 @@ having Beam understand their element schemas.</p>
<p class="language-java">In Java you could use the following set of classes
to represent the purchase schema. Beam will automatically
infer the correct schema based on the members of the class.</p>
<p class="language-py">In Python you can use the following set of classes
to represent the purchase schema. Beam will automatically infer the correct
schema based on the members of the class.</p>
+<p class="language-go">In Go, schema encoding is used by default for struct
types, with Exported fields becoming part of the schema.
+Beam will automatically infer the schema based on the fields and field tags of
the struct, and their order.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -6941,6 +6943,38 @@ infer the correct schema based on the members of the
class.</p>
<span class="n">purchase_amount</span><span class="p">:</span>
<span class="nb">float</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="kd">type</span> <span
class="nx">Purchase</span> <span class="kd">struct</span> <span
class="p">{</span>
+<span class="c1">// ID of the user who made the purchase.
+</span><span class="c1"></span> <span class="nx">UserID</span>
<span class="kt">string</span> <span
class="s">`beam:&#34;userId&#34;`</span>
+<span class="c1">// Identifier of the item that was purchased.
+</span><span class="c1"></span> <span class="nx">ItemID</span>
<span class="kt">int64</span> <span
class="s">`beam:&#34;itemId&#34;`</span>
+<span class="c1">// The shipping address, a nested type.
+</span><span class="c1"></span> <span
class="nx">ShippingAddress</span> <span
class="nx">ShippingAddress</span> <span
class="s">`beam:&#34;shippingAddress&#34;`</span>
+<span class="c1">// The cost of the item in cents.
+</span><span class="c1"></span> <span class="nx">Cost</span>
<span class="kt">int64</span> <span
class="s">`beam:&#34;cost&#34;`</span>
+<span class="c1">// The transactions that paid for this purchase.
+</span><span class="c1"></span> <span class="c1">// A slice since
the purchase might be spread out over multiple
+</span><span class="c1"></span> <span class="c1">// credit cards.
+</span><span class="c1"></span> <span
class="nx">Transactions</span> <span class="p">[]</span><span
class="nx">Transaction</span> <span
class="s">`beam:&#34;transactions&#34;`</span>
+<span class="p">}</span>
+<span class="kd">type</span> <span
class="nx">ShippingAddress</span> <span class="kd">struct</span>
<span class="p">{</span>
+<span class="nx">StreetAddress</span> <span
class="kt">string</span> <span
class="s">`beam:&#34;streetAddress&#34;`</span>
+<span class="nx">City</span> <span class="kt">string</span>
<span class="s">`beam:&#34;city&#34;`</span>
+<span class="nx">State</span> <span class="o">*</span><span
class="kt">string</span> <span
class="s">`beam:&#34;state&#34;`</span>
+<span class="nx">Country</span> <span class="kt">string</span>
<span class="s">`beam:&#34;country&#34;`</span>
+<span class="nx">PostCode</span> <span class="kt">string</span>
<span class="s">`beam:&#34;postCode&#34;`</span>
+<span class="p">}</span>
+<span class="kd">type</span> <span class="nx">Transaction</span>
<span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span>
<span class="s">`beam:&#34;bank&#34;`</span>
+<span class="nx">PurchaseAmount</span> <span
class="kt">float64</span> <span
class="s">`beam:&#34;purchaseAmount&#34;`</span>
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
<p class="language-java">Using JavaBean classes as above is one way to map
a schema to Java classes. However multiple Java classes might have
the same schema, in which case the different Java types can often be used
interchangeably. Beam will add implicit
conversions between types that have matching schemas. For example, the above
@@ -7087,6 +7121,9 @@ a nanosecond timestamp, with the INT64 field representing
seconds and the INT32
limited-precision decimal type would have an integer argument indicating how
many digits of precision are represented.
The argument is represented by a schema type, so can itself be a complex
type.</p>
<p class="language-java">In Java, a logical type is specified as a subclass
of the <code>LogicalType</code> class. A custom Java class can be
specified to represent the logical type and conversion functions must be
supplied to convert back and forth between this Java class and the underlying
Schema type representation. For example, the logical type representing
nanosecond timestamp might be implemented as follows</p>
+<p class="language-go">In Go, a logical type is specified with a custom
implementation of the <code>beam.SchemaProvider</code> interface.
+For example, the logical type provider representing nanosecond timestamps
+might be implemented as follows</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -7110,9 +7147,91 @@ The argument is represented by a schema type, so can
itself be a complex type.&l
<span class="o">}</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="c1">// Define a logical provider like so:
+</span><span class="c1"></span>
+<span class="c1">// TimestampNanos is a logical type using time.Time, but
+</span><span class="c1">// encodes as a schema type.
+</span><span class="c1"></span><span class="kd">type</span>
<span class="nx">TimestampNanos</span> <span
class="nx">time</span><span class="p">.</span><span
class="nx">Time</span>
+<span class="kd">func</span> <span class="p">(</span><span
class="nx">tn</span> <span class="nx">TimestampNanos</span><span
class="p">)</span> <span class="nf">Seconds</span><span
class="p">()</span> <span class="kt">int64</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="nx">time</span><span
class="p">.</span><span class="nf">Time</span><span
class="p">(</span><span class="nx">tn</span><span
class="p">).</span><span class="nf">Unix</span><span
class="p">()</span>
+<span class="p">}</span>
+<span class="kd">func</span> <span class="p">(</span><span
class="nx">tn</span> <span class="nx">TimestampNanos</span><span
class="p">)</span> <span class="nf">Nanos</span><span
class="p">()</span> <span class="kt">int32</span> <span
class="p">{</span>
+<span class="k">return</span> <span
class="nb">int32</span><span class="p">(</span><span
class="nx">time</span><span class="p">.</span><span
class="nf">Time</span><span class="p">(</span><span
class="nx">tn</span><span class="p">).</span><span
class="nf">UnixNano</span><span class="p">()</span> <span
class="o">%</span> <span class="mi">1000000000</span><span
class="p">)</span>
+<span class="p">}</span>
+<span class="c1">// tnStorage is the storage schema for TimestampNanos.
+</span><span class="c1"></span><span class="kd">type</span>
<span class="nx">tnStorage</span> <span class="kd">struct</span>
<span class="p">{</span>
+<span class="nx">Seconds</span> <span class="kt">int64</span>
<span class="s">`beam:&#34;seconds&#34;`</span>
+<span class="nx">Nanos</span> <span class="kt">int32</span>
<span class="s">`beam:&#34;nanos&#34;`</span>
+<span class="p">}</span>
+<span class="kd">var</span> <span class="p">(</span>
+<span class="c1">// reflect.Type of the Value type of TimestampNanos
+</span><span class="c1"></span> <span class="nx">tnType</span>
<span class="p">=</span> <span class="nx">reflect</span><span
class="p">.</span><span class="nf">TypeOf</span><span
class="p">((</span><span class="o">*</span><span
class="nx">TimestampNanos</span><span class="p">)(</span><span
class="kc">nil</span><span class="p">)).</span><span
class="nf">Elem</span><span class="p">()</span>
+<span class="nx">tnStorageType</span> <span class="p">=</span>
<span class="nx">reflect</span><span class="p">.</span><span
class="nf">TypeOf</span><span class="p">((</span><span
class="o">*</span><span class="nx">tnStorage</span><span
class="p">)(</span><span class="kc">nil</span><span
class="p">)).</span><span class="nf">Elem</span><span
class="p">()</span>
+<span class="p">)</span>
+<span class="c1">// TimestampNanosProvider implements the
beam.SchemaProvider interface.
+</span><span class="c1"></span><span class="kd">type</span>
<span class="nx">TimestampNanosProvider</span> <span
class="kd">struct</span><span class="p">{}</span>
+<span class="c1">// FromLogicalType converts checks if the given type is
TimestampNanos, and if so
+</span><span class="c1">// returns the storage type.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">p</span> <span
class="o">*</span><span
class="nx">TimestampNanosProvider</span><span class="p">)</span>
<span class="nf">FromLogicalType</span><span
class="p">(</span><span class="nx">rt</span> <span
class="nx">reflect</span><span class="p">.</span><span
class="nx">Type</span><span class="p">)</span> &l [...]
+<span class="k">if</span> <span class="nx">rt</span> <span
class="o">!=</span> <span class="nx">tnType</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">fmt</span><span
class="p">.</span><span class="nf">Errorf</span><span
class="p">(</span><span class="s">&#34;unable to provide
schema.LogicalType for type %v, want %v&#34;</span><span
class="p">,</span> <span class="nx">rt</span><span
class="p">,</span> <span class="nx">tnType</span><span
class="p">)</span>
+<span class="p">}</span>
+<span class="k">return</span> <span
class="nx">tnStorageType</span><span class="p">,</span> <span
class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// BuildEncoder builds a Beam schema encoder for the
TimestampNanos type.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">p</span> <span
class="o">*</span><span
class="nx">TimestampNanosProvider</span><span class="p">)</span>
<span class="nf">BuildEncoder</span><span
class="p">(</span><span class="nx">rt</span> <span
class="nx">reflect</span><span class="p">.</span><span
class="nx">Type</span><span class="p">)</span> <s [...]
+<span class="k">if</span> <span class="nx">_</span><span
class="p">,</span> <span class="nx">err</span> <span
class="o">:=</span> <span class="nx">p</span><span
class="p">.</span><span class="nf">FromLogicalType</span><span
class="p">(</span><span class="nx">rt</span><span
class="p">);</span> <span class="nx">err</span> <span
class="o">!=</span> <span class="kc">nil</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">enc</span><span class="p">,</span> <span
class="nx">err</span> <span class="o">:=</span> <span
class="nx">coder</span><span class="p">.</span><span
class="nf">RowEncoderForStruct</span><span class="p">(</span><span
class="nx">tnStorageType</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span
class="o">!=</span> <span class="kc">nil</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="k">return</span> <span class="kd">func</span><span
class="p">(</span><span class="nx">iface</span> <span
class="kd">interface</span><span class="p">{},</span> <span
class="nx">w</span> <span class="nx">io</span><span
class="p">.</span><span class="nx">Writer</span><span
class="p">)</span> <span class="kt">error</span> <span
class="p">{</span>
+<span class="nx">v</span> <span class="o">:=</span> <span
class="nx">iface</span><span class="p">.(</span><span
class="nx">TimestampNanos</span><span class="p">)</span>
+<span class="k">return</span> <span class="nf">enc</span><span
class="p">(</span><span class="nx">tnStorage</span><span
class="p">{</span>
+<span class="nx">Seconds</span><span class="p">:</span> <span
class="nx">v</span><span class="p">.</span><span
class="nf">Seconds</span><span class="p">(),</span>
+<span class="nx">Nanos</span><span class="p">:</span> <span
class="nx">v</span><span class="p">.</span><span
class="nf">Nanos</span><span class="p">(),</span>
+<span class="p">},</span> <span class="nx">w</span><span
class="p">)</span>
+<span class="p">},</span> <span class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// BuildDecoder builds a Beam schema decoder for the
TimestampNanos type.
+</span><span class="c1"></span><span class="kd">func</span>
<span class="p">(</span><span class="nx">p</span> <span
class="o">*</span><span
class="nx">TimestampNanosProvider</span><span class="p">)</span>
<span class="nf">BuildDecoder</span><span
class="p">(</span><span class="nx">rt</span> <span
class="nx">reflect</span><span class="p">.</span><span
class="nx">Type</span><span class="p">)</span> <s [...]
+<span class="k">if</span> <span class="nx">_</span><span
class="p">,</span> <span class="nx">err</span> <span
class="o">:=</span> <span class="nx">p</span><span
class="p">.</span><span class="nf">FromLogicalType</span><span
class="p">(</span><span class="nx">rt</span><span
class="p">);</span> <span class="nx">err</span> <span
class="o">!=</span> <span class="kc">nil</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">dec</span><span class="p">,</span> <span
class="nx">err</span> <span class="o">:=</span> <span
class="nx">coder</span><span class="p">.</span><span
class="nf">RowDecoderForStruct</span><span class="p">(</span><span
class="nx">tnStorageType</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span
class="o">!=</span> <span class="kc">nil</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="k">return</span> <span class="kd">func</span><span
class="p">(</span><span class="nx">r</span> <span
class="nx">io</span><span class="p">.</span><span
class="nx">Reader</span><span class="p">)</span> <span
class="p">(</span><span class="kd">interface</span><span
class="p">{},</span> <span class="kt">error</span><span
class="p">)</span> <span class="p">{</span>
+<span class="nx">s</span><span class="p">,</span> <span
class="nx">err</span> <span class="o">:=</span> <span
class="nf">dec</span><span class="p">(</span><span
class="nx">r</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span
class="o">!=</span> <span class="kc">nil</span> <span
class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span
class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">tn</span> <span class="o">:=</span> <span
class="nx">s</span><span class="p">.(</span><span
class="nx">tnStorage</span><span class="p">)</span>
+<span class="k">return</span> <span
class="nf">TimestampNanos</span><span class="p">(</span><span
class="nx">time</span><span class="p">.</span><span
class="nf">Unix</span><span class="p">(</span><span
class="nx">tn</span><span class="p">.</span><span
class="nx">Seconds</span><span class="p">,</span> <span
class="nb">int64</span><span class="p">(</span><span
class="nx">tn</span><span class="p">.</sp [...]
+<span class="p">},</span> <span class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// Register it like so:
+</span><span class="c1"></span>
+<span class="nx">beam</span><span class="p">.</span><span
class="nf">RegisterSchemaProvider</span><span
class="p">(</span><span class="nx">tnType</span><span
class="p">,</span> <span class="o">&amp;</span><span
class="nx">TimestampNanosProvider</span><span
class="p">{})</span></code></pre></div>
+</div>
+</div>
<h4 id="built-in-logical-types">6.4.2. Useful logical types</h4>
+<p class="language-py">Currently the Python SDK provides minimal
convenience logical types,
+other than to handle <code>MicrosInstant</code>.</p>
+<p class="language-go">Currently the Go SDK provides minimal convenience
logical types,
+other than to handle additional integer primitives, and
<code>time.Time</code>.</p>
<h5 id="enumerationtype"><strong>EnumerationType</strong></h5>
-<p>This logical type allows creating an enumeration type consisting of a
set of named constants.</p>
+<p class="language-py">This convenience builder doesn&rsquo;t yet exist
for the Python SDK.</p>
+<p class="language-go">This convenience builder doesn&rsquo;t yet exist
for the Go SDK.</p>
+<p class="language-java">This logical type allows creating an enumeration
type consisting of a set of named constants.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -7124,7 +7243,7 @@ The argument is represented by a schema type, so can
itself be a complex type.&l
<span class="o">.</span><span class="na">build</span><span
class="o">();</span></code></pre></div>
</div>
</div>
-<p>The value of this field is stored in the row as an INT32 type, however
the logical type defines a value type that lets
+<p class="language-java">The value of this field is stored in the row as an
INT32 type, however the logical type defines a value type that lets
you access the enumeration either as a string or a value. For example:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
@@ -7136,7 +7255,7 @@ you access the enumeration either as a string or a value.
For example:</p>
</span><span class="c1"></span><span
class="n">enumValue</span><span class="o">.</span><span
class="na">toString</span><span class="o">();</span> <span
class="o">//</span> <span class="n">Returns</span> <span
class="s">&#34;RED&#34;</span><span class="o">,</span>
<span class="n">the</span> <span class="n">string</span> <span
class="n">value</span> <span class="n">of</span> <span
class="n">the</ [...]
</div>
</div>
-<p>Given a row object with an enumeration field, you can also extract the
field as the enumeration value.</p>
+<p class="language-java">Given a row object with an enumeration field, you
can also extract the field as the enumeration value.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -7145,10 +7264,12 @@ you access the enumeration either as a string or a
value. For example:</p>
<div class="highlight"><pre class="chroma"><code
class="language-java" data-lang="java"><span
class="n">EnumerationType</span><span class="o">.</span><span
class="na">Value</span> <span class="n">enumValue</span> <span
class="o">=</span> <span class="n">row</span><span
class="o">.</span><span class="na">getLogicalTypeValue</span><span
class="o">(</span><span
class="s">&#34;color&#34;</span><span class="o">,&l [...]
</div>
</div>
-<p>Automatic schema inference from Java POJOs and JavaBeans automatically
converts Java enums to EnumerationType logical
+<p class="language-java">Automatic schema inference from Java POJOs and
JavaBeans automatically converts Java enums to EnumerationType logical
types.</p>
<h5 id="oneoftype"><strong>OneOfType</strong></h5>
-<p>OneOfType allows creating a disjoint union type over a set of schema
fields. For example:</p>
+<p class="language-py">This convenience builder doesn&rsquo;t yet exist
for the Python SDK.</p>
+<p class="language-go">This convenience builder doesn&rsquo;t yet exist
for the Go SDK.</p>
+<p class="language-java">OneOfType allows creating a disjoint union type
over a set of schema fields. For example:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -7163,7 +7284,7 @@ types.</p>
<span class="o">.</span><span class="na">build</span><span
class="o">();</span></code></pre></div>
</div>
</div>
-<p>The value of this field is stored in the row as another Row type, where
all the fields are marked as nullable. The
+<p class="language-java">The value of this field is stored in the row as
another Row type, where all the fields are marked as nullable. The
logical type however defines a Value object that contains an enumeration value
indicating which field was set and allows
getting just that field:</p>
<div class='language-java snippet'>
@@ -7188,12 +7309,27 @@ getting just that field:</p>
<span class="o">}</span></code></pre></div>
</div>
</div>
-<p>In the above example we used the field names in the switch statement for
clarity, however the enum integer values could
+<p class="language-java">In the above example we used the field names in
the switch statement for clarity, however the enum integer values could
also be used.</p>
<h3 id="creating-schemas">6.5. Creating Schemas</h3>
-<p>In order to take advantage of schemas, your
<code>PCollection</code>s must have a schema attached to it. Often, the
source itself will attach a schema to the PCollection. For example, when using
<code>AvroIO</code> to read Avro files, the source can automatically
infer a Beam schema from the Avro schema and attach that to the Beam
<code>PCollection</code>. However not all sources produce schemas. In
addition, often Beam pipelines have intermediate stages and types [...]
+<p>In order to take advantage of schemas, your
<code>PCollection</code>s must have a schema attached to it.
+Often, the source itself will attach a schema to the PCollection.
+For example, when using <code>AvroIO</code> to read Avro files, the
source can automatically infer a Beam schema from the Avro schema and attach
that to the Beam <code>PCollection</code>.
+However not all sources produce schemas.
+In addition, often Beam pipelines have intermediate stages and types, and
those also can benefit from the expressiveness of schemas.</p>
<h4 id="inferring-schemas">6.5.1. Inferring schemas</h4>
-<p class="language-java">Beam is able to infer schemas from a variety of
common Java types. The <code>@DefaultSchema</code> annotation can be used
to tell Beam to infer schemas from a specific type. The annotation takes a
<code>SchemaProvider</code> as an argument, and
<code>SchemaProvider</code> classes are already built in for common Java
types. The <code>SchemaRegistry</code> can also be invoked
programmatically for cases where it is not practical to annotat [...]
+<nav class="language-switcher">
+<strong>Adapt for:</strong>
+<ul>
+<li data-type="language-java" class="active">Java SDK</li>
+<li data-type="language-py">Python SDK</li>
+<li data-type="language-go">Go SDK</li>
+</ul>
+</nav>
+<p class="language-java">Beam is able to infer schemas from a variety of
common Java types.
+The <code>@DefaultSchema</code> annotation can be used to tell Beam to
infer schemas from a specific type.
+The annotation takes a <code>SchemaProvider</code> as an argument, and
<code>SchemaProvider</code> classes are already built in for common Java
types.
+The <code>SchemaRegistry</code> can also be invoked programmatically for
cases where it is not practical to annotate the Java type itself.</p>
<p class="language-java"><strong>Java POJOs</strong></p>
<p class="language-java">A POJO (Plain Old Java Object) is a Java object
that is not bound by any restriction other than the Java Language
Specification. A POJO can contain member variables that are primitives, that
are other POJOs, or are collections maps or
@@ -7359,10 +7495,48 @@ correctly infers types with <code>beam.Row</code>
or with <code>Select&
<span class="n">purchase_amount</span><span
class="o">=</span><span class="nb">float</span><span
class="p">(</span><span class="n">item</span><span
class="p">[</span><span
class="s2">&#34;purchase_amount&#34;</span><span
class="p">])))</span></code></pre></div>
</div>
</div>
+<p class="language-go">Beam currently only infers schemas for exported
fields in Go structs.</p>
+<p class="language-go"><strong>Structs</strong></p>
+<p class="language-go">Beam will automatically infer schemas for all Go
structs used
+as PCollection elements, and default to encoding them using
+schema encoding.</p>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="kd">type</span> <span
class="nx">Transaction</span> <span class="kd">struct</span><span
class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span>
+<span class="nx">PurchaseAmount</span> <span
class="kt">float64</span>
+<span class="nx">checksum</span> <span class="p">[]</span><span
class="kt">byte</span> <span class="c1">// ignored
+</span><span class="c1"></span><span
class="p">}</span></code></pre></div>
+</div>
+</div>
+<p class="language-go">Unexported fields are ignored, and cannot be
automatically infered as part of the schema.
+Fields of type func, channel, unsafe.Pointer, or uintptr will be ignored by
inference.
+Fields of interface types are ignored, unless a schema provider
+is registered for them.</p>
+<p class="language-go">By default, schema field names will match the
exported struct field names.
+In the above example, &ldquo;Bank&rdquo; and
&ldquo;PurchaseAmount&rdquo; are the schema field names.
+A schema field name can be overridden with a struct tag for the field.</p>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="kd">type</span> <span
class="nx">Transaction</span> <span class="kd">struct</span><span
class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span>
<span class="s">`beam:&#34;bank&#34;`</span>
+<span class="nx">PurchaseAmount</span> <span
class="kt">float64</span> <span
class="s">`beam:&#34;purchase_amount&#34;`</span>
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
+<p class="language-go">Overriding schema field names is useful for
compatibility cross language transforms,
+as schema fields may have different requirements or restrictions from Go
exported fields.</p>
<h3 id="using-schemas">6.6. Using Schema Transforms</h3>
<p>A schema on a <code>PCollection</code> enables a rich variety of
relational transforms. The fact that each record is composed of
named fields allows for simple and readable aggregations that reference fields
by name, similar to the aggregations in
a SQL expression.</p>
+<p class="language-go">Beam does not yet support Schema transforms natively
in Go. However, it will be implemented with the following behavior.</p>
<h4 id="661-field-selection-syntax">6.6.1. Field selection syntax</h4>
<p>The advantage of schemas is that they allow referencing of element
fields by name. Beam provides a selection syntax for
referencing fields, including nested and repeated fields. This syntax is used
by all of the schema transforms when
@@ -7892,6 +8066,7 @@ pipeline will fail to launch.</p>
<p>A <code>PCollection</code> with a schema can apply a
<code>ParDo</code>, just like any other <code>PCollection</code>.
However the Beam runner is aware
of schemas when applying a <code>ParDo</code>, which enables additional
functionality.</p>
<h5 id="input-conversion"><strong>Input conversion</strong></h5>
+<p class="language-go">Beam does not yet support input conversion in
Go.</p>
<p>Since Beam knows the schema of the source
<code>PCollection</code>, it can automatically convert the elements to
any Java type for
which a matching schema is known. For example, using the above-mentioned
Transaction schema, say we have the following
<code>PCollection</code>:</p>
@@ -7967,6 +8142,14 @@ using the above-described selection expressions, as
follows:</p>
<p>For more information, see the section on field-selection expressions.
When selecting subschemas, Beam will
automatically convert to any matching schema type, just like when reading the
entire row.</p>
<h2 id="data-encoding-and-type-safety">7. Data encoding and type
safety</h2>
+<nav class="language-switcher">
+<strong>Adapt for:</strong>
+<ul>
+<li data-type="language-java" class="active">Java SDK</li>
+<li data-type="language-py">Python SDK</li>
+<li data-type="language-go">Go SDK</li>
+</ul>
+</nav>
<p>When Beam runners execute your pipeline, they often need to materialize
the
intermediate data in your <code>PCollection</code>s, which requires
converting elements to
and from byte strings. The Beam SDKs use objects called
<code>Coder</code>s to describe how
@@ -7989,6 +8172,12 @@ types, Tuple, Iterable, StringUtf8 and more. You can
find all of the available
Coder subclasses in the
<a
href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders";>apache_beam.coders</a>
package.</p>
+<p class="language-go">Standard Go types like <code>int</code>,
<code>int64</code> <code>float64</code>, <code>[]byte</code>,
and <code>string</code> and more are coded using builtin coders.
+Structs and pointers to structs default using Beam Schema Row encoding.
+However, users can build and register custom coders with
<code>beam.RegisterCoder</code>.
+You can find available Coder functions in the
+<a
href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/coders";>coder</a>
+package.</p>
<blockquote>
<p>Note that coders do not necessarily have a 1:1 relationship with types.
For
example, the Integer type can have multiple valid coders, and input and output
@@ -8016,6 +8205,8 @@ mapping of Java types to the default coders that the
pipeline should use for
<p class="language-py">The Beam SDK for Python has a
<code>CoderRegistry</code> that represents a mapping of
Python types to the default coder that should be used for
<code>PCollection</code>s of each
type.</p>
+<p class="language-go">The Beam SDK for Go allows users to register default
coder
+implementations with <code>beam.RegisterCoder</code>.</p>
<p class="language-java">By default, the Beam SDK for Java automatically
infers the <code>Coder</code> for the
elements of a <code>PCollection</code> produced by a
<code>PTransform</code> using the type parameter
from the transform&rsquo;s function object, such as
<code>DoFn</code>. In the case of <code>ParDo</code>,
@@ -8032,13 +8223,20 @@ with the typehints
<code>@beam.typehints.with_input_types(int)</code> and
and produces an output element of type str. In such a case, the Beam SDK for
Python will automatically infer the default <code>Coder</code> for the
output <code>PCollection</code>
(in the default pipeline <code>CoderRegistry</code>, this is
<code>BytesCoder</code>).</p>
+<p class="language-go">By default, the Beam SDK for Go automatically infers
the <code>Coder</code> for the elements of an output
<code>PCollection</code> by the output of the transform&rsquo;s
function object, such as a <code>DoFn</code>.
+In the case of <code>ParDo</code>, for example a <code>DoFn</code>
+with the parameters of <code>v int, emit func(string)</code> accepts an
input element of type <code>int</code>
+and produces an output element of type <code>string</code>.
+In such a case, the Beam SDK for Go will automatically infer the default
<code>Coder</code> for the output <code>PCollection</code> to be
the <code>string_utf8</code> coder.</p>
+<span class="language-java">
<blockquote>
-<p>NOTE: If you create your <code>PCollection</code> from in-memory
data by using the
+<p><strong>Note:</strong> If you create your
<code>PCollection</code> from in-memory data by using the
<code>Create</code> transform, you cannot rely on coder inference and
default coders.
<code>Create</code> does not have access to any typing information for
its arguments, and
may not be able to infer a coder if the argument list contains a value whose
exact run-time class doesn&rsquo;t have a default coder registered.</p>
</blockquote>
+</span>
<p class="language-java">When using <code>Create</code>, the simplest
way to ensure that you have the correct coder
is by invoking <code>withCoder</code> when you apply the
<code>Create</code> transform.</p>
<h3 id="default-coders-and-the-coderregistry">7.2. Default coders and the
CoderRegistry</h3>
@@ -8153,8 +8351,9 @@ this pipeline, verify that Integer values are encoded
using
<p class="language-py">You can use the method
<code>CoderRegistry.get_coder</code> to determine the default Coder
for a Python type. You can use <code>coders.registry</code> to access
the <code>CoderRegistry</code>.
This allows you to determine (or set) the default Coder for a Python
type.</p>
+<p class="language-go">You can use the <code>beam.NewCoder</code>
function to determine the default Coder for a Go type.</p>
<h4 id="setting-default-coder">7.2.2. Setting the default coder for a
type</h4>
-<p>To set the default Coder for a
+<p class="language-java language-py">To set the default Coder for a
<span class="language-java">Java</span><span
class="language-py">Python</span>
type for a particular pipeline, you obtain and modify the pipeline&rsquo;s
<code>CoderRegistry</code>. You use the method
@@ -8164,10 +8363,13 @@ to get the <code>CoderRegistry</code> object, and
then use the method
<span
class="language-java"><code>CoderRegistry.registerCoder</code></span>
<span
class="language-py"><code>CoderRegistry.register_coder</code></span>
to register a new <code>Coder</code> for the target type.</p>
-<p>The following example code demonstrates how to set a default Coder, in
this case
+<p class="language-go">To set the default Coder for a Go type you use the
function <code>beam.RegisterCoder</code> to register a encoder and
decoder functions for the target type.
+However, built in types like <code>int</code>,
<code>string</code>, <code>float64</code>, etc cannot have their
coders overridde.</p>
+<p class="language-java language-py">The following example code
demonstrates how to set a default Coder, in this case
<code>BigEndianIntegerCoder</code>, for
<span class="language-java">Integer</span><span
class="language-py">int</span>
values for a pipeline.</p>
+<p class="language-go">The following example code demonstrates how to set a
custom Coder for <code>MyCustomType</code> elements.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -8187,8 +8389,26 @@ values for a pipeline.</p>
<div class="highlight"><pre class="chroma"><code class="language-py"
data-lang="py"><span class="n">apache_beam</span><span
class="o">.</span><span class="n">coders</span><span
class="o">.</span><span class="n">registry</span><span
class="o">.</span><span class="n">register_coder</span><span
class="p">(</span><span class="nb">int</span><span
class="p">,</span> <span
class="n">BigEndianIntegerCoder</span><span c [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go"
data-lang="go"><span class="kd">type</span> <span
class="nx">MyCustomType</span> <span class="kd">struct</span><span
class="p">{</span>
+<span class="o">...</span>
+<span class="p">}</span>
+<span class="c1">// See documentation on beam.RegisterCoder for other
supported coder forms.
+</span><span class="c1"></span>
+<span class="kd">func</span> <span
class="nf">encode</span><span class="p">(</span><span
class="nx">MyCustomType</span><span class="p">)</span> <span
class="p">[]</span><span class="kt">byte</span> <span
class="p">{</span> <span class="o">...</span> <span
class="p">}</span>
+<span class="kd">func</span> <span
class="nf">decode</span><span class="p">(</span><span
class="nx">b</span> <span class="p">[]</span><span
class="kt">byte</span><span class="p">)</span> <span
class="nx">MyCustomType</span> <span class="p">{</span> <span
class="o">...</span> <span class="p">}</span>
+<span class="kd">func</span> <span class="nf">init</span><span
class="p">()</span> <span class="p">{</span>
+<span class="nx">beam</span><span class="p">.</span><span
class="nf">RegisterCoder</span><span class="p">(</span><span
class="nx">reflect</span><span class="p">.</span><span
class="nf">TypeOf</span><span class="p">((</span><span
class="o">*</span><span class="nx">MyCustomType</span><span
class="p">)(</span><span class="kc">nil</span><span
class="p">)).</span><span class="nf">Elem</span><span class="p">
[...]
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
<h4 id="annotating-custom-type-default-coder">7.2.3. Annotating a custom
data type with a default coder</h4>
-<p class="language-java"><p>If your pipeline program defines a custom
data type, you can use the
+<span class="language-java">
+<p>If your pipeline program defines a custom data type, you can use the
<code>@DefaultCoder</code> annotation to specify the coder to use with
that type.
By default, Beam will use <code>SerializableCoder</code> which uses Java
serialization,
but it has drawbacks:</p>
@@ -8202,11 +8422,11 @@ See this <a
href="https://blog.softwaremill.com/the-best-serialization-strate
equivalent objects.</p>
<p>For key/value pairs, the correctness of key-based operations
(GroupByKey, Combine) and per-key State depends on having a deterministic
-coder for the key.</p>
+coder for the key</p>
</li>
</ol>
<p>You can use the <code>@DefaultCoder</code> annotation to set a new
default as follows:</p>
-</p>
+</span>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip"
data-bs-placement="bottom" title="Copy to clipboard">
@@ -8236,8 +8456,9 @@ the <code>@DefaultCoder</code> annotation, your
coder class must implement
<span class="o">}</span></code></pre></div>
</div>
</div>
-<p class="language-py">The Beam SDK for Python does not support annotating
data types with a default
-coder. If you would like to set a default coder, use the method described in
the
+<p class="language-py language-go">The Beam SDK for <span
class="language-py">Python</span><span class="language-go">Go</span>
+does not support annotating data types with a default coder.
+If you would like to set a default coder, use the method described in the
previous section, <em>Setting the default coder for a type</em>.</p>
<h2 id="windowing">8. Windowing</h2>
<p>Windowing subdivides a <code>PCollection</code> according to the
timestamps of its
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index b31a31e..9cc9792 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1367,7 +1367,7 @@ determined by the input data, or depend on a different
branch of your pipeline.<
<span class=kd>var</span> <span class=nx>cutOff</span> <span
class=kt>float64</span>
<span class=nx>ok</span> <span class=o>:=</span> <span
class=nf>lengthCutOffIter</span><span class=p>(</span><span
class=o>&</span><span class=nx>cutOff</span><span class=p>)</span>
<span class=k>if</span> <span class=p>!</span><span class=nx>ok</span>
<span class=p>{</span>
- <span class=k>return</span> <span class=nx>fmt</span><span
class=p>.</span><span class=nf>Errorf</span><span class=p>(</span><span
class=s>"No length cutoff provided."</span><span class=p>)</span>
+ <span class=k>return</span> <span class=nx>fmt</span><span
class=p>.</span><span class=nf>Errorf</span><span class=p>(</span><span
class=s>"no length cutoff provided"</span><span class=p>)</span>
<span class=p>}</span>
<span class=k>if</span> <span class=nb>float64</span><span
class=p>(</span><span class=nb>len</span><span class=p>(</span><span
class=nx>word</span><span class=p>))</span> <span class=p>></span> <span
class=nx>cutOff</span> <span class=p>{</span>
<span class=nf>emitAboveCutoff</span><span
class=p>(</span><span class=nx>word</span><span class=p>)</span>
@@ -1412,7 +1412,7 @@ gets called multiple times, once for each window. Each
call to <code>processElem
projects the “current” window for the main input element, and thus
might provide
a different view of the side input each time.</p><p>If the side input has
multiple trigger firings, Beam uses the value from the
latest trigger firing. This is particularly useful if you use a side input with
-a single global window and specify a trigger.</p><h3
id=additional-outputs>4.5. Additional outputs</h3><p class="language-java
language-python">While <code>ParDo</code> always produces a main output
<code>PCollection</code> (as the return value
+a single global window and specify a trigger.</p><h3
id=additional-outputs>4.5. Additional outputs</h3><p class="language-java
language-py">While <code>ParDo</code> always produces a main output
<code>PCollection</code> (as the return value
from <code>apply</code>), you can also have your <code>ParDo</code> produce
any number of additional
output <code>PCollection</code>s. If you choose to have multiple outputs, your
<code>ParDo</code>
returns all of the output <code>PCollection</code>s (including the main
output) bundled
@@ -1889,7 +1889,8 @@ types across different programming-language APIs.</p><p>A
<code>PCollection</cod
Schema rows; Beam uses a special coder to encode schema types.</p><h3
id=schemas-for-pl-types>6.2. Schemas for programming language
types</h3><p>While schemas themselves are language independent, they are
designed to embed naturally into the programming languages
of the Beam SDK being used. This allows Beam users to continue using native
types while reaping the advantage of
having Beam understand their element schemas.</p><p class=language-java>In
Java you could use the following set of classes to represent the purchase
schema. Beam will automatically
-infer the correct schema based on the members of the class.</p><p
class=language-py>In Python you can use the following set of classes to
represent the purchase schema. Beam will automatically infer the correct schema
based on the members of the class.</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre [...]
+infer the correct schema based on the members of the class.</p><p
class=language-py>In Python you can use the following set of classes to
represent the purchase schema. Beam will automatically infer the correct schema
based on the members of the class.</p><p class=language-go>In Go, schema
encoding is used by default for struct types, with Exported fields becoming
part of the schema.
+Beam will automatically infer the schema based on the fields and field tags of
the struct, and their order.</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=nd>@DefaultSchema</span><span
class=o>(</span><span class=n>JavaBeanSchema</spa [...]
<span class=kd>public</span> <span class=kd>class</span> <span
class=nc>Purchase</span> <span class=o>{</span>
<span class=kd>public</span> <span class=n>String</span> <span
class=nf>getUserId</span><span class=o>();</span> <span class=c1>// Returns
the id of the user who made the purchase.
</span><span class=c1></span> <span class=kd>public</span> <span
class=kt>long</span> <span class=nf>getItemId</span><span class=o>();</span>
<span class=c1>// Returns the identifier of the item that was purchased.
@@ -1946,7 +1947,33 @@ infer the correct schema based on the members of the
class.</p><p class=language
<span class=k>class</span> <span class=nc>Transaction</span><span
class=p>(</span><span class=n>typing</span><span class=o>.</span><span
class=n>NamedTuple</span><span class=p>):</span>
<span class=n>bank</span><span class=p>:</span> <span class=nb>str</span>
- <span class=n>purchase_amount</span><span class=p>:</span> <span
class=nb>float</span></code></pre></div></div></div><p
class=language-java>Using JavaBean classes as above is one way to map a schema
to Java classes. However multiple Java classes might have
+ <span class=n>purchase_amount</span><span class=p>:</span> <span
class=nb>float</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=kd>type</span> <span
class=nx>Purchase</span> <span class=kd>struct</span> <s [...]
+ <span class=c1>// ID of the user who made the purchase.
+</span><span class=c1></span> <span class=nx>UserID</span> <span
class=kt>string</span> <span class=s>`beam:"userId"`</span>
+ <span class=c1>// Identifier of the item that was purchased.
+</span><span class=c1></span> <span class=nx>ItemID</span> <span
class=kt>int64</span> <span class=s>`beam:"itemId"`</span>
+ <span class=c1>// The shipping address, a nested type.
+</span><span class=c1></span> <span class=nx>ShippingAddress</span> <span
class=nx>ShippingAddress</span> <span
class=s>`beam:"shippingAddress"`</span>
+ <span class=c1>// The cost of the item in cents.
+</span><span class=c1></span> <span class=nx>Cost</span> <span
class=kt>int64</span> <span class=s>`beam:"cost"`</span>
+ <span class=c1>// The transactions that paid for this purchase.
+</span><span class=c1></span> <span class=c1>// A slice since the purchase
might be spread out over multiple
+</span><span class=c1></span> <span class=c1>// credit cards.
+</span><span class=c1></span> <span class=nx>Transactions</span> <span
class=p>[]</span><span class=nx>Transaction</span> <span
class=s>`beam:"transactions"`</span>
+<span class=p>}</span>
+
+<span class=kd>type</span> <span class=nx>ShippingAddress</span> <span
class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>StreetAddress</span> <span class=kt>string</span> <span
class=s>`beam:"streetAddress"`</span>
+ <span class=nx>City</span> <span class=kt>string</span> <span
class=s>`beam:"city"`</span>
+ <span class=nx>State</span> <span class=o>*</span><span
class=kt>string</span> <span class=s>`beam:"state"`</span>
+ <span class=nx>Country</span> <span class=kt>string</span> <span
class=s>`beam:"country"`</span>
+ <span class=nx>PostCode</span> <span class=kt>string</span> <span
class=s>`beam:"postCode"`</span>
+<span class=p>}</span>
+
+<span class=kd>type</span> <span class=nx>Transaction</span> <span
class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span>
<span class=s>`beam:"bank"`</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span>
<span class=s>`beam:"purchaseAmount"`</span>
+<span class=p>}</span></code></pre></div></div></div><p
class=language-java>Using JavaBean classes as above is one way to map a schema
to Java classes. However multiple Java classes might have
the same schema, in which case the different Java types can often be used
interchangeably. Beam will add implicit
conversions between types that have matching schemas. For example, the above
<code>Transaction</code> class has the same schema as the following
class:</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=nd>@DefaultSchema</span><span
class=o>(</span><span class=n>JavaFieldSchema</span><span class=o>.</span><span
cl [...]
@@ -1985,7 +2012,9 @@ type to represent nanosecond timestamps is represented as
a schema containing an
alone does not say anything about how to interpret this type, however the
logical type tells you that this represents
a nanosecond timestamp, with the INT64 field representing seconds and the
INT32 field representing nanoseconds.</p><p>Logical types are also specified by
an argument, which allows creating a class of related types. For example, a
limited-precision decimal type would have an integer argument indicating how
many digits of precision are represented.
-The argument is represented by a schema type, so can itself be a complex
type.</p><p class=language-java>In Java, a logical type is specified as a
subclass of the <code>LogicalType</code> class. A custom Java class can be
specified to represent the logical type and conversion functions must be
supplied to convert back and forth between this Java class and the underlying
Schema type representation. For example, the logical type representing
nanosecond timestamp might be implemented as fol [...]
+The argument is represented by a schema type, so can itself be a complex
type.</p><p class=language-java>In Java, a logical type is specified as a
subclass of the <code>LogicalType</code> class. A custom Java class can be
specified to represent the logical type and conversion functions must be
supplied to convert back and forth between this Java class and the underlying
Schema type representation. For example, the logical type representing
nanosecond timestamp might be implemented as fol [...]
+For example, the logical type provider representing nanosecond timestamps
+might be implemented as follows</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=c1>// A Logical type using
java.time.Instant to represent the logical type.
</span><span class=c1></span><span class=kd>public</span> <span
class=kd>class</span> <span class=nc>TimestampNanos</span> <span
class=kd>implements</span> <span class=n>LogicalType</span><span
class=o><</span><span class=n>Instant</span><span class=o>,</span> <span
class=n>Row</span><span class=o>></span> <span class=o>{</span>
<span class=c1>// The underlying schema used to represent rows.
</span><span class=c1></span> <span class=kd>private</span> <span
class=kd>final</span> <span class=n>Schema</span> <span class=n>SCHEMA</span>
<span class=o>=</span> <span class=n>Schema</span><span class=o>.</span><span
class=na>builder</span><span class=o>().</span><span
class=na>addInt64Field</span><span class=o>(</span><span
class=s>"seconds"</span><span class=o>).</span><span
class=na>addInt32Field</span><span class=o>(</span><span
class=s>"nanos"</span><span class [...]
@@ -2003,20 +2032,101 @@ The argument is represented by a schema type, so can
itself be a complex type.</
<span class=o>}</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div></div><h4
id=built-in-logical-types>6.4.2. Useful logical types</h4><h5
id=enumerationtype><strong>EnumerationType</strong></h5><p>This logical type
allows creating an enumeration type consisting of a set of named
constants.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div cl [...]
+<span class=o>}</span></code></pre></div></div></div><div class="language-go
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=c1>// Define a logical provider like
so:
+</span><span class=c1></span>
+
+<span class=c1>// TimestampNanos is a logical type using time.Time, but
+</span><span class=c1>// encodes as a schema type.
+</span><span class=c1></span><span class=kd>type</span> <span
class=nx>TimestampNanos</span> <span class=nx>time</span><span
class=p>.</span><span class=nx>Time</span>
+
+<span class=kd>func</span> <span class=p>(</span><span class=nx>tn</span>
<span class=nx>TimestampNanos</span><span class=p>)</span> <span
class=nf>Seconds</span><span class=p>()</span> <span class=kt>int64</span>
<span class=p>{</span>
+ <span class=k>return</span> <span class=nx>time</span><span
class=p>.</span><span class=nf>Time</span><span class=p>(</span><span
class=nx>tn</span><span class=p>).</span><span class=nf>Unix</span><span
class=p>()</span>
+<span class=p>}</span>
+<span class=kd>func</span> <span class=p>(</span><span class=nx>tn</span>
<span class=nx>TimestampNanos</span><span class=p>)</span> <span
class=nf>Nanos</span><span class=p>()</span> <span class=kt>int32</span> <span
class=p>{</span>
+ <span class=k>return</span> <span class=nb>int32</span><span
class=p>(</span><span class=nx>time</span><span class=p>.</span><span
class=nf>Time</span><span class=p>(</span><span class=nx>tn</span><span
class=p>).</span><span class=nf>UnixNano</span><span class=p>()</span> <span
class=o>%</span> <span class=mi>1000000000</span><span class=p>)</span>
+<span class=p>}</span>
+
+<span class=c1>// tnStorage is the storage schema for TimestampNanos.
+</span><span class=c1></span><span class=kd>type</span> <span
class=nx>tnStorage</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>Seconds</span> <span class=kt>int64</span> <span
class=s>`beam:"seconds"`</span>
+ <span class=nx>Nanos</span> <span class=kt>int32</span> <span
class=s>`beam:"nanos"`</span>
+<span class=p>}</span>
+
+<span class=kd>var</span> <span class=p>(</span>
+ <span class=c1>// reflect.Type of the Value type of TimestampNanos
+</span><span class=c1></span> <span class=nx>tnType</span> <span
class=p>=</span> <span class=nx>reflect</span><span class=p>.</span><span
class=nf>TypeOf</span><span class=p>((</span><span class=o>*</span><span
class=nx>TimestampNanos</span><span class=p>)(</span><span
class=kc>nil</span><span class=p>)).</span><span class=nf>Elem</span><span
class=p>()</span>
+ <span class=nx>tnStorageType</span> <span class=p>=</span> <span
class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span
class=p>((</span><span class=o>*</span><span class=nx>tnStorage</span><span
class=p>)(</span><span class=kc>nil</span><span class=p>)).</span><span
class=nf>Elem</span><span class=p>()</span>
+<span class=p>)</span>
+
+<span class=c1>// TimestampNanosProvider implements the beam.SchemaProvider
interface.
+</span><span class=c1></span><span class=kd>type</span> <span
class=nx>TimestampNanosProvider</span> <span class=kd>struct</span><span
class=p>{}</span>
+
+<span class=c1>// FromLogicalType converts checks if the given type is
TimestampNanos, and if so
+</span><span class=c1>// returns the storage type.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>p</span> <span class=o>*</span><span
class=nx>TimestampNanosProvider</span><span class=p>)</span> <span
class=nf>FromLogicalType</span><span class=p>(</span><span class=nx>rt</span>
<span class=nx>reflect</span><span class=p>.</span><span
class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span
class=nx>reflect</span><span class=p>.</span><span class=nx>Type</span><span
class=p>,</s [...]
+ <span class=k>if</span> <span class=nx>rt</span> <span
class=o>!=</span> <span class=nx>tnType</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span
class=p>,</span> <span class=nx>fmt</span><span class=p>.</span><span
class=nf>Errorf</span><span class=p>(</span><span class=s>"unable to
provide schema.LogicalType for type %v, want %v"</span><span
class=p>,</span> <span class=nx>rt</span><span class=p>,</span> <span
class=nx>tnType</span><span class=p>)</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=nx>tnStorageType</span><span
class=p>,</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+<span class=c1>// BuildEncoder builds a Beam schema encoder for the
TimestampNanos type.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>p</span> <span class=o>*</span><span
class=nx>TimestampNanosProvider</span><span class=p>)</span> <span
class=nf>BuildEncoder</span><span class=p>(</span><span class=nx>rt</span>
<span class=nx>reflect</span><span class=p>.</span><span
class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span
class=kd>func</span><span class=p>(</span><span class=kd>interface</span><span
class=p>{},</ [...]
+ <span class=k>if</span> <span class=nx>_</span><span class=p>,</span>
<span class=nx>err</span> <span class=o>:=</span> <span class=nx>p</span><span
class=p>.</span><span class=nf>FromLogicalType</span><span
class=p>(</span><span class=nx>rt</span><span class=p>);</span> <span
class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span
class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span
class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>enc</span><span class=p>,</span> <span
class=nx>err</span> <span class=o>:=</span> <span class=nx>coder</span><span
class=p>.</span><span class=nf>RowEncoderForStruct</span><span
class=p>(</span><span class=nx>tnStorageType</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span
class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span
class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=kd>func</span><span
class=p>(</span><span class=nx>iface</span> <span
class=kd>interface</span><span class=p>{},</span> <span class=nx>w</span> <span
class=nx>io</span><span class=p>.</span><span class=nx>Writer</span><span
class=p>)</span> <span class=kt>error</span> <span class=p>{</span>
+ <span class=nx>v</span> <span class=o>:=</span> <span
class=nx>iface</span><span class=p>.(</span><span
class=nx>TimestampNanos</span><span class=p>)</span>
+ <span class=k>return</span> <span class=nf>enc</span><span
class=p>(</span><span class=nx>tnStorage</span><span class=p>{</span>
+ <span class=nx>Seconds</span><span class=p>:</span>
<span class=nx>v</span><span class=p>.</span><span class=nf>Seconds</span><span
class=p>(),</span>
+ <span class=nx>Nanos</span><span class=p>:</span>
<span class=nx>v</span><span class=p>.</span><span class=nf>Nanos</span><span
class=p>(),</span>
+ <span class=p>},</span> <span class=nx>w</span><span
class=p>)</span>
+ <span class=p>},</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+<span class=c1>// BuildDecoder builds a Beam schema decoder for the
TimestampNanos type.
+</span><span class=c1></span><span class=kd>func</span> <span
class=p>(</span><span class=nx>p</span> <span class=o>*</span><span
class=nx>TimestampNanosProvider</span><span class=p>)</span> <span
class=nf>BuildDecoder</span><span class=p>(</span><span class=nx>rt</span>
<span class=nx>reflect</span><span class=p>.</span><span
class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span
class=kd>func</span><span class=p>(</span><span class=nx>io</span><span
class=p>.</span><spa [...]
+ <span class=k>if</span> <span class=nx>_</span><span class=p>,</span>
<span class=nx>err</span> <span class=o>:=</span> <span class=nx>p</span><span
class=p>.</span><span class=nf>FromLogicalType</span><span
class=p>(</span><span class=nx>rt</span><span class=p>);</span> <span
class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span
class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span
class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>dec</span><span class=p>,</span> <span
class=nx>err</span> <span class=o>:=</span> <span class=nx>coder</span><span
class=p>.</span><span class=nf>RowDecoderForStruct</span><span
class=p>(</span><span class=nx>tnStorageType</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span
class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span
class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=kd>func</span><span
class=p>(</span><span class=nx>r</span> <span class=nx>io</span><span
class=p>.</span><span class=nx>Reader</span><span class=p>)</span> <span
class=p>(</span><span class=kd>interface</span><span class=p>{},</span> <span
class=kt>error</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>s</span><span class=p>,</span> <span
class=nx>err</span> <span class=o>:=</span> <span class=nf>dec</span><span
class=p>(</span><span class=nx>r</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span
class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span
class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>tn</span> <span class=o>:=</span> <span
class=nx>s</span><span class=p>.(</span><span class=nx>tnStorage</span><span
class=p>)</span>
+ <span class=k>return</span> <span
class=nf>TimestampNanos</span><span class=p>(</span><span
class=nx>time</span><span class=p>.</span><span class=nf>Unix</span><span
class=p>(</span><span class=nx>tn</span><span class=p>.</span><span
class=nx>Seconds</span><span class=p>,</span> <span class=nb>int64</span><span
class=p>(</span><span class=nx>tn</span><span class=p>.</span><span
class=nx>Nanos</span><span class=p>))),</span> <span class=kc>nil</span>
+ <span class=p>},</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+
+
+<span class=c1>// Register it like so:
+</span><span class=c1></span>
+<span class=nx>beam</span><span class=p>.</span><span
class=nf>RegisterSchemaProvider</span><span class=p>(</span><span
class=nx>tnType</span><span class=p>,</span> <span class=o>&</span><span
class=nx>TimestampNanosProvider</span><span
class=p>{})</span></code></pre></div></div></div><h4
id=built-in-logical-types>6.4.2. Useful logical types</h4><p
class=language-py>Currently the Python SDK provides minimal convenience logical
types,
+other than to handle <code>MicrosInstant</code>.</p><p
class=language-go>Currently the Go SDK provides minimal convenience logical
types,
+other than to handle additional integer primitives, and
<code>time.Time</code>.</p><h5
id=enumerationtype><strong>EnumerationType</strong></h5><p
class=language-py>This convenience builder doesn’t yet exist for the
Python SDK.</p><p class=language-go>This convenience builder doesn’t yet
exist for the Go SDK.</p><p class=language-java>This logical type allows
creating an enumeration type consisting of a set of named constants.</p><div
class="language-java snippet"><div class=" [...]
<span class=err>…</span>
<span class=o>.</span><span class=na>addLogicalTypeField</span><span
class=o>(</span><span class=s>"color"</span><span class=o>,</span>
<span class=n>EnumerationType</span><span class=o>.</span><span
class=na>create</span><span class=o>(</span><span
class=s>"RED"</span><span class=o>,</span> <span
class=s>"GREEN"</span><span class=o>,</span> <span
class=s>"BLUE"</span><span class=o>))</span>
- <span class=o>.</span><span class=na>build</span><span
class=o>();</span></code></pre></div></div></div><p>The value of this field is
stored in the row as an INT32 type, however the logical type defines a value
type that lets
+ <span class=o>.</span><span class=na>build</span><span
class=o>();</span></code></pre></div></div></div><p class=language-java>The
value of this field is stored in the row as an INT32 type, however the logical
type defines a value type that lets
you access the enumeration either as a string or a value. For example:</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>EnumerationType</span><span class=o>.</span><span
class=na>Value</span> <span class=n>enumValue</span> <span class [...]
<span class=n>enumValue</span><span class=o>.</span><span
class=na>getValue</span><span class=o>();</span> <span class=c1>// Returns 0,
the integer value of the constant.
-</span><span class=c1></span><span class=n>enumValue</span><span
class=o>.</span><span class=na>toString</span><span class=o>();</span> <span
class=o>//</span> <span class=n>Returns</span> <span
class=s>"RED"</span><span class=o>,</span> <span class=n>the</span>
<span class=n>string</span> <span class=n>value</span> <span class=n>of</span>
<span class=n>the</span> <span
class=n>constant</span></code></pre></div></div></div><p>Given a row object
with an enumeration field, you can [...]
-types.</p><h5 id=oneoftype><strong>OneOfType</strong></h5><p>OneOfType allows
creating a disjoint union type over a set of schema fields. For
example:</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>Schema</span> <span
class=n>schema [...]
+</span><span class=c1></span><span class=n>enumValue</span><span
class=o>.</span><span class=na>toString</span><span class=o>();</span> <span
class=o>//</span> <span class=n>Returns</span> <span
class=s>"RED"</span><span class=o>,</span> <span class=n>the</span>
<span class=n>string</span> <span class=n>value</span> <span class=n>of</span>
<span class=n>the</span> <span
class=n>constant</span></code></pre></div></div></div><p
class=language-java>Given a row object with an enumer [...]
+types.</p><h5 id=oneoftype><strong>OneOfType</strong></h5><p
class=language-py>This convenience builder doesn’t yet exist for the
Python SDK.</p><p class=language-go>This convenience builder doesn’t yet
exist for the Go SDK.</p><p class=language-java>OneOfType allows creating a
disjoint union type over a set of schema fields. For example:</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-plac [...]
<span class=err>…</span>
<span class=o>.</span><span class=na>addLogicalTypeField</span><span
class=o>(</span><span class=s>"oneOfField"</span><span class=o>,</span>
<span class=n>OneOfType</span><span class=o>.</span><span
class=na>create</span><span class=o>(</span><span class=n>Field</span><span
class=o>.</span><span class=na>of</span><span class=o>(</span><span
class=s>"intField"</span><span class=o>,</span> <span
class=n>FieldType</span><span class=o>.</span><span class=na>INT32</span><span
class=o>),</span>
<span class=n>Field</span><span class=o>.</span><span
class=na>of</span><span class=o>(</span><span
class=s>"stringField"</span><span class=o>,</span> <span
class=n>FieldType</span><span class=o>.</span><span class=na>STRING</span><span
class=o>),</span>
<span class=n>Field</span><span class=o>.</span><span
class=na>of</span><span class=o>(</span><span
class=s>"bytesField"</span><span class=o>,</span> <span
class=n>FieldType</span><span class=o>.</span><span class=na>BYTES</span><span
class=o>)))</span>
- <span class=o>.</span><span class=na>build</span><span
class=o>();</span></code></pre></div></div></div><p>The value of this field is
stored in the row as another Row type, where all the fields are marked as
nullable. The
+ <span class=o>.</span><span class=na>build</span><span
class=o>();</span></code></pre></div></div></div><p class=language-java>The
value of this field is stored in the row as another Row type, where all the
fields are marked as nullable. The
logical type however defines a Value object that contains an enumeration value
indicating which field was set and allows
getting just that field:</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=c1>// Returns an enumeration
indicating all possible case values for the enum.
</span><span class=c1>// For the above example, this will be
@@ -2034,8 +2144,15 @@ getting just that field:</p><div class="language-java
snippet"><div class="noteb
<span class=k>return</span> <span class=n>processString</span><span
class=o>(</span><span class=n>oneOfValue</span><span class=o>.</span><span
class=na>getValue</span><span class=o>(</span><span class=n>String</span><span
class=o>.</span><span class=na>class</span><span class=o>));</span>
<span class=k>case</span> <span class=s>"bytesField"</span><span
class=o>:</span>
<span class=k>return</span> <span class=n>processBytes</span><span
class=o>(</span><span class=n>oneOfValue</span><span class=o>.</span><span
class=na>getValue</span><span class=o>(</span><span class=n>bytes</span><span
class=o>[].</span><span class=na>class</span><span class=o>));</span>
-<span class=o>}</span></code></pre></div></div></div><p>In the above example
we used the field names in the switch statement for clarity, however the enum
integer values could
-also be used.</p><h3 id=creating-schemas>6.5. Creating Schemas</h3><p>In order
to take advantage of schemas, your <code>PCollection</code>s must have a schema
attached to it. Often, the source itself will attach a schema to the
PCollection. For example, when using <code>AvroIO</code> to read Avro files,
the source can automatically infer a Beam schema from the Avro schema and
attach that to the Beam <code>PCollection</code>. However not all sources
produce schemas. In addition, often Bea [...]
+<span class=o>}</span></code></pre></div></div></div><p class=language-java>In
the above example we used the field names in the switch statement for clarity,
however the enum integer values could
+also be used.</p><h3 id=creating-schemas>6.5. Creating Schemas</h3><p>In order
to take advantage of schemas, your <code>PCollection</code>s must have a schema
attached to it.
+Often, the source itself will attach a schema to the PCollection.
+For example, when using <code>AvroIO</code> to read Avro files, the source can
automatically infer a Beam schema from the Avro schema and attach that to the
Beam <code>PCollection</code>.
+However not all sources produce schemas.
+In addition, often Beam pipelines have intermediate stages and types, and
those also can benefit from the expressiveness of schemas.</p><h4
id=inferring-schemas>6.5.1. Inferring schemas</h4><nav
class=language-switcher><strong>Adapt for:</strong><ul><li
data-type=language-java class=active>Java SDK</li><li
data-type=language-py>Python SDK</li><li data-type=language-go>Go
SDK</li></ul></nav><p class=language-java>Beam is able to infer schemas from a
variety of common Java types.
+The <code>@DefaultSchema</code> annotation can be used to tell Beam to infer
schemas from a specific type.
+The annotation takes a <code>SchemaProvider</code> as an argument, and
<code>SchemaProvider</code> classes are already built in for common Java types.
+The <code>SchemaRegistry</code> can also be invoked programmatically for cases
where it is not practical to annotate the Java type itself.</p><p
class=language-java><strong>Java POJOs</strong></p><p class=language-java>A
POJO (Plain Old Java Object) is a Java object that is not bound by any
restriction other than the Java Language
Specification. A POJO can contain member variables that are primitives, that
are other POJOs, or are collections maps or
arrays thereof. POJOs do not have to extend prespecified classes or extend any
specific interfaces.</p><p class=language-java>If a POJO class is annotated
with <code>@DefaultSchema(JavaFieldSchema.class)</code>, Beam will
automatically infer a schema for
this class. Nested classes are supported as are classes with
<code>List</code>, array, and <code>Map</code> fields.</p><p
class=language-java>For example, annotating the following class tells Beam to
infer a schema from this POJO class and apply it to any
@@ -2106,9 +2223,25 @@ type information. If it’s unable to it will fall
back to the generic type
<code>Any</code>. Sometimes this is not ideal, you can use casts to make sure
Beam
correctly infers types with <code>beam.Row</code> or with
<code>Select</code>:</p><div class="language-py snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-py data-lang=py><span class=n>input_pc</span> <span
class=o>=</span> <span class=o>...</span> <span class=c1># {"bank":
..., "purc [...]
<span class=n>output_pc</span> <span class=o>=</span> <span
class=n>input_pc</span> <span class=o>|</span> <span class=n>beam</span><span
class=o>.</span><span class=n>Map</span><span class=p>(</span><span
class=k>lambda</span> <span class=n>item</span><span class=p>:</span> <span
class=n>beam</span><span class=o>.</span><span class=n>Row</span><span
class=p>(</span><span class=n>bank</span><span class=o>=</span><span
class=nb>str</span><span class=p>(</span><span class=n>item</span><spa [...]
- <span
class=n>purchase_amount</span><span class=o>=</span><span
class=nb>float</span><span class=p>(</span><span class=n>item</span><span
class=p>[</span><span class=s2>"purchase_amount"</span><span
class=p>])))</span></code></pre></div></div></div><h3 id=using-schemas>6.6.
Using Schema Transforms</h3><p>A schema on a <code>PCollection</code> enables a
rich variety of relational transforms. The fact that each record is composed of
+ <span
class=n>purchase_amount</span><span class=o>=</span><span
class=nb>float</span><span class=p>(</span><span class=n>item</span><span
class=p>[</span><span class=s2>"purchase_amount"</span><span
class=p>])))</span></code></pre></div></div></div><p class=language-go>Beam
currently only infers schemas for exported fields in Go structs.</p><p
class=language-go><strong>Structs</strong></p><p class=language-go>Beam will
automat [...]
+as PCollection elements, and default to encoding them using
+schema encoding.</p><div class="language-go snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-go data-lang=go><span class=kd>type</span> <span
class=nx>Transaction</span> <span class=kd>struct</span><span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span>
+
+ <span class=nx>checksum</span> <span class=p>[]</span><span
class=kt>byte</span> <span class=c1>// ignored
+</span><span class=c1></span><span
class=p>}</span></code></pre></div></div></div><p class=language-go>Unexported
fields are ignored, and cannot be automatically infered as part of the schema.
+Fields of type func, channel, unsafe.Pointer, or uintptr will be ignored by
inference.
+Fields of interface types are ignored, unless a schema provider
+is registered for them.</p><p class=language-go>By default, schema field names
will match the exported struct field names.
+In the above example, “Bank” and “PurchaseAmount” are
the schema field names.
+A schema field name can be overridden with a struct tag for the field.</p><div
class="language-go snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-go data-lang=go><span
class=kd>type</span> <span class=nx>Transaction</span> <span
class=kd>struct</span><span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span> <span
class=s>`beam:"bank"`</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span> <span
class=s>`beam:"purchase_amount"`</span>
+<span class=p>}</span></code></pre></div></div></div><p
class=language-go>Overriding schema field names is useful for compatibility
cross language transforms,
+as schema fields may have different requirements or restrictions from Go
exported fields.</p><h3 id=using-schemas>6.6. Using Schema Transforms</h3><p>A
schema on a <code>PCollection</code> enables a rich variety of relational
transforms. The fact that each record is composed of
named fields allows for simple and readable aggregations that reference fields
by name, similar to the aggregations in
-a SQL expression.</p><h4 id=661-field-selection-syntax>6.6.1. Field selection
syntax</h4><p>The advantage of schemas is that they allow referencing of
element fields by name. Beam provides a selection syntax for
+a SQL expression.</p><p class=language-go>Beam does not yet support Schema
transforms natively in Go. However, it will be implemented with the following
behavior.</p><h4 id=661-field-selection-syntax>6.6.1. Field selection
syntax</h4><p>The advantage of schemas is that they allow referencing of
element fields by name. Beam provides a selection syntax for
referencing fields, including nested and repeated fields. This syntax is used
by all of the schema transforms when
referencing the fields they operate on. The syntax can also be used inside of
a DoFn to specify which schema fields to
process.</p><p>Addressing fields by name still retains type safety as Beam
will check that schemas match at the time the pipeline graph
@@ -2198,7 +2331,7 @@ follows.</p><div class="language-java snippet"><div
class="notebook-skip code-sn
unboxing the row. For example, give a schema with a single INT64 field, the
following will convert it to a
<code>PCollection<Long></code></p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>Long</span><span class=o>></span> <span
class=n>longs</span> <span class=o>=</span> <span c [...]
pipeline will fail to launch.</p><h4 id=663-schemas-in-pardo>6.6.3. Schemas in
ParDo</h4><p>A <code>PCollection</code> with a schema can apply a
<code>ParDo</code>, just like any other <code>PCollection</code>. However the
Beam runner is aware
-of schemas when applying a <code>ParDo</code>, which enables additional
functionality.</p><h5 id=input-conversion><strong>Input
conversion</strong></h5><p>Since Beam knows the schema of the source
<code>PCollection</code>, it can automatically convert the elements to any Java
type for
+of schemas when applying a <code>ParDo</code>, which enables additional
functionality.</p><h5 id=input-conversion><strong>Input
conversion</strong></h5><p class=language-go>Beam does not yet support input
conversion in Go.</p><p>Since Beam knows the schema of the source
<code>PCollection</code>, it can automatically convert the elements to any Java
type for
which a matching schema is known. For example, using the above-mentioned
Transaction schema, say we have the following
<code>PCollection</code>:</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PCollection</span><span
class=o><</span><span class=n>PurchasePojo</span><span class=o>></span>
<span class=n>purchases</span> <span class=o>=</span> <sp [...]
since there is a schema, you could apply the following DoFn:</p><div
class="language-java snippet"><div class="notebook-skip code-snippet"><a
class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom
title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div
class=highlight><pre class=chroma><code class=language-java
data-lang=java><span class=n>purchases</span><span class=o>.</span><span
class=na>appy</span><span class=o>(</span><span class=n>ParDo</span><span
class= [...]
@@ -2223,7 +2356,7 @@ using the above-described selection expressions, as
follows:</p><div class="lang
<span class=o>...</span>
<span class=o>}</span>
<span class=o>}));</span></code></pre></div></div></div><p>For more
information, see the section on field-selection expressions. When selecting
subschemas, Beam will
-automatically convert to any matching schema type, just like when reading the
entire row.</p><h2 id=data-encoding-and-type-safety>7. Data encoding and type
safety</h2><p>When Beam runners execute your pipeline, they often need to
materialize the
+automatically convert to any matching schema type, just like when reading the
entire row.</p><h2 id=data-encoding-and-type-safety>7. Data encoding and type
safety</h2><nav class=language-switcher><strong>Adapt for:</strong><ul><li
data-type=language-java class=active>Java SDK</li><li
data-type=language-py>Python SDK</li><li data-type=language-go>Go
SDK</li></ul></nav><p>When Beam runners execute your pipeline, they often need
to materialize the
intermediate data in your <code>PCollection</code>s, which requires converting
elements to
and from byte strings. The Beam SDKs use objects called <code>Coder</code>s to
describe how
the elements of a given <code>PCollection</code> may be encoded and
decoded.</p><blockquote><p>Note that coders are unrelated to parsing or
formatting data when interacting
@@ -2239,6 +2372,11 @@ subclasses that work with a variety of standard Python
types, such as primitive
types, Tuple, Iterable, StringUtf8 and more. You can find all of the available
Coder subclasses in the
<a
href=https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders>apache_beam.coders</a>
+package.</p><p class=language-go>Standard Go types like <code>int</code>,
<code>int64</code> <code>float64</code>, <code>[]byte</code>, and
<code>string</code> and more are coded using builtin coders.
+Structs and pointers to structs default using Beam Schema Row encoding.
+However, users can build and register custom coders with
<code>beam.RegisterCoder</code>.
+You can find available Coder functions in the
+<a
href=https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/coders>coder</a>
package.</p><blockquote><p>Note that coders do not necessarily have a 1:1
relationship with types. For
example, the Integer type can have multiple valid coders, and input and output
data can use different Integer coders. A transform might have Integer-typed
@@ -2256,7 +2394,8 @@ not been set and cannot be inferred for the given
<code>PCollection</code>.</p><
mapping of Java types to the default coders that the pipeline should use for
<code>PCollection</code>s of each type.</p><p class=language-py>The Beam SDK
for Python has a <code>CoderRegistry</code> that represents a mapping of
Python types to the default coder that should be used for
<code>PCollection</code>s of each
-type.</p><p class=language-java>By default, the Beam SDK for Java
automatically infers the <code>Coder</code> for the
+type.</p><p class=language-go>The Beam SDK for Go allows users to register
default coder
+implementations with <code>beam.RegisterCoder</code>.</p><p
class=language-java>By default, the Beam SDK for Java automatically infers the
<code>Coder</code> for the
elements of a <code>PCollection</code> produced by a <code>PTransform</code>
using the type parameter
from the transform’s function object, such as <code>DoFn</code>. In the
case of <code>ParDo</code>,
for example, a <code>DoFn<Integer, String></code> function object accepts
an input element
@@ -2270,11 +2409,15 @@ with the typehints
<code>@beam.typehints.with_input_types(int)</code> and
<code>@beam.typehints.with_output_types(str)</code> accepts an input element
of type int
and produces an output element of type str. In such a case, the Beam SDK for
Python will automatically infer the default <code>Coder</code> for the output
<code>PCollection</code>
-(in the default pipeline <code>CoderRegistry</code>, this is
<code>BytesCoder</code>).</p><blockquote><p>NOTE: If you create your
<code>PCollection</code> from in-memory data by using the
+(in the default pipeline <code>CoderRegistry</code>, this is
<code>BytesCoder</code>).</p><p class=language-go>By default, the Beam SDK for
Go automatically infers the <code>Coder</code> for the elements of an output
<code>PCollection</code> by the output of the transform’s function
object, such as a <code>DoFn</code>.
+In the case of <code>ParDo</code>, for example a <code>DoFn</code>
+with the parameters of <code>v int, emit func(string)</code> accepts an input
element of type <code>int</code>
+and produces an output element of type <code>string</code>.
+In such a case, the Beam SDK for Go will automatically infer the default
<code>Coder</code> for the output <code>PCollection</code> to be the
<code>string_utf8</code> coder.</p><span
class=language-java><blockquote><p><strong>Note:</strong> If you create your
<code>PCollection</code> from in-memory data by using the
<code>Create</code> transform, you cannot rely on coder inference and default
coders.
<code>Create</code> does not have access to any typing information for its
arguments, and
may not be able to infer a coder if the argument list contains a value whose
-exact run-time class doesn’t have a default coder
registered.</p></blockquote><p class=language-java>When using
<code>Create</code>, the simplest way to ensure that you have the correct coder
+exact run-time class doesn’t have a default coder
registered.</p></blockquote></span><p class=language-java>When using
<code>Create</code>, the simplest way to ensure that you have the correct coder
is by invoking <code>withCoder</code> when you apply the <code>Create</code>
transform.</p><h3 id=default-coders-and-the-coderregistry>7.2. Default coders
and the CoderRegistry</h3><p>Each Pipeline object has a
<code>CoderRegistry</code> object, which maps language types to
the default coder the pipeline should use for those types. You can use the
<code>CoderRegistry</code> yourself to look up the default coder for a given
type, or to
@@ -2289,7 +2432,7 @@ by using the method
<code>Pipeline.getCoderRegistry</code>. This allows you to d
this pipeline, verify that Integer values are encoded using
<code>BigEndianIntegerCoder</code>.”</p><p class=language-py>You can use
the method <code>CoderRegistry.get_coder</code> to determine the default Coder
for a Python type. You can use <code>coders.registry</code> to access the
<code>CoderRegistry</code>.
-This allows you to determine (or set) the default Coder for a Python
type.</p><h4 id=setting-default-coder>7.2.2. Setting the default coder for a
type</h4><p>To set the default Coder for a
+This allows you to determine (or set) the default Coder for a Python
type.</p><p class=language-go>You can use the <code>beam.NewCoder</code>
function to determine the default Coder for a Go type.</p><h4
id=setting-default-coder>7.2.2. Setting the default coder for a type</h4><p
class="language-java language-py">To set the default Coder for a
<span class=language-java>Java</span><span class=language-py>Python</span>
type for a particular pipeline, you obtain and modify the pipeline’s
<code>CoderRegistry</code>. You use the method
@@ -2298,21 +2441,34 @@ type for a particular pipeline, you obtain and modify
the pipeline’s
to get the <code>CoderRegistry</code> object, and then use the method
<span class=language-java><code>CoderRegistry.registerCoder</code></span>
<span class=language-py><code>CoderRegistry.register_coder</code></span>
-to register a new <code>Coder</code> for the target type.</p><p>The following
example code demonstrates how to set a default Coder, in this case
+to register a new <code>Coder</code> for the target type.</p><p
class=language-go>To set the default Coder for a Go type you use the function
<code>beam.RegisterCoder</code> to register a encoder and decoder functions for
the target type.
+However, built in types like <code>int</code>, <code>string</code>,
<code>float64</code>, etc cannot have their coders overridde.</p><p
class="language-java language-py">The following example code demonstrates how
to set a default Coder, in this case
<code>BigEndianIntegerCoder</code>, for
<span class=language-java>Integer</span><span class=language-py>int</span>
-values for a pipeline.</p><div class="language-java snippet"><div
class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PipelineOptions</span> <span
class=n>options</span> <span class=o>=</span> <span
class=n>PipelineOptionsFactory</span><span class=o>.</span><span
class=na>create< [...]
+values for a pipeline.</p><p class=language-go>The following example code
demonstrates how to set a custom Coder for <code>MyCustomType</code>
elements.</p><div class="language-java snippet"><div class="notebook-skip
code-snippet"><a class=copy type=button data-bs-toggle=tooltip
data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=n>PipelineOptions</span> <span
cla [...]
<span class=n>Pipeline</span> <span class=n>p</span> <span class=o>=</span>
<span class=n>Pipeline</span><span class=o>.</span><span
class=na>create</span><span class=o>(</span><span class=n>options</span><span
class=o>);</span>
<span class=n>CoderRegistry</span> <span class=n>cr</span> <span
class=o>=</span> <span class=n>p</span><span class=o>.</span><span
class=na>getCoderRegistry</span><span class=o>();</span>
-<span class=n>cr</span><span class=o>.</span><span
class=na>registerCoder</span><span class=o>(</span><span
class=n>Integer</span><span class=o>.</span><span class=na>class</span><span
class=o>,</span> <span class=n>BigEndianIntegerCoder</span><span
class=o>.</span><span class=na>class</span><span
class=o>);</span></code></pre></div></div></div><div class="language-py
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=botto [...]
+<span class=n>cr</span><span class=o>.</span><span
class=na>registerCoder</span><span class=o>(</span><span
class=n>Integer</span><span class=o>.</span><span class=na>class</span><span
class=o>,</span> <span class=n>BigEndianIntegerCoder</span><span
class=o>.</span><span class=na>class</span><span
class=o>);</span></code></pre></div></div></div><div class="language-py
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=botto [...]
+ <span class=o>...</span>
+<span class=p>}</span>
+
+<span class=c1>// See documentation on beam.RegisterCoder for other supported
coder forms.
+</span><span class=c1></span>
+<span class=kd>func</span> <span class=nf>encode</span><span
class=p>(</span><span class=nx>MyCustomType</span><span class=p>)</span> <span
class=p>[]</span><span class=kt>byte</span> <span class=p>{</span> <span
class=o>...</span> <span class=p>}</span>
+
+<span class=kd>func</span> <span class=nf>decode</span><span
class=p>(</span><span class=nx>b</span> <span class=p>[]</span><span
class=kt>byte</span><span class=p>)</span> <span class=nx>MyCustomType</span>
<span class=p>{</span> <span class=o>...</span> <span class=p>}</span>
+
+<span class=kd>func</span> <span class=nf>init</span><span class=p>()</span>
<span class=p>{</span>
+ <span class=nx>beam</span><span class=p>.</span><span
class=nf>RegisterCoder</span><span class=p>(</span><span
class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span
class=p>((</span><span class=o>*</span><span class=nx>MyCustomType</span><span
class=p>)(</span><span class=kc>nil</span><span class=p>)).</span><span
class=nf>Elem</span><span class=p>(),</span> <span class=nx>encode</span><span
class=p>,</span> <span class=nx>decode</span><span class=p>)</span>
+<span class=p>}</span></code></pre></div></div></div><h4
id=annotating-custom-type-default-coder>7.2.3. Annotating a custom data type
with a default coder</h4><span class=language-java><p>If your pipeline program
defines a custom data type, you can use the
<code>@DefaultCoder</code> annotation to specify the coder to use with that
type.
By default, Beam will use <code>SerializableCoder</code> which uses Java
serialization,
but it has drawbacks:</p><ol><li><p>It is inefficient in encoding size and
speed.
See this <a
href=https://blog.softwaremill.com/the-best-serialization-strategy-for-event-sourcing-9321c299632b>comparison
of Java serialization methods.</a></p></li><li><p>It is non-deterministic: it
may produce different binary encodings for two
equivalent objects.</p><p>For key/value pairs, the correctness of key-based
operations
(GroupByKey, Combine) and per-key State depends on having a deterministic
-coder for the key.</p></li></ol><p>You can use the <code>@DefaultCoder</code>
annotation to set a new default as follows:</p></p><div class="language-java
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=nd>@DefaultCoder</span><span
class=o>(</span><span class=n>Av [...]
+coder for the key</p></li></ol><p>You can use the <code>@DefaultCoder</code>
annotation to set a new default as follows:</p></span><div class="language-java
snippet"><div class="notebook-skip code-snippet"><a class=copy type=button
data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img
src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code
class=language-java data-lang=java><span class=nd>@DefaultCoder</span><span
class=o>(</span><span class=n> [...]
<span class=kd>public</span> <span class=kd>class</span> <span
class=nc>MyCustomDataType</span> <span class=o>{</span>
<span class=o>...</span>
<span class=o>}</span></code></pre></div></div></div><p class=language-java>If
you’ve created a custom coder to match your data type, and you want to use
@@ -2325,8 +2481,9 @@ the <code>@DefaultCoder</code> annotation, your coder
class must implement a sta
<span class=nd>@DefaultCoder</span><span class=o>(</span><span
class=n>MyCustomCoder</span><span class=o>.</span><span
class=na>class</span><span class=o>)</span>
<span class=kd>public</span> <span class=kd>class</span> <span
class=nc>MyCustomDataType</span> <span class=o>{</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div></div><p class=language-py>The
Beam SDK for Python does not support annotating data types with a default
-coder. If you would like to set a default coder, use the method described in
the
+<span class=o>}</span></code></pre></div></div></div><p class="language-py
language-go">The Beam SDK for <span class=language-py>Python</span><span
class=language-go>Go</span>
+does not support annotating data types with a default coder.
+If you would like to set a default coder, use the method described in the
previous section, <em>Setting the default coder for a type</em>.</p><h2
id=windowing>8. Windowing</h2><p>Windowing subdivides a
<code>PCollection</code> according to the timestamps of its
individual elements. Transforms that aggregate multiple elements, such as
<code>GroupByKey</code> and <code>Combine</code>, work implicitly on a
per-window basis — they process
@@ -3974,7 +4131,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr,
beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to
the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/09/24</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3
id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently,
portable runners such as Flink, Spark, and the Direct runner can be used with
multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language
pipelines through the Dataflow Runner v2 backend architecture.</p><div
class=feedback><p class=update>Last updated on 2021/10/01</p><h3>Have you found
everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 265a3d4..980082e 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b
[...]
\ No newline at end of file
