This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8204cac Publishing website 2018/12/14 02:43:43 at commit 5df8cb2
8204cac is described below
commit 8204cac9a46ded6bc7beddd3a20b4a255673a27a
Author: jenkins <bui...@apache.org>
AuthorDate: Fri Dec 14 02:43:43 2018 +0000
Publishing website 2018/12/14 02:43:43 at commit 5df8cb2
---
.../2018/12/13/beam-2.9.0.html} | 276 +++-------
website/generated-content/blog/index.html | 32 ++
website/generated-content/feed.xml | 601 +++------------------
website/generated-content/index.html | 10 +-
4 files changed, 186 insertions(+), 733 deletions(-)
diff --git a/website/generated-content/index.html
b/website/generated-content/blog/2018/12/13/beam-2.9.0.html
similarity index 57%
copy from website/generated-content/index.html
copy to website/generated-content/blog/2018/12/13/beam-2.9.0.html
index cc1e6bf..3b1c737 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/blog/2018/12/13/beam-2.9.0.html
@@ -28,9 +28,8 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam</title>
- <meta name="description" content="Apache Beam is an open source, unified
model and set of language-specific SDKs for defining and executing data
processing workflows, and also data ingestion and integration flows, supporting
Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs).
Dataflow pipelines simplify the mechanics of large-scale batch and streaming
data processing and can run on a number of runtimes like Apache Flink, Apache
Spark, and Google Cloud Dataflow [...]
-">
+ <title>Apache Beam 2.9.0</title>
+ <meta name="description" content="We are happy to present the new 2.9.0
release of Beam. This release includes both improvements and new
functionality.See the download page for this release.">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400";
rel="stylesheet">
<link rel="stylesheet" href="/css/site.css">
<script
src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";></script>
@@ -39,7 +38,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/"; data-proofer-ignore>
+ <link rel="canonical"
href="https://beam.apache.org/blog/2018/12/13/beam-2.9.0.html";
data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam"
href="https://beam.apache.org/feed.xml";>
<link rel="stylesheet"
href="https://use.fontawesome.com/releases/v5.4.1/css/all.css";
integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz"
crossorigin="anonymous">
@@ -53,7 +52,7 @@
</script>
</head>
- <body class="body body--index">
+ <body class="body ">
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -125,7 +124,7 @@
GitHub links will not resolve until the markdown source is
available on the master branch.
New pages would fail validation during development / PR test
automation.
-->
- <a
href="https://github.com/apache/beam/edit/master/website/src/index.md";
data-proofer-ignore>
+ <a
href="https://github.com/apache/beam/edit/master/website/src/_posts/2018-12-13-beam-2.9.0.md";
data-proofer-ignore>
<i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on
GitHub"></i>
</a>
</li>
@@ -135,6 +134,31 @@
<div class="body__contained">
<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<article class="post" itemscope itemtype="http://schema.org/BlogPosting";>
+
+ <header class="post-header">
+ <h1 class="post-title" itemprop="name headline">Apache Beam 2.9.0</h1>
+ <p class="post-meta"><time datetime="2018-12-13T00:00:01-08:00"
itemprop="datePublished">Dec 13, 2018</time> •
+
+
+ </p>
+ </header>
+
+ <div class="post-content" itemprop="articleBody">
+ <!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -147,206 +171,58 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<div class="hero-bg">
- <div class="hero section">
- <div class="hero__cols">
- <div class="hero__cols__col">
- <div class="hero__cols__col__content">
- <div class="hero__title">
- Apache Beam: An advanced unified programming model
- </div>
- <div class="hero__subtitle">
- Implement batch and streaming data processing jobs that run on any
execution engine.
- </div>
- <div class="hero__ctas hero__ctas--first">
- <a class="button button--primary"
href="/get-started/beam-overview/">Learn more</a>
- <a class="button button--primary"
href="/get-started/downloads/">Download Beam SDK 2.9.0</a>
- </div>
- <div class="hero__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java
Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python
Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go
Quickstart</a>
- </div>
- </div>
- </div>
- <div class="hero__cols__col">
- <div class="hero__blog">
- <div class="hero__blog__title">
- The latest from the blog
- </div>
- <div class="hero__blog__cards">
-
- <a class="hero__blog__cards__card"
href="/blog/2018/10/31/beam-summit-aftermath.html">
- <div class="hero__blog__cards__card__title">Inaugural edition of
the Beam Summit Europe 2018 - aftermath</div>
- <div class="hero__blog__cards__card__date">Oct 31, 2018</div>
- </a>
-
- <a class="hero__blog__cards__card"
href="/blog/2018/10/29/beam-2.8.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam
2.8.0</div>
- <div class="hero__blog__cards__card__date">Oct 29, 2018</div>
- </a>
-
- <a class="hero__blog__cards__card"
href="/blog/2018/10/03/beam-2.7.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam
2.7.0</div>
- <div class="hero__blog__cards__card__date">Oct 3, 2018</div>
- </a>
-
- </div>
- </div>
- </div>
- </div>
- </div>
-</div>
-<div class="pillars section">
- <div class="pillars__title">
- All about Apache Beam
- </div>
- <div class="pillars__cols">
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Unified
- </div>
- <div class="pillars__cols__col__body">
- Use a single programming model for both batch and streaming use cases.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Portable
- </div>
- <div class="pillars__cols__col__body">
- Execute pipelines on multiple execution environments.
- </div>
- </div>
-
- <div class="pillars__cols__col">
- <div class="pillars__cols__col__title">
- Extensible
- </div>
- <div class="pillars__cols__col__body">
- Write and share new SDKs, IO connectors, and transformation libraries.
- </div>
- </div>
-
- </div>
-</div>
+<p>We are happy to present the new 2.9.0 release of Beam. This release
includes both improvements and new functionality.
+See the <a href="/get-started/downloads/#290-2018-12-13">download page</a> for
this release.<!--more-->
+For more information on changes in 2.9.0, check out the
+<a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344258";>detailed
release notes</a>.</p>
-<div class="graphic section">
-<div class="graphic__image">
-<img src="/images/beam_architecture.png" alt="Beam architecture" />
-</div>
-</div>
+<h2 id="new-features--improvements">New Features / Improvements</h2>
+<ul>
+ <li>Support set/delete of timers by ID in Flink runner.</li>
+ <li>Improvements to stabilize integration tests.</li>
+ <li>Updates Spark runner to show Beam metrics in web UI</li>
+ <li>Vendor gRPC and Protobuf separately from beam-model-* Java packages</li>
+ <li>Avoid reshuffle for zero and one element creates</li>
+</ul>
-<div class="logos section">
- <div class="logos__title">
- Works with
- </div>
- <div class="logos__logos">
-
- <div class="logos__logos__logo">
- <a href="http://apex.apache.org";><img src="/images/logo_apex.png"
alt="APEX" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://flink.apache.org";><img src="/images/logo_flink.png"
alt="Flink" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://spark.apache.org/";><img src="/images/logo_spark.png"
alt="Spark" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="https://cloud.google.com/dataflow/";><img
src="/images/logo_google_cloud.png" alt="Google Cloud Dataflow" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://gearpump.apache.org/";><img
src="/images/logo_gearpump.png" alt="Gearpump" /></a>
- </div>
-
- <div class="logos__logos__logo">
- <a href="http://samza.apache.org/";><img src="/images/logo_samza.png"
alt="Samza" /></a>
- </div>
-
- </div>
-</div>
+<h2 id="known-issues">Known Issues</h2>
-<div class="cards section section--wide">
- <div class="section__contained">
- <div class="cards__title">
- Testimonials
- </div>
- <div class="cards__cards">
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- A framework that delivers the flexibility and advanced functionality
our customers need.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –Talend
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam has powerful semantics that solve real-world challenges
of stream processing.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –PayPal
- </div>
- </div>
- </div>
-
- <div class="cards__cards__card">
- <div class="cards__cards__card__body">
- Apache Beam represents a principled approach for analyzing data
streams.
- </div>
- <div class="cards__cards__card__user">
- <!-- TODO: Implement icons.
- <div class="cards__cards__card__user__icon">
- </div>
- -->
- <div class="cards__cards__card__user__name">
- –data Artisans
- </div>
- </div>
- </div>
-
- </div>
- <div class="cards__body">
- Beam is an open source community and contributions are greatly
appreciated!
- If you’d like to contribute, please see the <a
href="/contribute/">Contribute</a> section.
- </div>
- </div>
-</div>
+<h3 id="dependency-upgrades">Dependency Upgrades</h3>
+<ul>
+ <li>Update google-api-client libraries to 1.27.0.</li>
+ <li>Update byte-buddy to 1.9.3</li>
+ <li>Update Flink Runner to 1.5.5</li>
+ <li>Upgrade google-apitools to 0.5.24</li>
+</ul>
+
+<h3 id="sdks">SDKs</h3>
+
+<h3 id="portability">Portability</h3>
+
+<h3 id="ios">I/Os</h3>
+
+<ul>
+ <li>I/O connector for RabbitMQ.</li>
+ <li>Update SpannerIO to support unbounded writes.</li>
+ <li>Add PFADD method to RedisIO.</li>
+</ul>
+
+<h2 id="miscellaneous-fixes">Miscellaneous Fixes</h2>
+<ul>
+ <li>DataflowRunner was updated to <strong>not</strong> use <a
href="https://github.com/google/conscrypt";>Conscrypt</a> as the default
security provider.</li>
+</ul>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following people contributed
+to the 2.9.0 release. Thank you to all contributors!</p>
+
+<p>Adam Horky ,Ahmet Altay ,Alan Myrvold ,Alex Amato ,Alexey Romanenko ,Andrea
Foegler ,Andrew Fulton ,Andrew Pilloud ,Ankur ,Ankur Goenka ,Anton Kedin ,Babu
,Ben Song ,Bingfeng Shu ,Boyuan Zhang ,Brian Martin ,Brian Quinlan ,Chamikara
Jayalath ,Charles Chen ,Christian Schneider ,Colm O hEigeartaigh ,Cory Brzycki
,CraigChambersG ,Daniel Oliveira ,David Moravek ,Dusan Rychnovsky ,Etienne
Chauchot ,Eugene Kirpichov ,Fabien Rousseau ,Gleb Kanterov ,Heejong Lee
,Henning Rohde ,Ismaël Mejía , [...]
-<div class="ctas section">
- <div class="ctas__title">
- Get started
- </div>
- <div class="ctas__ctas ctas__ctas--top">
- <a class="button button--primary" href="/get-started/beam-overview/">Learn
more</a>
- <a class="button button--primary" href="/get-started/downloads/">Download
Beam SDK 2.9.0</a>
- </div>
- <div class="ctas__ctas">
- <a class="button" href="/get-started/quickstart-java/">Java Quickstart</a>
- <a class="button" href="/get-started/quickstart-py/">Python Quickstart</a>
- <a class="button" href="/get-started/quickstart-go/">Go Quickstart</a>
</div>
-</div>
+
+</article>
</div>
<!--
diff --git a/website/generated-content/blog/index.html
b/website/generated-content/blog/index.html
index 0ad4753..c1dad8e 100644
--- a/website/generated-content/blog/index.html
+++ b/website/generated-content/blog/index.html
@@ -153,6 +153,38 @@ limitations under the License.
<p>This is the blog for the Apache Beam project. This blog contains news and
updates
for the project.</p>
+<h3 id="apache-beam-290"><a class="post-link"
href="/blog/2018/12/13/beam-2.9.0.html">Apache Beam 2.9.0</a></h3>
+<p><i>Dec 13, 2018 •</i></p>
+
+<p></i></p>
+
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>We are happy to present the new 2.9.0 release of Beam. This release
includes both improvements and new functionality.
+See the <a href="/get-started/downloads/#290-2018-12-13">download page</a> for
this release.</p>
+
+<!-- Render a "read more" button if the post is longer than the excerpt -->
+
+<p>
+<a class="btn btn-default btn-sm" href="/blog/2018/12/13/beam-2.9.0.html"
role="button">
+Read more <span class="glyphicon glyphicon-menu-right"
aria-hidden="true"></span>
+</a>
+</p>
+
+<hr />
+
<h3 id="inaugural-edition-of-the-beam-summit-europe-2018---aftermath"><a
class="post-link" href="/blog/2018/10/31/beam-summit-aftermath.html">Inaugural
edition of the Beam Summit Europe 2018 - aftermath</a></h3>
<p><i>Oct 31, 2018 •
Matthias Baetens [<a
href="https://twitter.com/matthiasbaetens";>@matthiasbaetens</a>]
diff --git a/website/generated-content/feed.xml
b/website/generated-content/feed.xml
index 4d7c7a0..6c90280 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -20,6 +20,79 @@
<generator>Jekyll v3.2.0</generator>
<item>
+ <title>Apache Beam 2.9.0</title>
+ <description><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+<p>We are happy to present the new 2.9.0 release of Beam. This release
includes both improvements and new functionality.
+See the <a
href="/get-started/downloads/#290-2018-12-13">download
page</a> for this release.<!--more-->
+For more information on changes in 2.9.0, check out the
+<a
href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344258">detailed
release notes</a>.</p>
+
+<h2 id="new-features--improvements">New Features /
Improvements</h2>
+<ul>
+ <li>Support set/delete of timers by ID in Flink runner.</li>
+ <li>Improvements to stabilize integration tests.</li>
+ <li>Updates Spark runner to show Beam metrics in web UI</li>
+ <li>Vendor gRPC and Protobuf separately from beam-model-* Java
packages</li>
+ <li>Avoid reshuffle for zero and one element creates</li>
+</ul>
+
+<h2 id="known-issues">Known Issues</h2>
+
+<h3 id="dependency-upgrades">Dependency Upgrades</h3>
+<ul>
+ <li>Update google-api-client libraries to 1.27.0.</li>
+ <li>Update byte-buddy to 1.9.3</li>
+ <li>Update Flink Runner to 1.5.5</li>
+ <li>Upgrade google-apitools to 0.5.24</li>
+</ul>
+
+<h3 id="sdks">SDKs</h3>
+
+<h3 id="portability">Portability</h3>
+
+<h3 id="ios">I/Os</h3>
+
+<ul>
+ <li>I/O connector for RabbitMQ.</li>
+ <li>Update SpannerIO to support unbounded writes.</li>
+ <li>Add PFADD method to RedisIO.</li>
+</ul>
+
+<h2 id="miscellaneous-fixes">Miscellaneous Fixes</h2>
+<ul>
+ <li>DataflowRunner was updated to <strong>not</strong> use
<a
href="https://github.com/google/conscrypt">Conscrypt</a> as
the default security provider.</li>
+</ul>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following people contributed
+to the 2.9.0 release. Thank you to all contributors!</p>
+
+<p>Adam Horky ,Ahmet Altay ,Alan Myrvold ,Alex Amato ,Alexey Romanenko
,Andrea Foegler ,Andrew Fulton ,Andrew Pilloud ,Ankur ,Ankur Goenka ,Anton
Kedin ,Babu ,Ben Song ,Bingfeng Shu ,Boyuan Zhang ,Brian Martin ,Brian Quinlan
,Chamikara Jayalath ,Charles Chen ,Christian Schneider ,Colm O hEigeartaigh
,Cory Brzycki ,CraigChambersG ,Daniel Oliveira ,David Moravek ,Dusan Rychnovsky
,Etienne Chauchot ,Eugene Kirpichov ,Fabien Rousseau ,Gleb Kanterov ,Heejong
Lee ,Henning Rohde ,Ismaël M [...]
+</description>
+ <pubDate>Thu, 13 Dec 2018 00:00:01 -0800</pubDate>
+ <link>https://beam.apache.org/blog/2018/12/13/beam-2.9.0.html</link>
+ <guid
isPermaLink="true">https://beam.apache.org/blog/2018/12/13/beam-2.9.0.html</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Inaugural edition of the Beam Summit Europe 2018 -
aftermath</title>
<description><!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -1108,533 +1181,5 @@ big-data deployments.</li>
</item>
- <item>
- <title>Timely (and Stateful) Processing with Apache Beam</title>
- <description><!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p>In a <a
href="/blog/2017/02/13/stateful-processing.html">prior blog
-post</a>, I
-introduced the basics of stateful processing in Apache Beam, focusing on the
-addition of state to per-element processing. So-called
<em>timely</em> processing
-complements stateful processing in Beam by letting you set timers to request a
-(stateful) callback at some point in the future.</p>
-
-<p>What can you do with timers in Beam? Here are some examples:</p>
-
-<ul>
- <li>You can output data buffered in state after some amount of
processing time.</li>
- <li>You can take special action when the watermark estimates that you
have
-received all data up to a specified point in event time.</li>
- <li>You can author workflows with timeouts that alter state and emit
output in
-response to the absence of additional input for some period of time.</li>
-</ul>
-
-<p>These are just a few possibilities. State and timers together form a
powerful
-programming paradigm for fine-grained control to express a huge variety of
-workflows. Stateful and timely processing in Beam is portable across data
-processing engines and integrated with Beam’s unified model of event time
-windowing in both streaming and batch processing.</p>
-
-<!--more-->
-
-<h2 id="what-is-stateful-and-timely-processing">What is
stateful and timely processing?</h2>
-
-<p>In my prior post, I developed an understanding of stateful processing
largely
-by contrast with associative, commutative combiners. In this post, I’ll
-emphasize a perspective that I had mentioned only briefly: that elementwise
-processing with access to per-key-and-window state and timers represents a
-fundamental pattern for “embarrassingly parallel” computation, distinct from
-the others in Beam.</p>
-
-<p>In fact, stateful and timely computation is the low-level
computational pattern
-that underlies the others. Precisely because it is lower level, it allows you
-to really micromanage your computations to unlock new use cases and new
-efficiencies. This incurs the complexity of manually managing your state and
-timers - it isn’t magic! Let’s first look again at the two primary
-computational patterns in Beam.</p>
-
-<h3 id="element-wise-processing-pardo-map-etc">Element-wise
processing (ParDo, Map, etc)</h3>
-
-<p>The most elementary embarrassingly parallel pattern is just using a
bunch of
-computers to apply the same function to every input element of a massive
-collection. In Beam, per-element processing like this is expressed as a basic
-<code class="highlighter-rouge">ParDo</code> - analogous
to “Map” from MapReduce - which is like an enhanced “map”,
-“flatMap”, etc, from functional programming.</p>
-
-<p>The following diagram illustrates per-element processing. Input
elements are
-squares, output elements are triangles. The colors of the elements represent
-their key, which will matter later. Each input element maps to the
-corresponding output element(s) completely independently. Processing may be
-distributed across computers in any way, yielding essentially limitless
-parallelism.</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/ParDo.png" alt="ParDo offers
limitless parallelism" width="600" /></p>
-
-<p>This pattern is obvious, exists in all data-parallel paradigms, and
has
-a simple stateless implementation. Every input element can be processed
-independently or in arbitrary bundles. Balancing the work between computers is
-actually the hard part, and can be addressed by splitting, progress estimation,
-work-stealing, etc.</p>
-
-<h3
id="per-key-and-window-aggregation-combine-reduce-groupbykey-etc">Per-key
(and window) aggregation (Combine, Reduce, GroupByKey, etc.)</h3>
-
-<p>The other embarassingly parallel design pattern at the heart of Beam
is per-key
-(and window) aggregation. Elements sharing a key are colocated and then
-combined using some associative and commutative operator. In Beam this is
-expressed as a <code
class="highlighter-rouge">GroupByKey</code> or <code
class="highlighter-rouge">Combine.perKey</code>, and
corresponds to the shuffle
-and “Reduce” from MapReduce. It is sometimes helpful to think of per-key
-<code class="highlighter-rouge">Combine</code> as the
fundamental operation, and raw <code
class="highlighter-rouge">GroupByKey</code> as a combiner
that
-just concatenates input elements. The communication pattern for the input
-elements is the same, modulo some optimizations possible for <code
class="highlighter-rouge">Combine</code>.</p>
-
-<p>In the illustration here, recall that the color of each element
represents the
-key. So all of the red squares are routed to the same location where they are
-aggregated and the red triangle is the output. Likewise for the yellow and
-green squares, etc. In a real application, you may have millions of keys, so
-the parallelism is still massive.</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/CombinePerKey.png"
alt="Gathering elements per key then combining them"
width="600" /></p>
-
-<p>The underlying data processing engine will, at some level of
abstraction, use
-state to perform this aggregation across all the elements arriving for a key.
-In particular, in a streaming execution, the aggregation process may need to
-wait for more data to arrive or for the watermark to estimate that all input
-for an event time window is complete. This requires some way to store the
-intermediate aggregation between input elements as well a way to a receive a
-callback when it is time to emit the result. As a result, the
<em>execution</em> of
-per key aggregation by a stream processing engine fundamentally involves state
-and timers.</p>
-
-<p>However, <em>your</em> code is just a declarative
expression of the aggregation
-operator. The runner can choose a variety of ways to execute your operator.
-I went over this in detail in <a
href="/blog/2017/02/13/stateful-processing.html">my prior post
focused on state alone</a>. Since you do not
-observe elements in any defined order, nor manipulate mutable state or timers
-directly, I call this neither stateful nor timely processing.</p>
-
-<h3
id="per-key-and-window-stateful-timely-processing">Per-key-and-window
stateful, timely processing</h3>
-
-<p>Both <code
class="highlighter-rouge">ParDo</code> and <code
class="highlighter-rouge">Combine.perKey</code> are standard
patterns for parallelism that go
-back decades. When implementing these in a massive-scale distributed data
-processing engine, we can highlight a few characteristics that are particularly
-important.</p>
-
-<p>Let us consider these characteristics of <code
class="highlighter-rouge">ParDo</code>:</p>
-
-<ul>
- <li>You write single-threaded code to process one element.</li>
- <li>Elements are processed in arbitrary order with no dependencies
-or interaction between processing of elements.</li>
-</ul>
-
-<p>And these characteristics for <code
class="highlighter-rouge">Combine.perKey</code>:</p>
-
-<ul>
- <li>Elements for a common key and window are gathered
together.</li>
- <li>A user-defined operator is applied to those elements.</li>
-</ul>
-
-<p>Combining some of the characteristics of unrestricted parallel
mapping and
-per-key-and-window combination, we can discern a megaprimitive from which we
-build stateful and timely processing:</p>
-
-<ul>
- <li>Elements for a common key and window are gathered
together.</li>
- <li>Elements are processed in arbitrary order.</li>
- <li>You write single-threaded code to process one element or timer,
possibly
-accessing state or setting timers.</li>
-</ul>
-
-<p>In the illustration below, the red squares are gathered and fed one
by one to
-the stateful, timely, <code
class="highlighter-rouge">DoFn</code>. As each element is
processed, the <code
class="highlighter-rouge">DoFn</code> has
-access to state (the color-partitioned cylinder on the right) and can set
-timers to receive callbacks (the colorful clocks on the left).</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/StateAndTimers.png"
alt="Gathering elements per key then timely, stateful processing"
width="600" /></p>
-
-<p>So that is the abstract notion of per-key-and-window stateful, timely
-processing in Apache Beam. Now let’s see what it looks like to write code that
-accesses state, sets timers, and receives callbacks.</p>
-
-<h2 id="example-batched-rpc">Example: Batched RPC</h2>
-
-<p>To demonstrate stateful and timely processing, let’s work through a
concrete
-example, with code.</p>
-
-<p>Suppose you are writing a system to analyze events. You have a ton
of data
-coming in and you need to enrich each event by RPC to an external system. You
-can’t just issue an RPC per event. Not only would this be terrible for
-performance, but it would also likely blow your quota with the external system.
-So you’d like to gather a number of events, make one RPC for them all, and then
-output all the enriched events.</p>
-
-<h3 id="state">State</h3>
-
-<p>Let’s set up the state we need to track batches of elements. As each
element
-comes in, we will write the element to a buffer while tracking the number of
-elements we have buffered. Here are the state cells in code:</p>
-
-<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span
class="k">new</span> <span
class="n">DoFn</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">,</span> <span
class="n">EnrichedEvent</span><span
class="o">&gt;()</span> <span class=&qu [...]
-
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">StateSpec</span><span
class="o">&lt;</span><span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;&gt;</span> <span
class="n">bufferedEvents</span> [...]
-
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">StateSpec</span><span
class="o">&lt;</span><span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;&gt;</span> <span
class="n">countState</span> [...]
-
- <span class="err">…</span> <span
class="n">TBD</span> <span
class="err">…</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre
class="highlight"><code><span class="c">#
State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Follow
https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
-</code></pre>
-</div>
-
-<p>Walking through the code, we have:</p>
-
-<ul>
- <li>The state cell <code
class="highlighter-rouge">"buffer"</code> is an
unordered bag of buffered events.</li>
- <li>The state cell <code
class="highlighter-rouge">"count"</code> tracks
how many events have been buffered.</li>
-</ul>
-
-<p>Next, as a recap of reading and writing state, let’s write our
<code class="highlighter-rouge">@ProcessElement</code>
-method. We will choose a limit on the size of the buffer, <code
class="highlighter-rouge">MAX_BUFFER_SIZE</code>. If
-our buffer reaches this size, we will perform a single RPC to enrich all the
-events, and output.</p>
-
-<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span
class="k">new</span> <span
class="n">DoFn</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">,</span> <span
class="n">EnrichedEvent</span><span
class="o">&gt;()</span> <span class=&qu [...]
-
- <span class="kd">private</span> <span
class="kd">static</span> <span
class="kd">final</span> <span
class="kt">int</span> <span
class="n">MAX_BUFFER_SIZE</span> <span
class="o">=</span> <span
class="mi">500</span><span
class="o">;</span>
-
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">StateSpec</span><span
class="o">&lt;</span><span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;&gt;</span> <span
class="n">bufferedEvents</span> [...]
-
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">StateSpec</span><span
class="o">&lt;</span><span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;&gt;</span> <span
class="n">countState</span> [...]
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">process</span><span
class="o">(</span>
- <span class="n">ProcessContext</span> <span
class="n">context</span><span
class="o">,</span>
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span> <span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;</span> <span
class="n">bufferState</span><span c [...]
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span> <span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;</span> <span
class="n">countState</span><span [...]
-
- <span class="kt">int</span> <span
class="n">count</span> <span
class="o">=</span> <span
class="n">firstNonNull</span><span
class="o">(</span><span
class="n">countState</span><span
class="o">.</span><span
class="na">read</span><span
class="o">(),</span> <span class="mi">0<
[...]
- <span class="n">count</span> <span
class="o">=</span> <span
class="n">count</span> <span
class="o">+</span> <span
class="mi">1</span><span
class="o">;</span>
- <span class="n">countState</span><span
class="o">.</span><span
class="na">write</span><span
class="o">(</span><span
class="n">count</span><span
class="o">);</span>
- <span class="n">bufferState</span><span
class="o">.</span><span
class="na">add</span><span
class="o">(</span><span
class="n">context</span><span
class="o">.</span><span
class="na">element</span><span
class="o">());</span>
-
- <span class="k">if</span> <span
class="o">(</span><span
class="n">count</span> <span
class="o">&gt;</span> <span
class="n">MAX_BUFFER_SIZE</span><span
class="o">)</span> <span
class="o">{</span>
- <span class="k">for</span> <span
class="o">(</span><span
class="n">EnrichedEvent</span> <span
class="n">enrichedEvent</span> <span
class="o">:</span> <span
class="n">enrichEvents</span><span
class="o">(</span><span
class="n">bufferState</span><span
class="o">.</span><span class=" [...]
- <span class="n">context</span><span
class="o">.</span><span
class="na">output</span><span
class="o">(</span><span
class="n">enrichedEvent</span><span
class="o">);</span>
- <span class="o">}</span>
- <span class="n">bufferState</span><span
class="o">.</span><span
class="na">clear</span><span
class="o">();</span>
- <span class="n">countState</span><span
class="o">.</span><span
class="na">clear</span><span
class="o">();</span>
- <span class="o">}</span>
- <span class="o">}</span>
-
- <span class="err">…</span> <span
class="n">TBD</span> <span
class="err">…</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre
class="highlight"><code><span class="c">#
State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Follow
https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
-</code></pre>
-</div>
-
-<p>Here is an illustration to accompany the code:</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/BatchedRpcState.png"
alt="Batching elements in state, then performing RPCs"
width="600" /></p>
-
-<ul>
- <li>The blue box is the <code
class="highlighter-rouge">DoFn</code>.</li>
- <li>The yellow box within it is the <code
class="highlighter-rouge">@ProcessElement</code>
method.</li>
- <li>Each input event is a red square - this diagram just shows the
activity for
-a single key, represented by the color red. Your <code
class="highlighter-rouge">DoFn</code> will run the same
-workflow in parallel for all keys which are perhaps user IDs.</li>
- <li>Each input event is written to the buffer as a red triangle,
representing
-the fact that you might actually buffer more than just the raw input, even
-though this code doesn’t.</li>
- <li>The external service is drawn as a cloud. When there are enough
buffered
-events, the <code
class="highlighter-rouge">@ProcessElement</code> method
reads the events from state and issues
-a single RPC.</li>
- <li>Each output enriched event is drawn as a red circle. To consumers
of this
-output, it looks just like an element-wise operation.</li>
-</ul>
-
-<p>So far, we have only used state, but not timers. You may have noticed
that
-there is a problem - there will usually be data left in the buffer. If no more
-input arrives, that data will never be processed. In Beam, every window has
-some point in event time when any further input for the window is considered
-too late and is discarded. At this point, we say that the window has “expired”.
-Since no further input can arrive to access the state for that window, the
-state is also discarded. For our example, we need to ensure that all leftover
-events are output when the window expires.</p>
-
-<h3 id="event-time-timers">Event Time Timers</h3>
-
-<p>An event time timer requests a call back when the watermark for an
input
-<code class="highlighter-rouge">PCollection</code>
reaches some threshold. In other words, you can use an event time
-timer to take action at a specific moment in event time - a particular point of
-completeness for a <code
class="highlighter-rouge">PCollection</code> - such as when
a window expires.</p>
-
-<p>For our example, let us add an event time timer so that when the
window expires,
-any events remaining in the buffer are processed.</p>
-
-<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span
class="k">new</span> <span
class="n">DoFn</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">,</span> <span
class="n">EnrichedEvent</span><span
class="o">&gt;()</span> <span class=&qu [...]
- <span class="err">…</span>
-
- <span class="nd">@TimerId</span><span
class="o">(</span><span
class="s">"expiry"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">TimerSpec</span> <span
class="n">expirySpec</span> <span
class="o">=</span> <span
class="n">TimerSpecs</span><span
class="o">.</span><span
class="na">timer</span><span
class="o">(</span><span class="n" [...]
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">process</span><span
class="o">(</span>
- <span class="n">ProcessContext</span> <span
class="n">context</span><span
class="o">,</span>
- <span class="n">BoundedWindow</span> <span
class="n">window</span><span
class="o">,</span>
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span> <span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;</span> <span
class="n">bufferState</span><span c [...]
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span> <span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;</span> <span
class="n">countState</span><span [...]
- <span class="nd">@TimerId</span><span
class="o">(</span><span
class="s">"expiry"</span><span
class="o">)</span> <span
class="n">Timer</span> <span
class="n">expiryTimer</span><span
class="o">)</span> <span
class="o">{</span>
-
- <span class="n">expiryTimer</span><span
class="o">.</span><span
class="na">set</span><span
class="o">(</span><span
class="n">window</span><span
class="o">.</span><span
class="na">maxTimestamp</span><span
class="o">().</span><span
class="na">plus</span><span class="o">(</
[...]
-
- <span class="err">…</span> <span
class="n">same</span> <span
class="n">logic</span> <span
class="n">as</span> <span
class="n">above</span> <span
class="err">…</span>
- <span class="o">}</span>
-
- <span class="nd">@OnTimer</span><span
class="o">(</span><span
class="s">"expiry"</span><span
class="o">)</span>
- <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">onExpiry</span><span
class="o">(</span>
- <span class="n">OnTimerContext</span> <span
class="n">context</span><span
class="o">,</span>
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span> <span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;</span> <span
class="n">bufferState</span><span c [...]
- <span class="k">if</span> <span
class="o">(!</span><span
class="n">bufferState</span><span
class="o">.</span><span
class="na">isEmpty</span><span
class="o">().</span><span
class="na">read</span><span
class="o">())</span> <span
class="o">{</span>
- <span class="k">for</span> <span
class="o">(</span><span
class="n">EnrichedEvent</span> <span
class="n">enrichedEvent</span> <span
class="o">:</span> <span
class="n">enrichEvents</span><span
class="o">(</span><span
class="n">bufferState</span><span
class="o">.</span><span class=" [...]
- <span class="n">context</span><span
class="o">.</span><span
class="na">output</span><span
class="o">(</span><span
class="n">enrichedEvent</span><span
class="o">);</span>
- <span class="o">}</span>
- <span class="o">}</span>
- <span class="o">}</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre
class="highlight"><code><span class="c">#
State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Follow
https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
-</code></pre>
-</div>
-
-<p>Let’s unpack the pieces of this snippet:</p>
-
-<ul>
- <li>
- <p>We declare an event time timer with <code
class="highlighter-rouge">@TimerId("expiry")</code>.
We will use the
-identifier <code
class="highlighter-rouge">"expiry"</code> to
identify the timer for setting the callback time as
-well as receiving the callback.</p>
- </li>
- <li>
- <p>The variable <code
class="highlighter-rouge">expiryTimer</code>, annotated with
<code class="highlighter-rouge">@TimerId</code>, is set
to the value
-<code
class="highlighter-rouge">TimerSpecs.timer(TimeDomain.EVENT_TIME)</code>,
indicating that we want a
-callback according to the event time watermark of the input elements.</p>
- </li>
- <li>
- <p>In the <code
class="highlighter-rouge">@ProcessElement</code> element we
annotate a parameter <code
class="highlighter-rouge">@TimerId("expiry")
-Timer</code>. The Beam runner automatically provides this <code
class="highlighter-rouge">Timer</code> parameter by which
-we can set (and reset) the timer. It is inexpensive to reset a timer
-repeatedly, so we simply set it on every element.</p>
- </li>
- <li>
- <p>We define the <code
class="highlighter-rouge">onExpiry</code> method, annotated
with <code
class="highlighter-rouge">@OnTimer("expiry")</code>,
that
-performs a final event enrichment RPC and outputs the result. The Beam runner
-delivers the callback to this method by matching its identifier.</p>
- </li>
-</ul>
-
-<p>Illustrating this logic, we have the diagram below:</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/BatchedRpcExpiry.png"
alt="Batched RPCs with window expiration" width="600"
/></p>
-
-<p>Both the <code
class="highlighter-rouge">@ProcessElement</code> and
<code
class="highlighter-rouge">@OnTimer("expiry")</code>
methods perform the same
-access to buffered state, perform the same batched RPC, and output enriched
-elements.</p>
-
-<p>Now, if we are executing this in a streaming real-time manner, we
might still
-have unbounded latency for particular buffered data. If the watermark is
advancing
-very slowly, or event time windows are chosen to be quite large, then a lot of
-time might pass before output is emitted based either on enough elements or
-window expiration. We can also use timers to limit the amount of wall-clock
-time, aka processing time, before we process buffered elements. We can choose
-some reasonable amount of time so that even though we are issuing RPCs that are
-not as large as they might be, it is still few enough RPCs to avoid blowing our
-quota with the external service.</p>
-
-<h3 id="processing-time-timers">Processing Time
Timers</h3>
-
-<p>A timer in processing time (time as it passes while your pipeline is
executing)
-is intuitively simple: you want to wait a certain amount of time and then
-receive a call back.</p>
-
-<p>To put the finishing touches on our example, we will set a processing
time
-timer as soon as any data is buffered. We track whether or not the timer has
-been set so we don’t continually reset it. When an element arrives, if the
-timer has not been set, then we set it for the current moment plus
-<code
class="highlighter-rouge">MAX_BUFFER_DURATION</code>. After
the allotted processing time has passed, a
-callback will fire and enrich and emit any buffered elements.</p>
-
-<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span
class="k">new</span> <span
class="n">DoFn</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">,</span> <span
class="n">EnrichedEvent</span><span
class="o">&gt;()</span> <span class=&qu [...]
- <span class="err">…</span>
-
- <span class="kd">private</span> <span
class="kd">static</span> <span
class="kd">final</span> <span
class="n">Duration</span> <span
class="n">MAX_BUFFER_DURATION</span> <span
class="o">=</span> <span
class="n">Duration</span><span
class="o">.</span><span
class="na">standardSeconds</span><s [...]
-
- <span class="nd">@TimerId</span><span
class="o">(</span><span
class="s">"stale"</span><span
class="o">)</span>
- <span class="kd">private</span> <span
class="kd">final</span> <span
class="n">TimerSpec</span> <span
class="n">staleSpec</span> <span
class="o">=</span> <span
class="n">TimerSpecs</span><span
class="o">.</span><span
class="na">timer</span><span
class="o">(</span><span class="n"& [...]
-
- <span class="nd">@ProcessElement</span>
- <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">process</span><span
class="o">(</span>
- <span class="n">ProcessContext</span> <span
class="n">context</span><span
class="o">,</span>
- <span class="n">BoundedWindow</span> <span
class="n">window</span><span
class="o">,</span>
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span> <span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;</span> <span
class="n">countState</span><span [...]
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span> <span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;</span> <span
class="n">bufferState</span><span c [...]
- <span class="nd">@TimerId</span><span
class="o">(</span><span
class="s">"stale"</span><span
class="o">)</span> <span
class="n">Timer</span> <span
class="n">staleTimer</span><span
class="o">,</span>
- <span class="nd">@TimerId</span><span
class="o">(</span><span
class="s">"expiry"</span><span
class="o">)</span> <span
class="n">Timer</span> <span
class="n">expiryTimer</span><span
class="o">)</span> <span
class="o">{</span>
-
- <span class="kt">boolean</span> <span
class="n">staleTimerSet</span> <span
class="o">=</span> <span
class="n">firstNonNull</span><span
class="o">(</span><span
class="n">staleSetState</span><span
class="o">.</span><span
class="na">read</span><span
class="o">(),</span> <span class="k [...]
- <span class="k">if</span> <span
class="o">(</span><span
class="n">firstNonNull</span><span
class="o">(</span><span
class="n">countState</span><span
class="o">.</span><span
class="na">read</span><span
class="o">(),</span> <span
class="mi">0</span><span
class="o">)</span&g [...]
- <span class="n">staleTimer</span><span
class="o">.</span><span
class="na">offset</span><span
class="o">(</span><span
class="n">MAX_BUFFER_DURATION</span><span
class="o">).</span><span
class="na">setRelative</span><span
class="o">());</span>
- <span class="o">}</span>
-
- <span class="err">…</span> <span
class="n">same</span> <span
class="n">processing</span> <span
class="n">logic</span> <span
class="n">as</span> <span
class="n">above</span> <span
class="err">…</span>
- <span class="o">}</span>
-
- <span class="nd">@OnTimer</span><span
class="o">(</span><span
class="s">"stale"</span><span
class="o">)</span>
- <span class="kd">public</span> <span
class="kt">void</span> <span
class="nf">onStale</span><span
class="o">(</span>
- <span class="n">OnTimerContext</span> <span
class="n">context</span><span
class="o">,</span>
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"buffer"</span><span
class="o">)</span> <span
class="n">BagState</span><span
class="o">&lt;</span><span
class="n">Event</span><span
class="o">&gt;</span> <span
class="n">bufferState</span><span c [...]
- <span class="nd">@StateId</span><span
class="o">(</span><span
class="s">"count"</span><span
class="o">)</span> <span
class="n">ValueState</span><span
class="o">&lt;</span><span
class="n">Integer</span><span
class="o">&gt;</span> <span
class="n">countState</span><span [...]
- <span class="k">if</span> <span
class="o">(!</span><span
class="n">bufferState</span><span
class="o">.</span><span
class="na">isEmpty</span><span
class="o">().</span><span
class="na">read</span><span
class="o">())</span> <span
class="o">{</span>
- <span class="k">for</span> <span
class="o">(</span><span
class="n">EnrichedEvent</span> <span
class="n">enrichedEvent</span> <span
class="o">:</span> <span
class="n">enrichEvents</span><span
class="o">(</span><span
class="n">bufferState</span><span
class="o">.</span><span class=" [...]
- <span class="n">context</span><span
class="o">.</span><span
class="na">output</span><span
class="o">(</span><span
class="n">enrichedEvent</span><span
class="o">);</span>
- <span class="o">}</span>
- <span class="n">bufferState</span><span
class="o">.</span><span
class="na">clear</span><span
class="o">();</span>
- <span class="n">countState</span><span
class="o">.</span><span
class="na">clear</span><span
class="o">();</span>
- <span class="o">}</span>
- <span class="o">}</span>
-
- <span class="err">…</span> <span
class="n">same</span> <span
class="n">expiry</span> <span
class="n">as</span> <span
class="n">above</span> <span
class="err">…</span>
-<span class="o">}</span>
-</code></pre>
-</div>
-
-<div class="language-py highlighter-rouge"><pre
class="highlight"><code><span class="c">#
State and timers are not yet supported in Beam's Python SDK.</span>
-<span class="c"># Follow
https://issues.apache.org/jira/browse/BEAM-2687 for updates.</span>
-</code></pre>
-</div>
-
-<p>Here is an illustration of the final code:</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/BatchedRpcStale.png"
alt="Batching elements in state, then performing RPCs"
width="600" /></p>
-
-<p>Recapping the entirety of the logic:</p>
-
-<ul>
- <li>As events arrive at <code
class="highlighter-rouge">@ProcessElement</code> they are
buffered in state.</li>
- <li>If the size of the buffer exceeds a maximum, the events are
enriched and output.</li>
- <li>If the buffer fills too slowly and the events get stale before the
maximum is reached,
-a timer causes a callback which enriches the buffered events and
outputs.</li>
- <li>Finally, as any window is expiring, any events buffered in that
window are
-processed and output prior to the state for that window being
discarded.</li>
-</ul>
-
-<p>In the end, we have a full example that uses state and timers to
explicitly
-manage the low-level details of a performance-sensitive transform in Beam. As
-we added more and more features, our <code
class="highlighter-rouge">DoFn</code> actually became pretty
large. That
-is a normal characteristic of stateful, timely processing. You are really
-digging in and managing a lot of details that are handled automatically when
-you express your logic using Beam’s higher-level APIs. What you gain from this
-extra effort is an ability to tackle use cases and achieve efficiencies that
-may not have been possible otherwise.</p>
-
-<h2 id="state-and-timers-in-beams-unified-model">State and
Timers in Beam’s Unified Model</h2>
-
-<p>Beam’s unified model for event time across streaming and batch
processing has
-novel implications for state and timers. Usually, you don’t need to do anything
-for your stateful and timely <code
class="highlighter-rouge">DoFn</code> to work well in the
Beam model. But it will
-help to be aware of the considerations below, especially if you have used
-similar features before outside of Beam.</p>
-
-<h3 id="event-time-windowing-just-works">Event Time Windowing
“Just Works”</h3>
-
-<p>One of the raisons d’etre for Beam is correct processing of
out-of-order event
-data, which is almost all event data. Beam’s solution to out-of-order data is
-event time windowing, where windows in event time yield correct results no
-matter what windowing a user chooses or what order the events come
in.</p>
-
-<p>If you write a stateful, timely transform, it should work no matter
how the
-surrounding pipeline chooses to window event time. If the pipeline chooses
-fixed windows of one hour (sometimes called tumbling windows) or windows of 30
-minutes sliding by 10 minutes, the stateful, timely transform should
-transparently work correctly.</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/WindowingChoices.png"
alt="Two windowing strategies for the same stateful and timely
transform" width="600" /></p>
-
-<p>This works in Beam automatically, because state and timers are
partitioned per
-key and window. Within each key and window, the stateful, timely processing is
-essentially independent. As an added benefit, the passing of event time (aka
-advancement of the watermark) allows automatic release of unreachable state
-when a window expires, so you often don’t have to worry about evicting old
-state.</p>
-
-<h3 id="unified-real-time-and-historical-processing">Unified
real-time and historical processing</h3>
-
-<p>A second tenet of Beam’s semantic model is that processing must be
unified
-between batch and streaming. One important use case for this unification
-is the ability to apply the same logic to a stream of events in real time and
-to archived storage of the same events.</p>
-
-<p>A common characteristic of archived data is that it may arrive
radically out of
-order. The sharding of archived files often results in a totally different
-ordering for processing than events coming in near-real-time. The data will
-also all be all available and hence delivered instantaneously from the point of
-view of your pipeline. Whether running experiments on past data or reprocessing
-past results to fix a data processing bug, it is critically important that your
-processing logic be applicable to archived events just as easily as incoming
-near-real-time data.</p>
-
-<p><img class="center-block"
src="/images/blog/timely-processing/UnifiedModel.png"
alt="Unified stateful processing over streams and file archives"
width="600" /></p>
-
-<p>It is (deliberately) possible to write a stateful and timely DoFn
that delivers
-results that depend on ordering or delivery timing, so in this sense there is
-additional burden on you, the <code
class="highlighter-rouge">DoFn</code> author, to ensure that
this nondeterminism
-falls within documented allowances.</p>
-
-<h2 id="go-use-it">Go use it!</h2>
-
-<p>I’ll end this post in the same way I ended the last. I hope you will
go try out
-Beam with stateful, timely processing. If it opens up new possibilities for
-you, then great! If not, we want to hear about it. Since this is a new feature,
-please check the <a
href="/documentation/runners/capability-matrix/">capability
matrix</a> to see the level of support for
-your preferred Beam backend(s).</p>
-
-<p>And please do join the Beam community at
-<a href="/get-started/support">u...@beam.apache.org</a>
and follow
-<a href="https://twitter.com/ApacheBeam">@ApacheBeam</a>
on Twitter.</p>
-</description>
- <pubDate>Mon, 28 Aug 2017 01:00:01 -0700</pubDate>
-
<link>https://beam.apache.org/blog/2017/08/28/timely-processing.html</link>
- <guid
isPermaLink="true">https://beam.apache.org/blog/2017/08/28/timely-processing.html</guid>
-
-
- <category>blog</category>
-
- </item>
-
</channel>
</rss>
diff --git a/website/generated-content/index.html
b/website/generated-content/index.html
index cc1e6bf..0ef189c 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/index.html
@@ -176,6 +176,11 @@ limitations under the License.
</div>
<div class="hero__blog__cards">
+ <a class="hero__blog__cards__card"
href="/blog/2018/12/13/beam-2.9.0.html">
+ <div class="hero__blog__cards__card__title">Apache Beam
2.9.0</div>
+ <div class="hero__blog__cards__card__date">Dec 13, 2018</div>
+ </a>
+
<a class="hero__blog__cards__card"
href="/blog/2018/10/31/beam-summit-aftermath.html">
<div class="hero__blog__cards__card__title">Inaugural edition of
the Beam Summit Europe 2018 - aftermath</div>
<div class="hero__blog__cards__card__date">Oct 31, 2018</div>
@@ -186,11 +191,6 @@ limitations under the License.
<div class="hero__blog__cards__card__date">Oct 29, 2018</div>
</a>
- <a class="hero__blog__cards__card"
href="/blog/2018/10/03/beam-2.7.0.html">
- <div class="hero__blog__cards__card__title">Apache Beam
2.7.0</div>
- <div class="hero__blog__cards__card__date">Oct 3, 2018</div>
- </a>
-
</div>
</div>
</div>
