11 17:15:01 at commit 40ccf08

git-site-role Wed, 11 Sep 2019 10:15:25 -0700

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git



The following commit(s) were added to refs/heads/asf-site by this push:
     new 67daf41  Publishing website 2019/09/11 17:15:01 at commit 40ccf08
67daf41 is described below

commit 67daf41daf5cf82751ee998261f2234c5a22dde4
Author: jenkins <bui...@apache.org>
AuthorDate: Wed Sep 11 17:15:01 2019 +0000

    Publishing website 2019/09/11 17:15:01 at commit 40ccf08
---
 .../transforms/python/elementwise/pardo/index.html | 306 ++++++++++++++++++++-
 1 file changed, 293 insertions(+), 13 deletions(-)

diff --git 
a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
 
b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
index cac7dd6..eb7ac6d 100644
--- 
a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
+++ 
b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
@@ -447,7 +447,13 @@
 
 
 <ul class="nav">
-  <li><a href="#examples">Examples</a></li>
+  <li><a href="#examples">Examples</a>
+    <ul>
+      <li><a href="#example-1-pardo-with-a-simple-dofn">Example 1: ParDo with 
a simple DoFn</a></li>
+      <li><a 
href="#example-2-pardo-with-timestamp-and-window-information">Example 2: ParDo 
with timestamp and window information</a></li>
+      <li><a href="#example-3-pardo-with-dofn-methods">Example 3: ParDo with 
DoFn methods</a></li>
+    </ul>
+  </li>
   <li><a href="#related-transforms">Related transforms</a></li>
 </ul>
 
@@ -470,30 +476,304 @@ limitations under the License.
 -->
 
 <h1 id="pardo">ParDo</h1>
-<table align="left">
-    <a target="_blank" class="button" 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo";>
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+<table>
+  <td>
+    <a class="button" target="_blank" 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo";>
       <img src="https://beam.apache.org/images/logos/sdks/python.png"; 
width="20px" height="20px" alt="Pydoc" />
-     Pydoc
+      Pydoc
     </a>
+  </td>
 </table>
-<p><br />
-A transform for generic parallel processing. A <code 
class="highlighter-rouge">ParDo</code> transform considers each
-element in the input <code class="highlighter-rouge">PCollection</code>, 
performs some processing function
-(your user code) on that element, and emits zero or more elements to
-an output PCollection.</p>
+<p><br /></p>
+
+<p>A transform for generic parallel processing.
+A <code class="highlighter-rouge">ParDo</code> transform considers each 
element in the input <code class="highlighter-rouge">PCollection</code>,
+performs some processing function (your user code) on that element,
+and emits zero or more elements to an output <code 
class="highlighter-rouge">PCollection</code>.</p>
 
-<p>See more information in the <a 
href="/documentation/programming-guide/#pardo">Beam Programming Guide</a>.</p>
+<p>See more information in the
+<a href="/documentation/programming-guide/#pardo">Beam Programming 
Guide</a>.</p>
 
 <h2 id="examples">Examples</h2>
-<p>See <a href="https://issues.apache.org/jira/browse/BEAM-7389";>BEAM-7389</a> 
for updates.</p>
+
+<p>In the following examples, we explore how to create custom <code 
class="highlighter-rouge">DoFn</code>s and access
+the timestamp and windowing information.</p>
+
+<h3 id="example-1-pardo-with-a-simple-dofn">Example 1: ParDo with a simple 
DoFn</h3>
+
+<p>The following example defines a simple <code 
class="highlighter-rouge">DoFn</code> class called <code 
class="highlighter-rouge">SplitWords</code>
+which stores the <code class="highlighter-rouge">delimiter</code> as an object 
field.
+The <code class="highlighter-rouge">process</code> method is called once per 
element,
+and it can yield zero or more output elements.</p>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span 
class="kn">import</span> <span class="nn">apache_beam</span> <span 
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">class</span> <span class="nc">SplitWords</span><span 
class="p">(</span><span class="n">beam</span><span class="o">.</span><span 
class="n">DoFn</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">__init__</span><span 
class="p">(</span><span class="bp">self</span><span class="p">,</span> <span 
class="n">delimiter</span><span class="o">=</span><span 
class="s">','</span><span class="p">):</span>
+    <span class="bp">self</span><span class="o">.</span><span 
class="n">delimiter</span> <span class="o">=</span> <span 
class="n">delimiter</span>
+
+  <span class="k">def</span> <span class="nf">process</span><span 
class="p">(</span><span class="bp">self</span><span class="p">,</span> <span 
class="n">text</span><span class="p">):</span>
+    <span class="k">for</span> <span class="n">word</span> <span 
class="ow">in</span> <span class="n">text</span><span class="o">.</span><span 
class="n">split</span><span class="p">(</span><span class="bp">self</span><span 
class="o">.</span><span class="n">delimiter</span><span class="p">):</span>
+      <span class="k">yield</span> <span class="n">word</span>
+
+<span class="k">with</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> 
<span class="k">as</span> <span class="n">pipeline</span><span 
class="p">:</span>
+  <span class="n">plants</span> <span class="o">=</span> <span 
class="p">(</span>
+      <span class="n">pipeline</span>
+      <span class="o">|</span> <span class="s">'Gardening plants'</span> <span 
class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Create</span><span class="p">([</span>
+          <span class="s">'🍓Strawberry,🥕Carrot,🍆Eggplant'</span><span 
class="p">,</span>
+          <span class="s">'🍅Tomato,🥔Potato'</span><span class="p">,</span>
+      <span class="p">])</span>
+      <span class="o">|</span> <span class="s">'Split words'</span> <span 
class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span 
class="n">SplitWords</span><span class="p">(</span><span 
class="s">','</span><span class="p">))</span>
+      <span class="o">|</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Map</span><span class="p">(</span><span 
class="k">print</span><span class="p">)</span>
+  <span class="p">)</span>
+</code></pre>
+</div>
+
+<p>Output <code class="highlighter-rouge">PCollection</code> after <code 
class="highlighter-rouge">ParDo</code>:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>plants = [
+    '🍓Strawberry',
+    '🥕Carrot',
+    '🍆Eggplant',
+    '🍅Tomato',
+    '🥔Potato',
+]
+</code></pre>
+</div>
+
+<table>
+  <td>
+    <a class="button" target="_blank" 
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py";>
+      <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"; 
width="20px" height="20px" alt="View on GitHub" />
+      View on GitHub
+    </a>
+  </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-2-pardo-with-timestamp-and-window-information">Example 2: 
ParDo with timestamp and window information</h3>
+
+<p>In this example, we add new parameters to the <code 
class="highlighter-rouge">process</code> method to bind parameter values at 
runtime.</p>
+
+<ul>
+  <li><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam";><code
 class="highlighter-rouge">beam.DoFn.TimestampParam</code></a>
+binds the timestamp information as an
+<a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp";><code
 class="highlighter-rouge">apache_beam.utils.timestamp.Timestamp</code></a>
+object.</li>
+  <li><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam";><code
 class="highlighter-rouge">beam.DoFn.WindowParam</code></a>
+binds the window information as the appropriate
+<a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html";><code
 class="highlighter-rouge">apache_beam.transforms.window.*Window</code></a>
+object.</li>
+</ul>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span 
class="kn">import</span> <span class="nn">apache_beam</span> <span 
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">class</span> <span class="nc">AnalyzeElement</span><span 
class="p">(</span><span class="n">beam</span><span class="o">.</span><span 
class="n">DoFn</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">process</span><span 
class="p">(</span><span class="bp">self</span><span class="p">,</span> <span 
class="n">elem</span><span class="p">,</span> <span 
class="n">timestamp</span><span class="o">=</span><span 
class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span 
class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> 
<span class="n">window</span><span class="o">=</span><span 
class="n">beam</span><sp [...]
+    <span class="k">yield</span> <span class="s">'</span><span 
class="se">\n</span><span class="s">'</span><span class="o">.</span><span 
class="n">join</span><span class="p">([</span>
+        <span class="s">'# timestamp'</span><span class="p">,</span>
+        <span class="s">'type(timestamp) -&gt; '</span> <span 
class="o">+</span> <span class="nb">repr</span><span class="p">(</span><span 
class="nb">type</span><span class="p">(</span><span 
class="n">timestamp</span><span class="p">)),</span>
+        <span class="s">'timestamp.micros -&gt; '</span> <span 
class="o">+</span> <span class="nb">repr</span><span class="p">(</span><span 
class="n">timestamp</span><span class="o">.</span><span 
class="n">micros</span><span class="p">),</span>
+        <span class="s">'timestamp.to_rfc3339() -&gt; '</span> <span 
class="o">+</span> <span class="nb">repr</span><span class="p">(</span><span 
class="n">timestamp</span><span class="o">.</span><span 
class="n">to_rfc3339</span><span class="p">()),</span>
+        <span class="s">'timestamp.to_utc_datetime() -&gt; '</span> <span 
class="o">+</span> <span class="nb">repr</span><span class="p">(</span><span 
class="n">timestamp</span><span class="o">.</span><span 
class="n">to_utc_datetime</span><span class="p">()),</span>
+        <span class="s">''</span><span class="p">,</span>
+        <span class="s">'# window'</span><span class="p">,</span>
+        <span class="s">'type(window) -&gt; '</span> <span class="o">+</span> 
<span class="nb">repr</span><span class="p">(</span><span 
class="nb">type</span><span class="p">(</span><span 
class="n">window</span><span class="p">)),</span>
+        <span class="s">'window.start -&gt; {} ({})'</span><span 
class="o">.</span><span class="n">format</span><span class="p">(</span><span 
class="n">window</span><span class="o">.</span><span 
class="n">start</span><span class="p">,</span> <span 
class="n">window</span><span class="o">.</span><span 
class="n">start</span><span class="o">.</span><span 
class="n">to_utc_datetime</span><span class="p">()),</span>
+        <span class="s">'window.end -&gt; {} ({})'</span><span 
class="o">.</span><span class="n">format</span><span class="p">(</span><span 
class="n">window</span><span class="o">.</span><span class="n">end</span><span 
class="p">,</span> <span class="n">window</span><span class="o">.</span><span 
class="n">end</span><span class="o">.</span><span 
class="n">to_utc_datetime</span><span class="p">()),</span>
+        <span class="s">'window.max_timestamp() -&gt; {} ({})'</span><span 
class="o">.</span><span class="n">format</span><span class="p">(</span><span 
class="n">window</span><span class="o">.</span><span 
class="n">max_timestamp</span><span class="p">(),</span> <span 
class="n">window</span><span class="o">.</span><span 
class="n">max_timestamp</span><span class="p">()</span><span 
class="o">.</span><span class="n">to_utc_datetime</span><span 
class="p">()),</span>
+    <span class="p">])</span>
+
+<span class="k">with</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> 
<span class="k">as</span> <span class="n">pipeline</span><span 
class="p">:</span>
+  <span class="n">dofn_params</span> <span class="o">=</span> <span 
class="p">(</span>
+      <span class="n">pipeline</span>
+      <span class="o">|</span> <span class="s">'Create a single test 
element'</span> <span class="o">&gt;&gt;</span> <span 
class="n">beam</span><span class="o">.</span><span class="n">Create</span><span 
class="p">([</span><span class="s">':)'</span><span class="p">])</span>
+      <span class="o">|</span> <span class="s">'Add timestamp (Spring equinox 
2020)'</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Map</span><span class="p">(</span>
+          <span class="k">lambda</span> <span class="n">elem</span><span 
class="p">:</span> <span class="n">beam</span><span class="o">.</span><span 
class="n">window</span><span class="o">.</span><span 
class="n">TimestampedValue</span><span class="p">(</span><span 
class="n">elem</span><span class="p">,</span> <span 
class="mi">1584675660</span><span class="p">))</span>
+      <span class="o">|</span> <span class="s">'Fixed 30sec windows'</span> 
<span class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">WindowInto</span><span 
class="p">(</span><span class="n">beam</span><span class="o">.</span><span 
class="n">window</span><span class="o">.</span><span 
class="n">FixedWindows</span><span class="p">(</span><span 
class="mi">30</span><span class="p">))</span>
+      <span class="o">|</span> <span class="s">'Analyze element'</span> <span 
class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span 
class="n">AnalyzeElement</span><span class="p">())</span>
+      <span class="o">|</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Map</span><span class="p">(</span><span 
class="k">print</span><span class="p">)</span>
+  <span class="p">)</span>
+</code></pre>
+</div>
+
+<p><code class="highlighter-rouge">stdout</code> output:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code># timestamp
+type(timestamp) -&gt; &lt;class 'apache_beam.utils.timestamp.Timestamp'&gt;
+timestamp.micros -&gt; 1584675660000000
+timestamp.to_rfc3339() -&gt; '2020-03-20T03:41:00Z'
+timestamp.to_utc_datetime() -&gt; datetime.datetime(2020, 3, 20, 3, 41)
+
+# window
+type(window) -&gt; &lt;class 'apache_beam.transforms.window.IntervalWindow'&gt;
+window.start -&gt; Timestamp(1584675660) (2020-03-20 03:41:00)
+window.end -&gt; Timestamp(1584675690) (2020-03-20 03:41:30)
+window.max_timestamp() -&gt; Timestamp(1584675689.999999) (2020-03-20 
03:41:29.999999)
+</code></pre>
+</div>
+
+<table>
+  <td>
+    <a class="button" target="_blank" 
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py";>
+      <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"; 
width="20px" height="20px" alt="View on GitHub" />
+      View on GitHub
+    </a>
+  </td>
+</table>
+<p><br /></p>
+
+<h3 id="example-3-pardo-with-dofn-methods">Example 3: ParDo with DoFn 
methods</h3>
+
+<p>A <a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn";><code
 class="highlighter-rouge">DoFn</code></a>
+can be customized with a number of methods that can help create more complex 
behaviors.
+You can customize what a worker does when it starts and shuts down with <code 
class="highlighter-rouge">setup</code> and <code 
class="highlighter-rouge">teardown</code>.
+You can also customize what to do when a
+<a 
href="https://beam.apache.org/documentation/execution-model/#bundling-and-persistence";><em>bundle
 of elements</em></a>
+starts and finishes with <code class="highlighter-rouge">start_bundle</code> 
and <code class="highlighter-rouge">finish_bundle</code>.</p>
+
+<ul>
+  <li>
+    <p><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup";><code
 class="highlighter-rouge">DoFn.setup()</code></a>:
+Called <em>once per <code class="highlighter-rouge">DoFn</code> instance</em> 
when the <code class="highlighter-rouge">DoFn</code> instance is initialized.
+<code class="highlighter-rouge">setup</code> need not to be cached, so it 
could be called more than once per worker.
+This is a good place to connect to database instances, open network 
connections or other resources.</p>
+  </li>
+  <li>
+    <p><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle";><code
 class="highlighter-rouge">DoFn.start_bundle()</code></a>:
+Called <em>once per bundle of elements</em> before calling <code 
class="highlighter-rouge">process</code> on the first element of the bundle.
+This is a good place to start keeping track of the bundle elements.</p>
+  </li>
+  <li>
+    <p><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process";><strong><code
 class="highlighter-rouge">DoFn.process(element, *args, 
**kwargs)</code></strong></a>:
+Called <em>once per element</em>, can <em>yield zero or more elements</em>.
+Additional <code class="highlighter-rouge">*args</code> or <code 
class="highlighter-rouge">**kwargs</code> can be passed through
+<a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo";><code
 class="highlighter-rouge">beam.ParDo()</code></a>.
+<strong>[required]</strong></p>
+  </li>
+  <li>
+    <p><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle";><code
 class="highlighter-rouge">DoFn.finish_bundle()</code></a>:
+Called <em>once per bundle of elements</em> after calling <code 
class="highlighter-rouge">process</code> after the last element of the bundle,
+can <em>yield zero or more elements</em>. This is a good place to do batch 
calls on a bundle of elements,
+such as running a database query.</p>
+
+    <p>For example, you can initialize a batch in <code 
class="highlighter-rouge">start_bundle</code>,
+add elements to the batch in <code class="highlighter-rouge">process</code> 
instead of yielding them,
+then running a batch query on those elements on <code 
class="highlighter-rouge">finish_bundle</code>, and yielding all the 
results.</p>
+
+    <p>Note that yielded elements from <code 
class="highlighter-rouge">finish_bundle</code> must be of the type
+<a 
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py";><code
 
class="highlighter-rouge">apache_beam.utils.windowed_value.WindowedValue</code></a>.
+You need to provide a timestamp as a unix timestamp, which you can get from 
the last processed element.
+You also need to provide a window, which you can get from the last processed 
element like in the example below.</p>
+  </li>
+  <li>
+    <p><a 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown";><code
 class="highlighter-rouge">DoFn.teardown()</code></a>:
+Called <em>once (as a best effort) per <code 
class="highlighter-rouge">DoFn</code> instance</em> when the <code 
class="highlighter-rouge">DoFn</code> instance is shutting down.
+This is a good place to close database instances, close network connections or 
other resources.</p>
+
+    <p>Note that <code class="highlighter-rouge">teardown</code> is called as 
a <em>best effort</em> and is <em>not guaranteed</em>.
+For example, if the worker crashes, <code 
class="highlighter-rouge">teardown</code> might not be called.</p>
+  </li>
+</ul>
+
+<div class="language-py highlighter-rouge"><pre class="highlight"><code><span 
class="kn">import</span> <span class="nn">apache_beam</span> <span 
class="kn">as</span> <span class="nn">beam</span>
+
+<span class="k">class</span> <span class="nc">DoFnMethods</span><span 
class="p">(</span><span class="n">beam</span><span class="o">.</span><span 
class="n">DoFn</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">__init__</span><span 
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">print</span><span class="p">(</span><span 
class="s">'__init__'</span><span class="p">)</span>
+    <span class="bp">self</span><span class="o">.</span><span 
class="n">window</span> <span class="o">=</span> <span 
class="n">beam</span><span class="o">.</span><span class="n">window</span><span 
class="o">.</span><span class="n">GlobalWindow</span><span class="p">()</span>
+
+  <span class="k">def</span> <span class="nf">setup</span><span 
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">print</span><span class="p">(</span><span 
class="s">'setup'</span><span class="p">)</span>
+
+  <span class="k">def</span> <span class="nf">start_bundle</span><span 
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">print</span><span class="p">(</span><span 
class="s">'start_bundle'</span><span class="p">)</span>
+
+  <span class="k">def</span> <span class="nf">process</span><span 
class="p">(</span><span class="bp">self</span><span class="p">,</span> <span 
class="n">element</span><span class="p">,</span> <span 
class="n">window</span><span class="o">=</span><span class="n">beam</span><span 
class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span 
class="n">WindowParam</span><span class="p">):</span>
+    <span class="bp">self</span><span class="o">.</span><span 
class="n">window</span> <span class="o">=</span> <span class="n">window</span>
+    <span class="k">yield</span> <span class="s">'* process: '</span> <span 
class="o">+</span> <span class="n">element</span>
+
+  <span class="k">def</span> <span class="nf">finish_bundle</span><span 
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">yield</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">utils</span><span class="o">.</span><span 
class="n">windowed_value</span><span class="o">.</span><span 
class="n">WindowedValue</span><span class="p">(</span>
+        <span class="n">value</span><span class="o">=</span><span class="s">'* 
finish_bundle: 🌱🌳🌍'</span><span class="p">,</span>
+        <span class="n">timestamp</span><span class="o">=</span><span 
class="mi">0</span><span class="p">,</span>
+        <span class="n">windows</span><span class="o">=</span><span 
class="p">[</span><span class="bp">self</span><span class="o">.</span><span 
class="n">window</span><span class="p">],</span>
+    <span class="p">)</span>
+
+  <span class="k">def</span> <span class="nf">teardown</span><span 
class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">print</span><span class="p">(</span><span 
class="s">'teardown'</span><span class="p">)</span>
+
+<span class="k">with</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> 
<span class="k">as</span> <span class="n">pipeline</span><span 
class="p">:</span>
+  <span class="n">results</span> <span class="o">=</span> <span 
class="p">(</span>
+      <span class="n">pipeline</span>
+      <span class="o">|</span> <span class="s">'Create inputs'</span> <span 
class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Create</span><span class="p">([</span><span 
class="s">'🍓'</span><span class="p">,</span> <span class="s">'🥕'</span><span 
class="p">,</span> <span class="s">'🍆'</span><span class="p">,</span> <span 
class="s">'🍅'</span><span class="p">,</span> <span class="s">'🥔'</span><span 
class="p">])</span>
+      <span class="o">|</span> <span class="s">'DoFn methods'</span> <span 
class="o">&gt;&gt;</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span 
class="n">DoFnMethods</span><span class="p">())</span>
+      <span class="o">|</span> <span class="n">beam</span><span 
class="o">.</span><span class="n">Map</span><span class="p">(</span><span 
class="k">print</span><span class="p">)</span>
+  <span class="p">)</span>
+</code></pre>
+</div>
+
+<p><code class="highlighter-rouge">stdout</code> output:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>__init__
+setup
+start_bundle
+* process: 🍓
+* process: 🥕
+* process: 🍆
+* process: 🍅
+* process: 🥔
+* finish_bundle: 🌱🌳🌍
+teardown
+</code></pre>
+</div>
+
+<table>
+  <td>
+    <a class="button" target="_blank" 
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py";>
+      <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"; 
width="20px" height="20px" alt="View on GitHub" />
+      View on GitHub
+    </a>
+  </td>
+</table>
+<p><br /></p>
+
+<blockquote>
+  <p><em>Known issues:</em></p>
+
+  <ul>
+    <li><a 
href="https://issues.apache.org/jira/browse/BEAM-7885";>[BEAM-7885]</a>
+<code class="highlighter-rouge">DoFn.setup()</code> doesn’t run for streaming 
jobs running in the <code class="highlighter-rouge">DirectRunner</code>.</li>
+    <li><a 
href="https://issues.apache.org/jira/browse/BEAM-7340";>[BEAM-7340]</a>
+<code class="highlighter-rouge">DoFn.teardown()</code> metrics are lost.</li>
+  </ul>
+</blockquote>
 
 <h2 id="related-transforms">Related transforms</h2>
+
 <ul>
-  <li><a 
href="/documentation/transforms/python/elementwise/flatmap">FlatMap</a> behaves 
the same as <code class="highlighter-rouge">Map</code>, but for each input it 
may produce zero or more outputs.</li>
-  <li><a href="/documentation/transforms/python/elementwise/filter">Filter</a> 
is useful if the function is just 
+  <li><a href="/documentation/transforms/python/elementwise/map">Map</a> 
behaves the same, but produces exactly one output for each input.</li>
+  <li><a 
href="/documentation/transforms/python/elementwise/flatmap">FlatMap</a> behaves 
the same as <code class="highlighter-rouge">Map</code>,
+but for each input it may produce zero or more outputs.</li>
+  <li><a href="/documentation/transforms/python/elementwise/filter">Filter</a> 
is useful if the function is just
 deciding whether to output an element or not.</li>
 </ul>
 
+<table>
+  <td>
+    <a class="button" target="_blank" 
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo";>
+      <img src="https://beam.apache.org/images/logos/sdks/python.png"; 
width="20px" height="20px" alt="Pydoc" />
+      Pydoc
+    </a>
+  </td>
+</table>
+<p><br /></p>
+
       </div>
     </div>
     <!--

[beam] branch asf-site updated: Publishing website 2019/09/11 17:15:01 at commit 40ccf08

Reply via email to