This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 6d0d4fe Publishing website 2020/04/17 22:33:49 at commit be57a61 6d0d4fe is described below commit 6d0d4fefb08dd655fa1e82e88d9008d0d863956f Author: jenkins <bui...@apache.org> AuthorDate: Fri Apr 17 22:33:50 2020 +0000 Publishing website 2020/04/17 22:33:49 at commit be57a61 --- .../sdks/python-type-safety/index.html | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/website/generated-content/documentation/sdks/python-type-safety/index.html b/website/generated-content/documentation/sdks/python-type-safety/index.html index 11e9f20..f36bfa4 100644 --- a/website/generated-content/documentation/sdks/python-type-safety/index.html +++ b/website/generated-content/documentation/sdks/python-type-safety/index.html @@ -374,7 +374,8 @@ Introducing type hints for the <code class="highlighter-rouge">PTransforms</code </code></pre></div></div> -<p>When you call <code class="highlighter-rouge">p.run()</code>, this code generates an error because <code class="highlighter-rouge">Filter</code> expects a <code class="highlighter-rouge">PCollection</code> of integers, but is given a <code class="highlighter-rouge">PCollection</code> of strings instead.</p> +<p>When you call <code class="highlighter-rouge">p.run()</code>, this code generates an error when trying to execute this transform because <code class="highlighter-rouge">Filter</code> expects a <code class="highlighter-rouge">PCollection</code> of integers, but is given a <code class="highlighter-rouge">PCollection</code> of strings instead. +With type hints, this error could have been caught at pipeline construction time, before the pipeline even started running.</p> <p>The Beam SDK for Python includes some automatic type hinting: for example, some <code class="highlighter-rouge">PTransforms</code>, such as <code class="highlighter-rouge">Create</code> and simple <code class="highlighter-rouge">ParDo</code> transforms, attempt to deduce their output type given their input. However, Beam cannot deduce types in all cases. @@ -415,16 +416,15 @@ Two such are the <code class="highlighter-rouge">expand</code> of a composite tr </code></pre></div></div> <p>The following code declares <code class="highlighter-rouge">int</code> input and output type hints on <code class="highlighter-rouge">filter_evens</code>, using annotations on <code class="highlighter-rouge">FilterEvensDoFn.process</code>. -Since <code class="highlighter-rouge">process</code> returns a generator, the output type is annotated as <code class="highlighter-rouge">Iterable[int]</code> (<code class="highlighter-rouge">Generator[int, None, None]</code> would also work here). -Beam will remove the outer iterable of the return type on the <code class="highlighter-rouge">DoFn.process</code> method and functions passed to <code class="highlighter-rouge">ParDo</code> and <code class="highlighter-rouge">FlatMap</code>. +Since <code class="highlighter-rouge">process</code> returns a generator, the output type for a DoFn producing a <code class="highlighter-rouge">PCollection[int]</code> is annotated as <code class="highlighter-rouge">Iterable[int]</code> (<code class="highlighter-rouge">Generator[int, None, None]</code> would also work here). +Beam will remove the outer iterable of the return type on the <code class="highlighter-rouge">DoFn.process</code> method and functions passed to <code class="highlighter-rouge">FlatMap</code> to deduce the element type of resulting PCollection . It is an error to have a non-iterable return type annotation for these functions. Other supported iterable types include: <code class="highlighter-rouge">Iterator</code>, <code class="highlighter-rouge">Generator</code>, <code class="highlighter-rouge">Tuple</code>, <code class="highlighter-rouge">List</code>.</p> <div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Iterable</span> <span class="k">class</span> <span class="nc">FilterEvensDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> - <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="o">*</span><span class="n">unused_args</span><span class="p">,</span> - <span class="o">**</span><span class="n">unused_kwargs</span><span class="p">)</span> <span class="o">-></span> <span class="n">Iterable</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span> + <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Iterable</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span> <span class="k">if</span> <span class="n">element</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">yield</span> <span class="n">element</span> @@ -434,13 +434,12 @@ Other supported iterable types include: <code class="highlighter-rouge">Iterator <p>The following code declares <code class="highlighter-rouge">int</code> input and output type hints on <code class="highlighter-rouge">double_evens</code>, using annotations on <code class="highlighter-rouge">FilterEvensDoubleDoFn.process</code>. Since <code class="highlighter-rouge">process</code> returns a <code class="highlighter-rouge">list</code> or <code class="highlighter-rouge">None</code>, the output type is annotated as <code class="highlighter-rouge">Optional[List[int]]</code>. -Beam will remove the outer <code class="highlighter-rouge">Optional</code> and (as above) the outer iterable of the return type, only on the <code class="highlighter-rouge">DoFn.process</code> method and functions passed to <code class="highlighter-rouge">ParDo</code> and <code class="highlighter-rouge">FlatMap</code>.</p> +Beam will also remove the outer <code class="highlighter-rouge">Optional</code> and (as above) the outer iterable of the return type, only on the <code class="highlighter-rouge">DoFn.process</code> method and functions passed to <code class="highlighter-rouge">FlatMap</code>.</p> <div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span> <span class="k">class</span> <span class="nc">FilterEvensDoubleDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> - <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="o">*</span><span class="n">unused_args</span><span class="p">,</span> - <span class="o">**</span><span class="n">unused_kwargs</span><span class="p">)</span> <span class="o">-></span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]:</span> + <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Optional</span><span class="p">[</span><span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]:</span> <span class="k">if</span> <span class="n">element</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="p">[</span><span class="n">element</span><span class="p">,</span> <span class="n">element</span><span class="p">]</span> <span class="k">return</span> <span class="bp">None</span> @@ -582,15 +581,15 @@ It also supports Python’s typing module types, which are internally converted <p>When your pipeline reads, writes, or otherwise materializes its data, the elements in your <code class="highlighter-rouge">PCollection</code> need to be encoded and decoded to and from byte strings. Byte strings are used for intermediate storage, for comparing keys in <code class="highlighter-rouge">GroupByKey</code> operations, and for reading from sources and writing to sinks.</p> -<p>The Beam SDK for Python uses Python’s native support for serializing objects, a process called <strong>pickling</strong>, to serialize user functions. However, using the <code class="highlighter-rouge">PickleCoder</code> comes with several drawbacks: it is less efficient in time and space, and the encoding used is not deterministic, which hinders distributed partitioning, grouping, and state lookup.</p> +<p>The Beam SDK for Python uses Python’s native support for serializing objects of unknown type, a process called <strong>pickling</strong>. However, using the <code class="highlighter-rouge">PickleCoder</code> comes with several drawbacks: it is less efficient in time and space, and the encoding used is not deterministic, which hinders distributed partitioning, grouping, and state lookup.</p> <p>To avoid these drawbacks, you can define <code class="highlighter-rouge">Coder</code> classes for encoding and decoding types in a more efficient way. You can specify a <code class="highlighter-rouge">Coder</code> to describe how the elements of a given <code class="highlighter-rouge">PCollection</code> should be encoded and decoded.</p> -<p>In order to be correct and efficient, a <code class="highlighter-rouge">Coder</code> needs type information and for <code class="highlighter-rouge">PCollections</code> to be associated with a specific type. Type hints are what make this type information available. The Beam SDK for Python provides built-in coders for the standard Python types <code class="highlighter-rouge">int</code>, <code class="highlighter-rouge">float</code>, <code class="highlighter-rouge">str</code>, <code class [...] +<p>In order to be correct and efficient, a <code class="highlighter-rouge">Coder</code> needs type information and for <code class="highlighter-rouge">PCollections</code> to be associated with a specific type. Type hints are what make this type information available. The Beam SDK for Python provides built-in coders for the standard Python types such as <code class="highlighter-rouge">int</code>, <code class="highlighter-rouge">float</code>, <code class="highlighter-rouge">str</code>, <co [...] <h3 id="deterministic-coders">Deterministic Coders</h3> -<p>If you don’t define a <code class="highlighter-rouge">Coder</code>, the default is <code class="highlighter-rouge">PickleCoder</code>, which is nondeterministic. In some cases, you must specify a deterministic <code class="highlighter-rouge">Coder</code> or else you will get a runtime error.</p> +<p>If you don’t define a <code class="highlighter-rouge">Coder</code>, the default is a coder that falls back to pickling for unknown types. In some cases, you must specify a deterministic <code class="highlighter-rouge">Coder</code> or else you will get a runtime error.</p> <p>For example, suppose you have a <code class="highlighter-rouge">PCollection</code> of key-value pairs whose keys are <code class="highlighter-rouge">Player</code> objects. If you apply a <code class="highlighter-rouge">GroupByKey</code> transform to such a collection, its key objects might be serialized differently on different machines when a nondeterministic coder, such as the default pickle coder, is used. Since <code class="highlighter-rouge">GroupByKey</code> uses this serialized [...]