This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 6dca566 Publishing website 2019/07/31 22:10:43 at commit 7472f81 6dca566 is described below commit 6dca5662bba8610b372df461a5aa2145b8ec00d7 Author: jenkins <bui...@apache.org> AuthorDate: Wed Jul 31 22:10:44 2019 +0000 Publishing website 2019/07/31 22:10:43 at commit 7472f81 --- .../python/elementwise/filter/index.html | 284 ++++++++++++++++++++- 1 file changed, 277 insertions(+), 7 deletions(-) diff --git a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html index 9b8871c..8c58ae6 100644 --- a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html +++ b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html @@ -437,7 +437,16 @@ <ul class="nav"> - <li><a href="#examples">Examples</a></li> + <li><a href="#examples">Examples</a> + <ul> + <li><a href="#example-1-filtering-with-a-function">Example 1: Filtering with a function</a></li> + <li><a href="#example-2-filtering-with-a-lambda-function">Example 2: Filtering with a lambda function</a></li> + <li><a href="#example-3-filtering-with-multiple-arguments">Example 3: Filtering with multiple arguments</a></li> + <li><a href="#example-4-filtering-with-side-inputs-as-singletons">Example 4: Filtering with side inputs as singletons</a></li> + <li><a href="#example-5-filtering-with-side-inputs-as-iterators">Example 5: Filtering with side inputs as iterators</a></li> + <li><a href="#example-6-filtering-with-side-inputs-as-dictionaries">Example 6: Filtering with side inputs as dictionaries</a></li> + </ul> + </li> <li><a href="#related-transforms">Related transforms</a></li> </ul> @@ -460,21 +469,272 @@ limitations under the License. --> <h1 id="filter">Filter</h1> -<table align="left"> - <a target="_blank" class="button" href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> + +<script type="text/javascript"> +localStorage.setItem('language', 'language-py') +</script> + +<table> + <td> + <a class="button" target="_blank" href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px" alt="Pydoc" /> - Pydoc + Pydoc </a> + </td> </table> -<p><br /> -Given a predicate, filter out all elements that donβt satisfy that predicate. +<p><br /></p> + +<p>Given a predicate, filter out all elements that donβt satisfy that predicate. May also be used to filter based on an inequality with a given value based on the comparison ordering of the element.</p> <h2 id="examples">Examples</h2> -<p>See <a href="https://issues.apache.org/jira/browse/BEAM-7389">BEAM-7389</a> for updates.</p> + +<p>In the following examples, we create a pipeline with a <code class="highlighter-rouge">PCollection</code> of produce their icon, name, and duration. +Then, we apply <code class="highlighter-rouge">Filter</code> in multiple ways to filter out produce by their duration value.</p> + +<h3 id="example-1-filtering-with-a-function">Example 1: Filtering with a function</h3> + +<p>We define a function <code class="highlighter-rouge">is_perennial</code> which returns <code class="highlighter-rouge">True</code> if the elementβs duration equals <code class="highlighter-rouge">'perennial'</code>, and <code class="highlighter-rouge">False</code> otherwise. +<code class="highlighter-rouge">Filter</code> accepts this function, keeps elements that return <code class="highlighter-rouge">True</code>, and filters out the remaining elements.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">def</span> <span class="nf">is_perennial</span><span class="p">(</span><span class="n">plant</span><span class="p">):</span> + <span class="k">return</span> <span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]</span> <span class="o">==</span> <span class="s">'perennial'</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">perennials</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter perennials'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="n">is_perennial</span><span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> + +<h3 id="example-2-filtering-with-a-lambda-function">Example 2: Filtering with a lambda function</h3> + +<p>We can also use lambda functions to simplify <strong>Example 1</strong>.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">perennials</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter perennials'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> + <span class="k">lambda</span> <span class="n">plant</span><span class="p">:</span> <span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]</span> <span class="o">==</span> <span class="s">'perennial'</span><span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> + +<h3 id="example-3-filtering-with-multiple-arguments">Example 3: Filtering with multiple arguments</h3> + +<p>You can pass functions with multiple arguments to <code class="highlighter-rouge">Filter</code>. +They are passed as additional positional arguments or keyword arguments to the function.</p> + +<p>In this example, <code class="highlighter-rouge">has_duration</code> takes <code class="highlighter-rouge">plant</code> and <code class="highlighter-rouge">duration</code> as arguments.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">def</span> <span class="nf">has_duration</span><span class="p">(</span><span class="n">plant</span><span class="p">,</span> <span class="n">duration</span><span class="p">):</span> + <span class="k">return</span> <span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]</span> <span class="o">==</span> <span class="n">duration</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">perennials</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter perennials'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="n">has_duration</span><span class="p">,</span> <span class="s">'perennial'</span><span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> + +<h3 id="example-4-filtering-with-side-inputs-as-singletons">Example 4: Filtering with side inputs as singletons</h3> + +<p>If the <code class="highlighter-rouge">PCollection</code> has a single value, such as the average from another computation, +passing the <code class="highlighter-rouge">PCollection</code> as a <em>singleton</em> accesses that value.</p> + +<p>In this example, we pass a <code class="highlighter-rouge">PCollection</code> the value <code class="highlighter-rouge">'perennial'</code> as a singleton. +We then use that value to filter out perennials.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">perennial</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s">'Perennial'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="s">'perennial'</span><span class="p">])</span> + + <span class="n">perennials</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter perennials'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> + <span class="k">lambda</span> <span class="n">plant</span><span class="p">,</span> <span class="n">duration</span><span class="p">:</span> <span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]</span> <span class="o">==</span> <span class="n">duration</span><span class="p">,</span> + <span class="n">duration</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">perennial</span><span class="p">),</span> + <span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> + +<h3 id="example-5-filtering-with-side-inputs-as-iterators">Example 5: Filtering with side inputs as iterators</h3> + +<p>If the <code class="highlighter-rouge">PCollection</code> has multiple values, pass the <code class="highlighter-rouge">PCollection</code> as an <em>iterator</em>. +This accesses elements lazily as they are needed, +so it is possible to iterate over large <code class="highlighter-rouge">PCollection</code>s that wonβt fit into memory.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">valid_durations</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s">'Valid durations'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="s">'annual'</span><span class="p">,</span> + <span class="s">'biennial'</span><span class="p">,</span> + <span class="s">'perennial'</span><span class="p">,</span> + <span class="p">])</span> + + <span class="n">valid_plants</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'PERENNIAL'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter valid plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> + <span class="k">lambda</span> <span class="n">plant</span><span class="p">,</span> <span class="n">valid_durations</span><span class="p">:</span> <span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]</span> <span class="ow">in</span> <span class="n">valid_durations</span><span class="p">,</span> + <span class="n">valid_durations</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsIter</span><span class="p">(</span><span class="n">valid_durations</span><span class="p">),</span> + <span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> + +<blockquote> + <p><strong>Note</strong>: You can pass the <code class="highlighter-rouge">PCollection</code> as a <em>list</em> with <code class="highlighter-rouge">beam.pvalue.AsList(pcollection)</code>, +but this requires that all the elements fit into memory.</p> +</blockquote> + +<h3 id="example-6-filtering-with-side-inputs-as-dictionaries">Example 6: Filtering with side inputs as dictionaries</h3> + +<p>If a <code class="highlighter-rouge">PCollection</code> is small enough to fit into memory, then that <code class="highlighter-rouge">PCollection</code> can be passed as a <em>dictionary</em>. +Each element must be a <code class="highlighter-rouge">(key, value)</code> pair. +Note that all the elements of the <code class="highlighter-rouge">PCollection</code> must fit into memory for this. +If the <code class="highlighter-rouge">PCollection</code> wonβt fit into memory, use <code class="highlighter-rouge">beam.pvalue.AsIter(pcollection)</code> instead.</p> + +<div class="language-py highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="kn">as</span> <span class="nn">beam</span> + +<span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> + <span class="n">keep_duration</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s">'Duration filters'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">(</span><span class="s">'annual'</span><span class="p">,</span> <span class="bp">False</span><span class="p">),</span> + <span class="p">(</span><span class="s">'biennial'</span><span class="p">,</span> <span class="bp">False</span><span class="p">),</span> + <span class="p">(</span><span class="s">'perennial'</span><span class="p">,</span> <span class="bp">True</span><span class="p">),</span> + <span class="p">])</span> + + <span class="n">perennials</span> <span class="o">=</span> <span class="p">(</span> + <span class="n">pipeline</span> + <span class="o">|</span> <span class="s">'Gardening plants'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Strawberry'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Carrot'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'biennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Eggplant'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π '</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Tomato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'annual'</span><span class="p">},</span> + <span class="p">{</span><span class="s">'icon'</span><span class="p">:</span> <span class="s">'π₯'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'Potato'</span><span class="p">,</span> <span class="s">'duration'</span><span class="p">:</span> <span class="s">'perennial'</span><span class="p">},</span> + <span class="p">])</span> + <span class="o">|</span> <span class="s">'Filter plants by duration'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> + <span class="k">lambda</span> <span class="n">plant</span><span class="p">,</span> <span class="n">keep_duration</span><span class="p">:</span> <span class="n">keep_duration</span><span class="p">[</span><span class="n">plant</span><span class="p">[</span><span class="s">'duration'</span><span class="p">]],</span> + <span class="n">keep_duration</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsDict</span><span class="p">(</span><span class="n">keep_duration</span><span class="p">),</span> + <span class="p">)</span> + <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">print</span><span class="p">)</span> + <span class="p">)</span> +</code></pre> +</div> + +<table> + <td> + <a class="button" target="_blank" href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"> + <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" width="20px" height="20px" alt="View on GitHub" /> + View on GitHub + </a> + </td> +</table> +<p><br /></p> <h2 id="related-transforms">Related transforms</h2> + <ul> <li><a href="/documentation/transforms/python/elementwise/flatmap">FlatMap</a> behaves the same as <code class="highlighter-rouge">Map</code>, but for each input it might produce zero or more outputs.</li> @@ -482,6 +742,16 @@ each input it might produce zero or more outputs.</li> operation, and includes other abilities such as multiple output collections and side-inputs.</li> </ul> +<table> + <td> + <a class="button" target="_blank" href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> + <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px" alt="Pydoc" /> + Pydoc + </a> + </td> +</table> +<p><br /></p> + </div> </div> <!--