This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 5d314b9 Publishing website 2021/10/05 00:03:05 at commit e891d24 5d314b9 is described below commit 5d314b9842088e2c7e4ff46b3ae8810dfd9d2f55 Author: jenkins <bui...@apache.org> AuthorDate: Tue Oct 5 00:03:06 2021 +0000 Publishing website 2021/10/05 00:03:05 at commit e891d24 --- website/generated-content/documentation/index.xml | 102 ++++++++++++++++++++- .../documentation/programming-guide/index.html | 82 +++++++++++++++-- website/generated-content/sitemap.xml | 2 +- 3 files changed, 177 insertions(+), 9 deletions(-) diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml index c4ca3be..e3aa7ac 100644 --- a/website/generated-content/documentation/index.xml +++ b/website/generated-content/documentation/index.xml @@ -9364,6 +9364,9 @@ after jobs have completed.</p> <h3 id="types-of-metrics">10.2. Types of metrics</h3> <p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and <code>Gauge</code>.</p> +<p class="language-go">In the Beam SDK for Go, a <code>context.Context</code> provided by the framework must be passed to the metric +or the metric value will not be recorded. The framework will automatically provide a valid +<code>context.Context</code> to <code>ProcessElement</code> and similar methods when it&rsquo;s the first parameter.</p> <p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p> <div class='language-java snippet'> <div class="notebook-skip code-snippet"> @@ -9379,6 +9382,19 @@ after jobs have completed.</p> <span class="o">}</span></code></pre></div> </div> </div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">var</span> <span class="nx">counter</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewCounter</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;counter1&#34;</span><span class="p">) [...] +<span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="o">...</span><span class="p">)&l [...] +<span class="c1">// count the elements +</span><span class="c1"></span> <span class="nx">counter</span><span class="p">.</span><span class="nf">Inc</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> +<span class="o">...</span> +<span class="p">}</span></code></pre></div> +</div> +</div> <p><strong>Distribution</strong>: A metric that reports information about the distribution of reported values.</p> <div class='language-java snippet'> <div class="notebook-skip code-snippet"> @@ -9395,6 +9411,19 @@ after jobs have completed.</p> <span class="o">}</span></code></pre></div> </div> </div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">var</span> <span class="nx">distribution</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewDistribution</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;distribution1&#34;</span><s [...] +<span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">in [...] +<span class="c1">// create a distribution (histogram) of the values +</span><span class="c1"></span> <span class="nx">distribution</span><span class="p">.</span><span class="nf">Inc</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">v</span><span class="p">)</span> +<span class="o">...</span> +<span class="p">}</span></code></pre></div> +</div> +</div> <p><strong>Gauge</strong>: A metric that reports the latest value out of reported values. Since metrics are collected from many workers the value may not be the absolute last, but one of the latest values.</p> <div class='language-java snippet'> @@ -9412,10 +9441,27 @@ collected from many workers the value may not be the absolute last, but one of t <span class="o">}</span></code></pre></div> </div> </div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">var</span> <span class="nx">gauge</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewGauge</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;gauge1&#34;</span><span class="p">)</span> +<span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">in [...] +<span class="c1">// create a gauge (latest value received) of the values +</span><span class="c1"></span> <span class="nx">gauge</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">v</span><span class="p">)</span> +<span class="o">...</span> +<span class="p">}</span></code></pre></div> +</div> +</div> <h3 id="querying-metrics">10.3. Querying metrics</h3> -<p><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows +<p class="language-java language-python"><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows accessing metrics. The main method available in <code>MetricResults</code> allows querying for all metrics matching a given filter.</p> +<p class="language-go"><code>beam.PipelineResult</code> has a method <code>Metrics()</code> which returns a <code>metrics.Results</code> object that allows +accessing metrics. The main method available in <code>metrics.Results</code> allows querying for all metrics +matching a given filter. It takes in a predicate with a <code>SingleResult</code> parameter type, which can +be used for custom filters.</p> <div class='language-java snippet'> <div class="notebook-skip code-snippet"> <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> @@ -9440,6 +9486,18 @@ matching a given filter.</p> <span class="o">}</span></code></pre></div> </div> </div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">queryMetrics</span><span class="p">(</span><span class="nx">pr</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PipelineResult</span><span class="p">,</span> <span class="nx">ns</span><span class="p">,</span> <span class="nx">n</span> <span class="kt">strin [...] +<span class="k">return</span> <span class="nx">pr</span><span class="p">.</span><span class="nf">Metrics</span><span class="p">().</span><span class="nf">Query</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">metrics</span><span class="p">.</span><span class="nx">SingleResult</span><span class="p">)</span> <span class="kt">bool [...] +<span class="k">return</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">()</span> <span class="o">==</span> <span class="nx">ns</span> <span class="o">&amp;&amp;</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Name</span><span class="p">()</span> <span class="o">==</span> <span class="nx">n</span> +<span class="p">})</span> +<span class="p">}</span></code></pre></div> +</div> +</div> <h3 id="using-metrics">10.4. Using metrics in pipeline</h3> <p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p> <div class='language-java snippet'> @@ -9476,6 +9534,48 @@ matching a given filter.</p> <span class="o">}</span></code></pre></div> </div> </div> +<div class='language-go snippet'> +<div class="notebook-skip code-snippet"> +<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> +<img src="/images/copy-icon.svg"/> +</a> +<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">addMetricDoFnToPipeline</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">input</span> <span class="nx">beam</span><span class="p">.</span><span class="nx" [...] +<span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">MyMetricsDoFn</span><span class="p">{},</span> <span class="nx">input</span><span class="p">)</span> +<span class="p">}</span> +<span class="kd">func</span> <span class="nf">executePipelineAndGetMetrics</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">p</span> <span class="o">*</span><span class="nx">beam</span><span class="p">.</span><span class="nx">Pipeline</span><span class="p">)</span> <s [...] +<span class="nx">pr</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">runner</span><span class="p">,</span> <span class="nx">p</span><span class="p">)</span> +<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> +<span class="k">return</span> <span class="nx">metrics</span><span class="p">.</span><span class="nx">QueryResults</span><span class="p">{},</span> <span class="nx">err</span> +<span class="p">}</span> +<span class="c1">// Request the metric called &#34;counter1&#34; in namespace called &#34;namespace&#34; +</span><span class="c1"></span> <span class="nx">ms</span> <span class="o">:=</span> <span class="nx">pr</span><span class="p">.</span><span class="nf">Metrics</span><span class="p">().</span><span class="nf">Query</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">metrics</span><span class="p">.</span><span class="nx">SingleR [...] +<span class="k">return</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">()</span> <span class="o">==</span> <span class="s">&#34;namespace&#34;</span> <span class="o">&amp;&amp;</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Name</span><span class="p">()</span> <span class="o">==</span> <span class="s">&#34; [...] +<span class="p">})</span> +<span class="c1">// Print the metric value - there should be only one line because there is +</span><span class="c1"></span> <span class="c1">// only one metric called &#34;counter1&#34; in the namespace called &#34;namespace&#34; +</span><span class="c1"></span> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">c</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">ms</span><span class="p">.</span><span class="nf">Counters</span><span class="p">()</span> <span class="p">{</span> +<span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">(),</span> <span class="s">&#34;-&#34;</span><span class="p">,</span> <span class="nx">c</span><span class="p">.</span><span class="nf">Name</span><span class="p">(),</span> <span class="s">&am [...] +<span class="p">}</span> +<span class="k">return</span> <span class="nx">ms</span><span class="p">,</span> <span class="kc">nil</span> +<span class="p">}</span> +<span class="kd">type</span> <span class="nx">MyMetricsDoFn</span> <span class="kd">struct</span> <span class="p">{</span> +<span class="nx">counter</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Counter</span> +<span class="p">}</span> +<span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> +<span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterType</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)))</span> +<span class="p">}</span> +<span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)</span> <span class="nf">Setup</span><span class="p">()</span> <span class="p">{</span> +<span class="c1">// While metrics can be defined in package scope or dynamically +</span><span class="c1"></span> <span class="c1">// it&#39;s most efficient to include them in the DoFn. +</span><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">counter</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewCounter</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;counter1&#34;</span><span class="p">)</span> +<span class="p">}</span> +<span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class= [...] +<span class="c1">// count the elements +</span><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">counter</span><span class="p">.</span><span class="nf">Inc</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> +<span class="nf">emit</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span> +<span class="p">}</span></code></pre></div> +</div> +</div> <h3 id="export-metrics">10.5. Export metrics</h3> <p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period. The configuration is held in the <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html">MetricsOptions</a> class. diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index c092a4c..b86067b 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -2930,14 +2930,22 @@ doesn’t support some part of querying metrics, the runner will not return the This allows reporting the same metric name in multiple places and identifying the value each transform reported, as well as aggregating the metric across the entire pipeline.</p><blockquote><p><strong>Note:</strong> It is runner-dependent whether metrics are accessible during pipeline execution or only after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2. Types of metrics</h3><p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and -<code>Gauge</code>.</p><p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Counter</span> <span class=n>counter</span> <span cl [...] +<code>Gauge</code>.</p><p class=language-go>In the Beam SDK for Go, a <code>context.Context</code> provided by the framework must be passed to the metric +or the metric value will not be recorded. The framework will automatically provide a valid +<code>context.Context</code> to <code>ProcessElement</code> and similar methods when it’s the first parameter.</p><p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code clas [...] <span class=nd>@ProcessElement</span> <span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=n>ProcessContext</span> <span class=n>context</span><span class=o>)</span> <span class=o>{</span> <span class=c1>// count the elements </span><span class=c1></span> <span class=n>counter</span><span class=o>.</span><span class=na>inc</span><span class=o>();</span> <span class=o>...</span> -<span class=o>}</span></code></pre></div></div></div><p><strong>Distribution</strong>: A metric that reports information about the distribution of reported values.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Distribution</span> < [...] +<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>var</span> <span class=nx>counter</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewCounter</s [...] + +<span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>MyDoFn</span><span class=p>)</span> <span class=nf>ProcessElement</span><span class=p>(</span><span class=nx>ctx</span> <span class=nx>context</span><span class=p>.</span><span class=nx>Context</span><span class=p>,</span> <span class=o>...</span><span class=p>)</span> <span class=p>{</span> + <span class=c1>// count the elements +</span><span class=c1></span> <span class=nx>counter</span><span class=p>.</span><span class=nf>Inc</span><span class=p>(</span><span class=nx>ctx</span><span class=p>,</span> <span class=mi>1</span><span class=p>)</span> + <span class=o>...</span> +<span class=p>}</span></code></pre></div></div></div><p><strong>Distribution</strong>: A metric that reports information about the distribution of reported values.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Distribution</span> < [...] <span class=nd>@ProcessElement</span> <span class=kd>public</span> <span class=kt>void</span> <span class=nf>processElement</span><span class=o>(</span><span class=n>ProcessContext</span> <span class=n>context</span><span class=o>)</span> <span class=o>{</span> @@ -2945,7 +2953,13 @@ after jobs have completed.</p></blockquote><h3 id=types-of-metrics>10.2. Types o <span class=c1>// create a distribution (histogram) of the values </span><span class=c1></span> <span class=n>distribution</span><span class=o>.</span><span class=na>update</span><span class=o>(</span><span class=n>element</span><span class=o>);</span> <span class=o>...</span> -<span class=o>}</span></code></pre></div></div></div><p><strong>Gauge</strong>: A metric that reports the latest value out of reported values. Since metrics are +<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>var</span> <span class=nx>distribution</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewDistr [...] + +<span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>MyDoFn</span><span class=p>)</span> <span class=nf>ProcessElement</span><span class=p>(</span><span class=nx>ctx</span> <span class=nx>context</span><span class=p>.</span><span class=nx>Context</span><span class=p>,</span> <span class=nx>v</span> <span class=kt>int64</span><span class=p>,</span> <span class=o>...</span><span class=p>)</span> <span class=p>{</span> + <span class=c1>// create a distribution (histogram) of the values +</span><span class=c1></span> <span class=nx>distribution</span><span class=p>.</span><span class=nf>Inc</span><span class=p>(</span><span class=nx>ctx</span><span class=p>,</span> <span class=nx>v</span><span class=p>)</span> + <span class=o>...</span> +<span class=p>}</span></code></pre></div></div></div><p><strong>Gauge</strong>: A metric that reports the latest value out of reported values. Since metrics are collected from many workers the value may not be the absolute last, but one of the latest values.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Gauge</span> <span class=n>gauge</span> <span class=o>=</span> <span class=n>Metrics</s [...] <span class=nd>@ProcessElement</span> @@ -2954,9 +2968,18 @@ collected from many workers the value may not be the absolute last, but one of t <span class=c1>// create a gauge (latest value received) of the values </span><span class=c1></span> <span class=n>gauge</span><span class=o>.</span><span class=na>set</span><span class=o>(</span><span class=n>element</span><span class=o>);</span> <span class=o>...</span> -<span class=o>}</span></code></pre></div></div></div><h3 id=querying-metrics>10.3. Querying metrics</h3><p><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows +<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>var</span> <span class=nx>gauge</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewGauge</span> [...] + +<span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>MyDoFn</span><span class=p>)</span> <span class=nf>ProcessElement</span><span class=p>(</span><span class=nx>ctx</span> <span class=nx>context</span><span class=p>.</span><span class=nx>Context</span><span class=p>,</span> <span class=nx>v</span> <span class=kt>int64</span><span class=p>,</span> <span class=o>...</span><span class=p>)</span> <span class=p>{</span> + <span class=c1>// create a gauge (latest value received) of the values +</span><span class=c1></span> <span class=nx>gauge</span><span class=p>.</span><span class=nf>Set</span><span class=p>(</span><span class=nx>ctx</span><span class=p>,</span> <span class=nx>v</span><span class=p>)</span> + <span class=o>...</span> +<span class=p>}</span></code></pre></div></div></div><h3 id=querying-metrics>10.3. Querying metrics</h3><p class="language-java language-python"><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows accessing metrics. The main method available in <code>MetricResults</code> allows querying for all metrics -matching a given filter.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span class=nc>PipelineResult</span> <span class=o>{</span> +matching a given filter.</p><p class=language-go><code>beam.PipelineResult</code> has a method <code>Metrics()</code> which returns a <code>metrics.Results</code> object that allows +accessing metrics. The main method available in <code>metrics.Results</code> allows querying for all metrics +matching a given filter. It takes in a predicate with a <code>SingleResult</code> parameter type, which can +be used for custom filters.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=kd>public</span> <span class=kd>interface</span> <span class=nc>PipelineResult</span> <span class=o>{</span> <span class=n>MetricResults</span> <span class=nf>metrics</span><span class=o>();</span> <span class=o>}</span> @@ -2975,7 +2998,11 @@ matching a given filter.</p><div class="language-java snippet"><div class="noteb <span class=n>String</span> <span class=nf>getStep</span><span class=o>();</span> <span class=n>T</span> <span class=nf>getCommitted</span><span class=o>();</span> <span class=n>T</span> <span class=nf>getAttempted</span><span class=o>();</span> -<span class=o>}</span></code></pre></div></div></div><h3 id=using-metrics>10.4. Using metrics in pipeline</h3><p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data [...] +<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>queryMetrics</span><span class=p>(</span><span class=nx>pr</span> <span class=nx>beam</span><span class=p>.</span [...] + <span class=k>return</span> <span class=nx>pr</span><span class=p>.</span><span class=nf>Metrics</span><span class=p>().</span><span class=nf>Query</span><span class=p>(</span><span class=kd>func</span><span class=p>(</span><span class=nx>r</span> <span class=nx>metrics</span><span class=p>.</span><span class=nx>SingleResult</span><span class=p>)</span> <span class=kt>bool</span> <span class=p>{</span> + <span class=k>return</span> <span class=nx>r</span><span class=p>.</span><span class=nf>Namespace</span><span class=p>()</span> <span class=o>==</span> <span class=nx>ns</span> <span class=o>&&</span> <span class=nx>r</span><span class=p>.</span><span class=nf>Name</span><span class=p>()</span> <span class=o>==</span> <span class=nx>n</span> + <span class=p>})</span> +<span class=p>}</span></code></pre></div></div></div><h3 id=using-metrics>10.4. Using metrics in pipeline</h3><p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data [...] </span><span class=c1></span><span class=n>pipeline</span> <span class=o>.</span><span class=na>apply</span><span class=o>(...)</span> <span class=o>.</span><span class=na>apply</span><span class=o>(</span><span class=n>ParDo</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=k>new</span> <span class=n>MyMetricsDoFn</span><span class=o>()));</span> @@ -3006,7 +3033,48 @@ matching a given filter.</p><div class="language-java snippet"><div class="noteb </span><span class=c1></span> <span class=n>counter</span><span class=o>.</span><span class=na>inc</span><span class=o>();</span> <span class=n>context</span><span class=o>.</span><span class=na>output</span><span class=o>(</span><span class=n>context</span><span class=o>.</span><span class=na>element</span><span class=o>());</span> <span class=o>}</span> -<span class=o>}</span></code></pre></div></div></div><h3 id=export-metrics>10.5. Export metrics</h3><p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period. +<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>func</span> <span class=nf>addMetricDoFnToPipeline</span><span class=p>(</span><span class=nx>s</span> <span class=nx>beam</span><span class [...] + <span class=k>return</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>ParDo</span><span class=p>(</span><span class=nx>s</span><span class=p>,</span> <span class=o>&</span><span class=nx>MyMetricsDoFn</span><span class=p>{},</span> <span class=nx>input</span><span class=p>)</span> +<span class=p>}</span> + +<span class=kd>func</span> <span class=nf>executePipelineAndGetMetrics</span><span class=p>(</span><span class=nx>ctx</span> <span class=nx>context</span><span class=p>.</span><span class=nx>Context</span><span class=p>,</span> <span class=nx>p</span> <span class=o>*</span><span class=nx>beam</span><span class=p>.</span><span class=nx>Pipeline</span><span class=p>)</span> <span class=p>(</span><span class=nx>metrics</span><span class=p>.</span><span class=nx>QueryResults</span><span clas [...] + <span class=nx>pr</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>Run</span><span class=p>(</span><span class=nx>ctx</span><span class=p>,</span> <span class=nx>runner</span><span class=p>,</span> <span class=nx>p</span><span class=p>)</span> + <span class=k>if</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span> + <span class=k>return</span> <span class=nx>metrics</span><span class=p>.</span><span class=nx>QueryResults</span><span class=p>{},</span> <span class=nx>err</span> + <span class=p>}</span> + + <span class=c1>// Request the metric called "counter1" in namespace called "namespace" +</span><span class=c1></span> <span class=nx>ms</span> <span class=o>:=</span> <span class=nx>pr</span><span class=p>.</span><span class=nf>Metrics</span><span class=p>().</span><span class=nf>Query</span><span class=p>(</span><span class=kd>func</span><span class=p>(</span><span class=nx>r</span> <span class=nx>metrics</span><span class=p>.</span><span class=nx>SingleResult</span><span class=p>)</span> <span class=kt>bool</span> <span class=p>{</span> + <span class=k>return</span> <span class=nx>r</span><span class=p>.</span><span class=nf>Namespace</span><span class=p>()</span> <span class=o>==</span> <span class=s>"namespace"</span> <span class=o>&&</span> <span class=nx>r</span><span class=p>.</span><span class=nf>Name</span><span class=p>()</span> <span class=o>==</span> <span class=s>"counter1"</span> + <span class=p>})</span> + + <span class=c1>// Print the metric value - there should be only one line because there is +</span><span class=c1></span> <span class=c1>// only one metric called "counter1" in the namespace called "namespace" +</span><span class=c1></span> <span class=k>for</span> <span class=nx>_</span><span class=p>,</span> <span class=nx>c</span> <span class=o>:=</span> <span class=k>range</span> <span class=nx>ms</span><span class=p>.</span><span class=nf>Counters</span><span class=p>()</span> <span class=p>{</span> + <span class=nx>fmt</span><span class=p>.</span><span class=nf>Println</span><span class=p>(</span><span class=nx>c</span><span class=p>.</span><span class=nf>Namespace</span><span class=p>(),</span> <span class=s>"-"</span><span class=p>,</span> <span class=nx>c</span><span class=p>.</span><span class=nf>Name</span><span class=p>(),</span> <span class=s>":"</span><span class=p>,</span> <span class=nx>c</span><span class=p>.</span><span class=nx>Committed</span><span cla [...] + <span class=p>}</span> + <span class=k>return</span> <span class=nx>ms</span><span class=p>,</span> <span class=kc>nil</span> +<span class=p>}</span> + +<span class=kd>type</span> <span class=nx>MyMetricsDoFn</span> <span class=kd>struct</span> <span class=p>{</span> + <span class=nx>counter</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>Counter</span> +<span class=p>}</span> + +<span class=kd>func</span> <span class=nf>init</span><span class=p>()</span> <span class=p>{</span> + <span class=nx>beam</span><span class=p>.</span><span class=nf>RegisterType</span><span class=p>(</span><span class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span class=p>((</span><span class=o>*</span><span class=nx>MyMetricsDoFn</span><span class=p>)(</span><span class=kc>nil</span><span class=p>)))</span> +<span class=p>}</span> + +<span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>MyMetricsDoFn</span><span class=p>)</span> <span class=nf>Setup</span><span class=p>()</span> <span class=p>{</span> + <span class=c1>// While metrics can be defined in package scope or dynamically +</span><span class=c1></span> <span class=c1>// it's most efficient to include them in the DoFn. +</span><span class=c1></span> <span class=nx>fn</span><span class=p>.</span><span class=nx>counter</span> <span class=p>=</span> <span class=nx>beam</span><span class=p>.</span><span class=nf>NewCounter</span><span class=p>(</span><span class=s>"namespace"</span><span class=p>,</span> <span class=s>"counter1"</span><span class=p>)</span> +<span class=p>}</span> + +<span class=kd>func</span> <span class=p>(</span><span class=nx>fn</span> <span class=o>*</span><span class=nx>MyMetricsDoFn</span><span class=p>)</span> <span class=nf>ProcessElement</span><span class=p>(</span><span class=nx>ctx</span> <span class=nx>context</span><span class=p>.</span><span class=nx>Context</span><span class=p>,</span> <span class=nx>v</span> <span class=nx>beam</span><span class=p>.</span><span class=nx>V</span><span class=p>,</span> <span class=nx>emit</span> <span [...] + <span class=c1>// count the elements +</span><span class=c1></span> <span class=nx>fn</span><span class=p>.</span><span class=nx>counter</span><span class=p>.</span><span class=nf>Inc</span><span class=p>(</span><span class=nx>ctx</span><span class=p>,</span> <span class=mi>1</span><span class=p>)</span> + <span class=nf>emit</span><span class=p>(</span><span class=nx>v</span><span class=p>)</span> +<span class=p>}</span></code></pre></div></div></div><h3 id=export-metrics>10.5. Export metrics</h3><p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period. The configuration is held in the <a href=https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html>MetricsOptions</a> class. It contains push period configuration and also sink specific options such as type and URL. As for now only the REST HTTP and the Graphite sinks are supported and only Flink and Spark runners support metrics export.</p><p>Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.</p><h2 id=state-and-timers>11. State and Timers</h2><p>Beam’s windowing and triggering facilities provide a powerful abstraction for grouping and aggregating unbounded input diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 4816d58..1897f4b 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...] \ No newline at end of file +<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...] \ No newline at end of file