This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1d5fe5c4090 Publishing website 2025/09/08 23:40:53 at commit fe188e3
1d5fe5c4090 is described below
commit 1d5fe5c409055effe17c8d7f1817159f05533933
Author: runner <runner@main-runner-frrkx-l6522>
AuthorDate: Mon Sep 8 23:40:53 2025 +0000
Publishing website 2025/09/08 23:40:53 at commit fe188e3
---
.../documentation/sdks/yaml-errors/index.html | 294 ++++++++++-----------
.../sdks/{yaml-errors => yaml-schema}/index.html | 259 ++++++------------
website/generated-content/sitemap.xml | 2 +-
3 files changed, 233 insertions(+), 322 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/yaml-errors/index.html
b/website/generated-content/documentation/sdks/yaml-errors/index.html
index 82042b6f753..1f1d9d4d1d6 100644
--- a/website/generated-content/documentation/sdks/yaml-errors/index.html
+++ b/website/generated-content/documentation/sdks/yaml-errors/index.html
@@ -48,34 +48,34 @@ another transform that will process the errors (e.g. by
writing them out).
For example,
the following code will write all “good” processed records to one
file and
any “bad” records, along with metadata about what error was
encountered,
-to a separate file.</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
-
- - type: WriteToJson
- input: MapToFields
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- input: MapToFields.my_error_output
- config:
- path: /path/to/errors.json
-</code></pre><p>Note that with <code>error_handling</code> declared,
<code>MapToFields.my_error_output</code>
+to a separate file.</p><div class=highlight><pre tabindex=0 class=chroma><code
class=language-yaml data-lang=yaml><span class=line><span class=cl><span
class=nt>pipeline</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/input*.csv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># This could raise a divide-by-zero error.</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>ratio</span><span class=p>:</span><span class=w>
</span><span class=l>col2 / col3</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/output.json</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>WriteErrorsToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields.my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/errors.json</span><span class=w>
+</span></span></span></code></pre></div><p>Note that with
<code>error_handling</code> declared, <code>MapToFields.my_error_output</code>
<strong>must</strong> be consumed; to ignore it will be an error. Any use is
fine, e.g.
logging the bad records to stdout would be sufficient (though not recommended
for a robust pipeline).</p><p>Note also that the exact format of the error
outputs is still being finalized.
@@ -86,125 +86,125 @@ the error that was encountered (e.g. error messages and
tracebacks).
To recover the bad record alone one can process the error output with the
<code>StripErrorMetadata</code> transformation.</p><p>Some transforms allow
for extra arguments in their error_handling config, e.g.
for Python functions one can give a <code>threshold</code> which limits the
relative number
-of records that can be bad before considering the entire pipeline a
failure</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
- # If more than 10% of records throw an error, stop the pipeline.
- threshold: 0.1
-
- - type: WriteToJson
- input: MapToFields
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- input: MapToFields.my_error_output
- config:
- path: /path/to/errors.json
-</code></pre><p>One can do arbitrary further processing on these failed
records if desired,
-e.g.</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- name: ComputeRatio
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
-
- - type: StripErrorMetadata
- name: FailedRecordsWithoutMetadata
- # Takes the error information from ComputeRatio and returns just the
- # failing records themselves for another attempt with a different
- # transform.
- input: ComputeRatio.my_error_output
-
- - type: MapToFields
- name: ComputeRatioForBadRecords
- input: FailedRecordsWithoutMetadata
- config:
- language: python
- fields:
- col1: col1
- ratio: col2 / (col3 + 1)
- error_handling:
- output: still_bad
-
- - type: WriteToJson
- # Takes as input everything from the "success" path of both
transforms.
- input: [ComputeRatio, ComputeRatioForBadRecords]
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- # These failed the first and the second transform.
- input: ComputeRatioForBadRecords.still_bad
- config:
- path: /path/to/errors.json
-</code></pre><p>When using the <code>chain</code> syntax, the required error
consumption can happen
-in an <code>extra_transforms</code> block.</p><pre tabindex=0><code>pipeline:
- type: chain
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- name: SomeStep
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: errors
-
- - type: MapToFields
- name: AnotherStep
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- inverse_ratio: 1 / ratio
- error_handling:
- output: errors
-
- - type: WriteToJson
- config:
- path: /path/to/output.json
-
- extra_transforms:
- - type: WriteToJson
- name: WriteErrors
- input: [SomeStep.errors, AnotherStep.errors]
- config:
- path: /path/to/errors.json
-</code></pre></div></div><footer class=footer><div
class=footer__contained><div class=footer__cols><div class="footer__cols__col
footer__cols__col__logos"><div class=footer__cols__col__logo><img
src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div
class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg
class=footer__logo alt="Apache logo"></div></div><div class=footer-wrapper><div
class=wrapper-grid><div class=footer__cols__col><div class=footer__c [...]
+of records that can be bad before considering the entire pipeline a
failure</p><div class=highlight><pre tabindex=0 class=chroma><code
class=language-yaml data-lang=yaml><span class=line><span class=cl><span
class=nt>pipeline</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/input*.csv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># This could raise a divide-by-zero error.</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>ratio</span><span class=p>:</span><span class=w>
</span><span class=l>col2 / col3</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># If more than 10% of records throw an error, stop the
pipeline.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>threshold</span><span class=p>:</span><span class=w>
</span><span class=m>0.1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/output.json</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>WriteErrorsToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields.my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/errors.json</span><span class=w>
+</span></span></span></code></pre></div><p>One can do arbitrary further
processing on these failed records if desired,
+e.g.</p><div class=highlight><pre tabindex=0 class=chroma><code
class=language-yaml data-lang=yaml><span class=line><span class=cl><span
class=nt>pipeline</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/input*.csv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>ComputeRatio</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># This could raise a divide-by-zero error.</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>ratio</span><span class=p>:</span><span class=w>
</span><span class=l>col2 / col3</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>StripErrorMetadata</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>FailedRecordsWithoutMetadata</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># Takes the error information from ComputeRatio and
returns just the</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># failing records themselves for another attempt with a
different</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># transform.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ComputeRatio.my_error_output</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>ComputeRatioForBadRecords</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>FailedRecordsWithoutMetadata</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>ratio</span><span class=p>:</span><span class=w>
</span><span class=l>col2 / (col3 + 1)</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>still_bad</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># Takes as input everything from the "success"
path of both transforms.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=p>[</span><span class=l>ComputeRatio,
ComputeRatioForBadRecords]</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/output.json</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>WriteErrorsToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># These failed the first and the second
transform.</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ComputeRatioForBadRecords.still_bad</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/errors.json</span><span class=w>
+</span></span></span></code></pre></div><p>When using the <code>chain</code>
syntax, the required error consumption can happen
+in an <code>extra_transforms</code> block.</p><div class=highlight><pre
tabindex=0 class=chroma><code class=language-yaml data-lang=yaml><span
class=line><span class=cl><span class=nt>pipeline</span><span
class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>chain</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>ReadFromCsv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/input*.csv</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>SomeStep</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># This could raise a divide-by-zero error.</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>ratio</span><span class=p>:</span><span class=w>
</span><span class=l>col2 / col3</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>errors</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>AnotherStep</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>col1</span><span class=p>:</span><span class=w>
</span><span class=l>col1</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=c># This could raise a divide-by-zero error.</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>inverse_ratio</span><span class=p>:</span><span class=w>
</span><span class=m>1</span><span class=w> </span><span class=l>/
ratio</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>errors</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/output.json</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>extra_transforms</span><span class=p>:</span><span
class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>WriteToJson</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>WriteErrors</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=p>[</span><span class=l>SomeStep.errors,
AnotherStep.errors]</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>path</span><span class=p>:</span><span class=w>
</span><span class=l>/path/to/errors.json</span><span class=w>
+</span></span></span></code></pre></div><p>See YAML schema <a
href=https://beam.apache.org/documentation/sdks/yaml-schema/>info</a> for
another use of error_handling in a schema context.</p></div></div><footer
class=footer><div class=footer__contained><div class=footer__cols><div
class="footer__cols__col footer__cols__col__logos"><div
class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg
class=footer__logo alt="Beam logo"></div><div
class=footer__cols__col__logo><img src=/i [...]
<a href=https://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div><div class="footer__cols__col
footer__cols__col__logos"><div class=footer__cols__col--group><div
class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git
a/website/generated-content/documentation/sdks/yaml-errors/index.html
b/website/generated-content/documentation/sdks/yaml-schema/index.html
similarity index 55%
copy from website/generated-content/documentation/sdks/yaml-errors/index.html
copy to website/generated-content/documentation/sdks/yaml-schema/index.html
index 82042b6f753..cc10e63f705 100644
--- a/website/generated-content/documentation/sdks/yaml-errors/index.html
+++ b/website/generated-content/documentation/sdks/yaml-schema/index.html
@@ -1,4 +1,4 @@
-<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta
http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport
content="width=device-width,initial-scale=1"><title>Apache Beam YAML Error
Handling</title><meta name=description content="Apache Beam is an open source,
unified model and set of language-specific SDKs for defining and executing data
processing workflows, and also data ingestion and integration flows, supporting
Enterprise Integration Patterns (EIPs) a [...]
+<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta
http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport
content="width=device-width,initial-scale=1"><title>Apache Beam YAML
Schema</title><meta name=description content="Apache Beam is an open source,
unified model and set of language-specific SDKs for defining and executing data
processing workflows, and also data ingestion and integration flows, supporting
Enterprise Integration Patterns (EIPs) and Domai [...]
<script type=text/javascript
src=/js/language-switch-v2.min.121952b7980b920320ab229551857669209945e39b05ba2b433a565385ca44c6.js
defer></script>
<script type=text/javascript
src=/js/fix-menu.min.039174b67107465f2090a493f91e126f7aa797f29420f9edab8a54d9dd4b3d2d.js
defer></script>
<script type=text/javascript
src=/js/section-nav.min.1405fd5e70fab5f6c54037c269b1d137487d8f3d1b3009032525f6db3fbce991.js
defer></script>
@@ -8,7 +8,7 @@
<script type=text/javascript
src=/js/calendar.min.336664054fa0f52b08bbd4e3c59b5cb6d63dcfb2b4d602839746516b0817446b.js
defer></script>
<script type=text/javascript
src=/js/fix-playground-nested-scroll.min.0283f1037cb1b9d5074c6eaf041292b524a8148a7cdb803d5ccd6d1fc4eb3253.js
defer></script>
<script type=text/javascript
src=/js/anchor-content-jump-fix.min.22d3240f81632e4c11179b9d2aaf37a40da9414333c43aa97344e8b21a7df0e4.js
defer></script>
-<link rel=alternate type=application/rss+xml title="Apache Beam"
href=/feed.xml><link rel=canonical href=/documentation/sdks/yaml-errors/
data-proofer-ignore><link rel="shortcut icon" type=image/x-icon
href=/images/favicon.ico><link rel=stylesheet
href=https://use.fontawesome.com/releases/v5.4.1/css/all.css
integrity=sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz
crossorigin=anonymous><link rel=stylesheet
href=https://unpkg.com/swiper@8/swiper-bundle.min.css><scr [...]
+<link rel=alternate type=application/rss+xml title="Apache Beam"
href=/feed.xml><link rel=canonical href=/documentation/sdks/yaml-schema/
data-proofer-ignore><link rel="shortcut icon" type=image/x-icon
href=/images/favicon.ico><link rel=stylesheet
href=https://use.fontawesome.com/releases/v5.4.1/css/all.css
integrity=sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz
crossorigin=anonymous><link rel=stylesheet
href=https://unpkg.com/swiper@8/swiper-bundle.min.css><scr [...]
<script>(function(e,t,n,s,o,i){e.hj=e.hj||function(){(e.hj.q=e.hj.q||[]).push(arguments)},e._hjSettings={hjid:2182187,hjsv:6},o=t.getElementsByTagName("head")[0],i=t.createElement("script"),i.async=1,i.src=n+e._hjSettings.hjid+s+e._hjSettings.hjsv,o.appendChild(i)})(window,document,"https://static.hotjar.com/c/hotjar-",".js?sv=";)</script></head><body
class=body data-spy=scroll data-target=.page-nav data-offset=0><nav
class="navigation-bar-mobile header navbar navbar-fixed-top"><div class [...]
<a class=navbar-link href=/get-started/>Get Started</a>
<a class=navbar-link href=/documentation/>Documentation</a>
@@ -20,7 +20,7 @@
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span>
<span class=icon-bar></span>
-<span class=icon-bar></span></button><ul class="nav navbar-nav"><li><div
class=searchBar-mobile><script>(function(){var
t,n="012923275103528129024:4emlchv9wzi",e=document.createElement("script");e.type="text/javascript",e.async=!0,e.src="https://cse.google.com/cse.js?cx="+n,t=document.getElementsByTagName("script")[0],t.parentNode.insertBefore(e,t)})()</script><gcse:search></gcse:search></div></li><li><a
class=navbar-link href=/about>About</a></li><li><a class=navbar-link
href=/get-start [...]
+<span class=icon-bar></span></button><ul class="nav navbar-nav"><li><div
class=searchBar-mobile><script>(function(){var
t,n="012923275103528129024:4emlchv9wzi",e=document.createElement("script");e.type="text/javascript",e.async=!0,e.src="https://cse.google.com/cse.js?cx="+n,t=document.getElementsByTagName("script")[0],t.parentNode.insertBefore(e,t)})()</script><gcse:search></gcse:search></div></li><li><a
class=navbar-link href=/about>About</a></li><li><a class=navbar-link
href=/get-start [...]
Apache
<span class=arrow-icon><svg xmlns="http://www.w3.org/2000/svg"; width="20"
height="20" fill="none" viewBox="0 0 20 20"><circle cx="10" cy="10" r="10"
fill="#ff6d00"/><path stroke="#fff" stroke-linecap="round"
stroke-linejoin="round" stroke-width="2" d="M8.535 5.28l4.573 4.818-4.573
4.403"/></svg></span></a><ul class="dropdown-menu dropdown-menu-right"><li><a
target=_blank href=https://www.apache.org/>ASF Homepage</a></li><li><a
target=_blank href=https://www.apache.org/licenses/>License</ [...]
<a class=navbar-link href=/get-started/>Get Started</a><li class="dropdown
navbar-dropdown navbar-dropdown-documentation"><a href=# class="dropdown-toggle
navbar-link" role=button aria-haspopup=true aria-expanded=false>Documentation
@@ -28,183 +28,94 @@
<a class=navbar-link href=/community/>Community</a>
<a class=navbar-link href=/contribute/>Contribute</a>
<a class=navbar-link href=/blog/>Blog</a>
-<a class=navbar-link href=/case-studies/>Case Studies</a></div><div
id=iconsBar><a type=button onclick=showSearch()><svg
xmlns="http://www.w3.org/2000/svg"; width="25" height="24" fill="none"
viewBox="0 0 25 24"><path stroke="#ff6d00" stroke-linecap="round"
stroke-linejoin="round" stroke-width="2.75" d="M10.191 17c3.866.0 7-3.134
7-7s-3.134-7-7-7-7 3.134-7 7 3.134 7 7 7zm11 4-6-6"/></svg></a><a target=_blank
href=https://github.com/apache/beam/edit/master/website/www/site/content/en/docum
[...]
+<a class=navbar-link href=/case-studies/>Case Studies</a></div><div
id=iconsBar><a type=button onclick=showSearch()><svg
xmlns="http://www.w3.org/2000/svg"; width="25" height="24" fill="none"
viewBox="0 0 25 24"><path stroke="#ff6d00" stroke-linecap="round"
stroke-linejoin="round" stroke-width="2.75" d="M10.191 17c3.866.0 7-3.134
7-7s-3.134-7-7-7-7 3.134-7 7 3.134 7 7 7zm11 4-6-6"/></svg></a><a target=_blank
href=https://github.com/apache/beam/edit/master/website/www/site/content/en/docum
[...]
Apache
<span class=arrow-icon><svg xmlns="http://www.w3.org/2000/svg"; width="20"
height="20" fill="none" viewBox="0 0 20 20"><circle cx="10" cy="10" r="10"
fill="#ff6d00"/><path stroke="#fff" stroke-linecap="round"
stroke-linejoin="round" stroke-width="2" d="M8.535 5.28l4.573 4.818-4.573
4.403"/></svg></span></a><ul class=dropdown-menu><li><a
class=navbar-dropdown-menu-link target=_blank href=https://www.apache.org/>ASF
Homepage</a></li><li><a class=navbar-dropdown-menu-link target=_blank href=
[...]
<a type=button onclick=endSearch()><svg xmlns="http://www.w3.org/2000/svg";
width="25" height="25" fill="none" viewBox="0 0 25 25"><path stroke="#ff6d00"
stroke-linecap="round" stroke-linejoin="round" stroke-width="2.75" d="M21.122
20.827 4.727 4.432M21.122 4.43 4.727 20.827"/></svg></a></div></div></nav><div
class=header-push></div><div class="top-banners swiper"><div
class=swiper-wrapper><div class=swiper-slide><a
href=https://tour.beam.apache.org><img class=banner-img-desktop src=/imag [...]
<img class=banner-img-mobile
src=/images/banners/tour-of-beam/tour-of-beam-mobile.png alt="Start Tour of
Beam"></a></div><div class=swiper-slide><a
href=https://beam.apache.org/documentation/ml/overview/><img
class=banner-img-desktop
src=/images/banners/machine-learning/machine-learning-desktop.jpg alt="Machine
Learning">
<img class=banner-img-mobile
src=/images/banners/machine-learning/machine-learning-mobile.jpg alt="Machine
Learning"></a></div></div><div class=swiper-pagination></div><div
class=swiper-button-prev></div><div
class=swiper-button-next></div></div><script
src=/js/swiper-bundle.min.min.e0e8f81b0b15728d35ff73c07f42ddbb17a108d6f23df4953cb3e60df7ade675.js></script>
<script
src=/js/sliders/top-banners.min.afa7d0a19acf7a3b28ca369490b3d401a619562a2a4c9612577be2f66a4b9855.js></script>
-<script>function showSearch(){addPlaceholder();var
e,t=document.querySelector(".searchBar");t.classList.remove("disappear"),e=document.querySelector("#iconsBar"),e.classList.add("disappear")}function
addPlaceholder(){$("input:text").attr("placeholder","What are you looking
for?")}function endSearch(){var
e,t=document.querySelector(".searchBar");t.classList.add("disappear"),e=document.querySelector("#iconsBar"),e.classList.remove("disappear")}function
blockScroll(){$("body").toggleClass(" [...]
-data that is malformatted, doesn’t handle the proper preconditions, or
otherwise
-breaks during processing. Generally any such record will cause the pipeline to
-permanently fail, but often it is desirable to allow the pipeline to continue,
-re-directing bad records to another path for special handling or simply
-recording them for later off-line analysis. This is often called the
-“dead letter queue” pattern.</p><p>Beam YAML has special support
for this pattern if the transform supports a
-<code>error_handling</code> config parameter with an <code>output</code> field.
-The <code>output</code> parameter is a name that must referenced as an input to
-another transform that will process the errors (e.g. by writing them out).
-For example,
-the following code will write all “good” processed records to one
file and
-any “bad” records, along with metadata about what error was
encountered,
-to a separate file.</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
-
- - type: WriteToJson
- input: MapToFields
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- input: MapToFields.my_error_output
- config:
- path: /path/to/errors.json
-</code></pre><p>Note that with <code>error_handling</code> declared,
<code>MapToFields.my_error_output</code>
-<strong>must</strong> be consumed; to ignore it will be an error. Any use is
fine, e.g.
-logging the bad records to stdout would be sufficient (though not recommended
-for a robust pipeline).</p><p>Note also that the exact format of the error
outputs is still being finalized.
-They can be safely printed and written to outputs, but their precise schema
-may change in a future version of Beam and should not yet be depended on.
-It generally contains the failed record itself as well as information about
-the error that was encountered (e.g. error messages and tracebacks).
-To recover the bad record alone one can process the error output with the
-<code>StripErrorMetadata</code> transformation.</p><p>Some transforms allow
for extra arguments in their error_handling config, e.g.
-for Python functions one can give a <code>threshold</code> which limits the
relative number
-of records that can be bad before considering the entire pipeline a
failure</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
- # If more than 10% of records throw an error, stop the pipeline.
- threshold: 0.1
-
- - type: WriteToJson
- input: MapToFields
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- input: MapToFields.my_error_output
- config:
- path: /path/to/errors.json
-</code></pre><p>One can do arbitrary further processing on these failed
records if desired,
-e.g.</p><pre tabindex=0><code>pipeline:
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- name: ComputeRatio
- input: ReadFromCsv
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: my_error_output
-
- - type: StripErrorMetadata
- name: FailedRecordsWithoutMetadata
- # Takes the error information from ComputeRatio and returns just the
- # failing records themselves for another attempt with a different
- # transform.
- input: ComputeRatio.my_error_output
-
- - type: MapToFields
- name: ComputeRatioForBadRecords
- input: FailedRecordsWithoutMetadata
- config:
- language: python
- fields:
- col1: col1
- ratio: col2 / (col3 + 1)
- error_handling:
- output: still_bad
-
- - type: WriteToJson
- # Takes as input everything from the "success" path of both
transforms.
- input: [ComputeRatio, ComputeRatioForBadRecords]
- config:
- path: /path/to/output.json
-
- - type: WriteToJson
- name: WriteErrorsToJson
- # These failed the first and the second transform.
- input: ComputeRatioForBadRecords.still_bad
- config:
- path: /path/to/errors.json
-</code></pre><p>When using the <code>chain</code> syntax, the required error
consumption can happen
-in an <code>extra_transforms</code> block.</p><pre tabindex=0><code>pipeline:
- type: chain
- transforms:
- - type: ReadFromCsv
- config:
- path: /path/to/input*.csv
-
- - type: MapToFields
- name: SomeStep
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- ratio: col2 / col3
- error_handling:
- output: errors
-
- - type: MapToFields
- name: AnotherStep
- config:
- language: python
- fields:
- col1: col1
- # This could raise a divide-by-zero error.
- inverse_ratio: 1 / ratio
- error_handling:
- output: errors
-
- - type: WriteToJson
- config:
- path: /path/to/output.json
-
- extra_transforms:
- - type: WriteToJson
- name: WriteErrors
- input: [SomeStep.errors, AnotherStep.errors]
- config:
- path: /path/to/errors.json
-</code></pre></div></div><footer class=footer><div
class=footer__contained><div class=footer__cols><div class="footer__cols__col
footer__cols__col__logos"><div class=footer__cols__col__logo><img
src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div
class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg
class=footer__logo alt="Apache logo"></div></div><div class=footer-wrapper><div
class=wrapper-grid><div class=footer__cols__col><div class=footer__c [...]
+<script>function showSearch(){addPlaceholder();var
e,t=document.querySelector(".searchBar");t.classList.remove("disappear"),e=document.querySelector("#iconsBar"),e.classList.add("disappear")}function
addPlaceholder(){$("input:text").attr("placeholder","What are you looking
for?")}function endSearch(){var
e,t=document.querySelector(".searchBar");t.classList.add("disappear"),e=document.querySelector("#iconsBar"),e.classList.remove("disappear")}function
blockScroll(){$("body").toggleClass(" [...]
+data that is malformed, doesn’t meet preconditions, or otherwise causes
issues
+during processing.</p><p>Beam YAML helps the user detect and capture these
issues by using the optional
+<code>output_schema</code> configuration, which is available for any transform
in the YAML
+SDK. For example, the following code creates a few “good” records
and specifies
+that the output schema from the <code>Create</code> transform should have
records that
+follow the expected schema: <code>sdk</code> as a string and <code>year</code>
as an integer.</p><div class=highlight><pre tabindex=0 class=chroma><code
class=language-yaml data-lang=yaml><span class=line><span class=cl><span
class=nt>pipeline</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>chain</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>Create</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>elements</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>sdk: MapReduce, year</span><span class=p>:</span><span
class=w> </span><span class=m>2004</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>sdk: MillWheel, year</span><span class=p>:</span><span
class=w> </span><span class=m>2008</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output_schema</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>object</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>properties</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>sdk</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>string</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>year</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>integer</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>AssertEqual</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>elements</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>sdk: MapReduce, year</span><span class=p>:</span><span
class=w> </span><span class=m>2004</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>sdk: MillWheel, year</span><span class=p>:</span><span
class=w> </span><span class=m>2008</span>}<span class=w>
+</span></span></span></code></pre></div><p>However, a user will more likely
want to detect and handle schema errors. If a
+transform has a built-in error_handling configuration, the user can specify
that
+error_handling configuration and any errors found will be appended to the
+transform error_handling output. For example, the following code will
+create a few “good” and “bad” records with a specified
schema of <code>user</code> as a
+string and <code>timestamp</code> as a boolean. The <code>alice</code> row
will fail in the standard
+way because of not being an integer for the AssignTimestamps transform, while
+the <code>bob</code> row will fail because after the AssignTimestamp
transformation, the
+output row will have the timestamp as an integer when it should be a
boolean.</p><div class=highlight><pre tabindex=0 class=chroma><code
class=language-yaml data-lang=yaml><span class=line><span class=cl><span
class=nt>pipeline</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>composite</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>transforms</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>Create</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>CreateVisits</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>elements</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>user: alice, timestamp</span><span
class=p>:</span><span class=w> </span><span
class=s2>"not-valid"</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>user: bob, timestamp</span><span class=p>:</span><span
class=w> </span><span class=m>3</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>AssignTimestamps</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>CreateVisits</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>timestamp</span><span class=p>:</span><span class=w>
</span><span class=l>timestamp</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>error_handling</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output</span><span class=p>:</span><span class=w>
</span><span class=l>invalid_rows</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>output_schema</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>object</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>properties</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>user</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>string</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>timestamp</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>boolean</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>MapToFields</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>name</span><span class=p>:</span><span class=w>
</span><span class=l>ExtractInvalidTimestamp</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>AssignTimestamps.invalid_rows</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>language</span><span class=p>:</span><span class=w>
</span><span class=l>python</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>fields</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>user</span><span class=p>:</span><span class=w>
</span><span class=s2>"element.user"</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>timestamp</span><span class=p>:</span><span class=w>
</span><span class=s2>"element.timestamp"</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>AssertEqual</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>ExtractInvalidTimestamp</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>elements</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>user</span><span class=p>:</span><span class=w>
</span><span class=s2>"alice"</span><span class=nt>,
timestamp</span><span class=p>:</span><span class=w> </span><span
class=s2>"not-valid"</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- {<span class=nt>user: bob, timestamp</span><span class=p>:</span><span
class=w> </span><span class=m>3</span>}<span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span>- <span class=nt>type</span><span class=p>:</span><span class=w>
</span><span class=l>AssertEqual</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>input</span><span class=p>:</span><span class=w>
</span><span class=l>AssignTimestamps</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>config</span><span class=p>:</span><span class=w>
+</span></span></span><span class=line><span class=cl><span class=w>
</span><span class=nt>elements</span><span class=p>:</span><span class=w>
</span><span class=p>[]</span><span class=w>
+</span></span></span></code></pre></div><p>WARNING: If a transform
doesn’t have the error_handling configuration available
+and a user chooses to use this optional output_schema feature, any failures
+found will result in the entire pipeline failing. If the user would still like
+to have some kind of output schema validation, please use the
ValidateWithSchema
+transform instead.</p><p>NOTE: When using the output_schema config, the main
output key to validate on
+will be determined based on these criteria:</p><ol><li>An output with the key
‘output’.</li><li>An output with the key
‘good’.</li><li>The single output if there is only
one.</li></ol><p>Failures will result if the main output cannot be determined
because there are
+multiple outputs and none are named ‘output’ or
‘good’.</p><p>For more detailed information on error handling, see
this <a
href=https://beam.apache.org/documentation/sdks/yaml-errors/>page</a>.</p></div></div><footer
class=footer><div class=footer__contained><div class=footer__cols><div
class="footer__cols__col footer__cols__col__logos"><div
class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg
class=footer__logo alt="Beam logo"></div><div class=foot [...]
<a href=https://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam
logo, and the Apache feather logo are either registered trademarks or
trademarks of The Apache Software Foundation. All other products or name brands
are trademarks of their respective holders, including The Apache Software
Foundation.</div></div><div class="footer__cols__col
footer__cols__col__logos"><div class=footer__cols__col--group><div
class=footer__cols__col__logo><a href=https://github.com/apache/beam><im [...]
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml
b/website/generated-content/sitemap.xml
index 7492a03cfaf..1737c8026de 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.67.0/</loc><lastmod>2025-09-08T09:20:42-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2025-09-08T09:20:42-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2025-09-08T09:20:42-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2025-09-08T09:20:42-07:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";
xmlns:xhtml="http://www.w3.org/1999/xhtml";><url><loc>/blog/beam-2.67.0/</loc><lastmod>2025-09-08T16:51:33-04:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2025-09-08T16:51:33-04:00</lastmod></url><url><loc>/blog/</loc><lastmod>2025-09-08T16:51:33-04:00</lastmod></url><url><loc>/categories/</loc><lastmod>2025-09-08T16:51:33-04:00</lastmod></url><url><loc>/catego
[...]
\ No newline at end of file
