This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 0b495f8c508389fed8a9f467104bc03b15d84b83 Author: melissa <meliss...@google.com> AuthorDate: Mon Sep 25 13:52:15 2017 -0700 Add new execution model page and associated images --- src/_includes/header.html | 1 + src/documentation/execution-model.md | 163 +++++++++++++++++++++ src/documentation/index.md | 9 +- src/images/execution_model_bundling.svg | 33 +++++ .../execution_model_bundling_coupled_failure.svg | 38 +++++ src/images/execution_model_bundling_gantt.svg | 33 +++++ src/images/execution_model_bundling_gantt_max.svg | 45 ++++++ src/images/execution_model_bundling_multi.svg | 35 +++++ .../execution_model_bundling_multi_gantt.svg | 34 +++++ src/images/execution_model_failure_retry.svg | 36 +++++ 10 files changed, 423 insertions(+), 4 deletions(-) diff --git a/src/_includes/header.html b/src/_includes/header.html index 839b4ea..1981fb1 100644 --- a/src/_includes/header.html +++ b/src/_includes/header.html @@ -35,6 +35,7 @@ <li role="separator" class="divider"></li> <li class="dropdown-header">Beam Concepts</li> <li><a href="{{ site.baseurl }}/documentation/programming-guide/">Programming Guide</a></li> + <li><a href="{{ site.baseurl }}/documentation/execution-model/">Execution Model</a></li> <li><a href="{{ site.baseurl }}/documentation/resources/">Additional Resources</a></li> <li role="separator" class="divider"></li> <li class="dropdown-header">Pipeline Fundamentals</li> diff --git a/src/documentation/execution-model.md b/src/documentation/execution-model.md new file mode 100644 index 0000000..a8ebee0 --- /dev/null +++ b/src/documentation/execution-model.md @@ -0,0 +1,163 @@ +--- +layout: default +title: "Beam Execution Model" +permalink: /documentation/execution-model/ +--- + +# Apache Beam Execution Model + +The Beam model allows runners to execute your pipeline in different ways. +Depending on the runner’s choices, you may observe various effects as a result. +This page describes the effects of these choices so you can better understand +how Beam pipelines execute. + +**Table of Contents:** +* TOC +{:toc} + +## Processing of elements + +The serialization and communication of elements between machines is one of the +most expensive operations in a distributed execution of your pipeline. Avoiding +this serialization may require re-processing elements after failures or may +limit the distribution of output to other machines. + +The runner processes elements on many machines and may serialize elements in +between machines for other communication and persistence reasons. + +### Serialization and communication + +The runner may serialize elements for communication or persistence. + +The runner may decide to transfer elements between transforms in a variety of +ways, such as: + +1. Routing elements to a worker for processing as part of a grouping operation. + This may involve serializing elements and sorting them by their key. +1. Redistributing elements between workers to adjust parallelism. This may + involve serializing elements and communicating them to other workers. +1. Using the elements in a side input to a `ParDo`. This may require + serializing the elements and broadcasting them to all the workers executing + the `ParDo`. +1. Passing elements between transforms that are running on the same worker. + This may avoid having to serialize the elements, and instead just passing + them in memory. + +Additionally, the runner may serialize and persist elements for other reasons: + +1. When used as part of a Stateful `DoFn`, the runner may persist values to some + state mechanism. +1. When committing the results of processing, the runner may persist the outputs + as a checkpoint. + +### Bundling and persistence + +Beam pipelines often focus on ["embarassingly parallel"](https://en.wikipedia.org/wiki/Embarrassingly_parallel) +problems. Because of this, the APIs emphasize processing elements in parallel, +which makes it difficult to express things like "assign a sequence number to +each element in a PCollection." This is intentional since such algorithms are +much more likely to suffer from scalability problems. + +Processing all elements in parallel also has some drawbacks. Specifically, it +makes it impossible to batch any operations, such as writing elements to a sink +or checkpointing progress during processing. + +Instead of processing all elements simultaneously, the elements in a +`PCollection` are processed in _bundles_. The division of the collection into +bundles is arbitrary and selected by the runner. This allows the runner to +choose an appropriate middle-ground between persisting results after every +element, and having to retry everything if there is a failure. + +## Failures and parallelism within and between transforms + +In this section, we discuss how elements in the input collection are processed +in parallel, and how transforms are retried when failures occur. + +### Data-parallelism within one transform {#data-parallelism} + +The bundling of elements when executing a single `ParDo` may look something like +this. In this diagram, a collection with 9 elements is divided into 2 +bundles: + +![bundling]({{ site.baseurl }}/images/execution_model_bundling.svg) + +When the `ParDo` executes, these bundles may be processed in parallel worker +threads. The elements in each bundle are processed in sequence, as shown in this +diagram: + +![bundling_gantt]({{ site.baseurl }}/images/execution_model_bundling_gantt.svg) + +Since elements cannot be split, the maximum parallelism for a transform depends +on the number of elements in the collection. In this case, the maximum +parallelism is 9: + +![bundling_gantt_max]({{ site.baseurl }}/images/execution_model_bundling_gantt_max.svg) + +Note: Splittable ParDo allows splitting the processing of a single input across +multiple bundles. This feature is still a work in progress, but may already be +useful in some cases. + +### Dependent-parallelism between transforms {#dependent-parallellism} + +When two transforms are connected as shown below, the runner may choose to +execute them in a way such that the bundling of the two transforms are +dependent. + +![bundling_multi]({{ site.baseurl }}/images/execution_model_bundling_multi.svg) + +In this picture, `ParDo1` and `ParDo2` are _dependently parallel_ if the output +of `ParDo1` for a given element must be processed on the same worker thread. + +![bundling_multi_gantt.svg]({{ site.baseurl }}/images/execution_model_bundling_multi_gantt.svg) + +For example, two `ParDo` transforms in sequence may be _dependently parallel_ if +the runner chooses to execute the consuming transform on each output from the +producing transform without altering the bundling in between. + +Executing transforms this way allows a runner to avoid redistributing elements +between workers, saving on communication costs. However, the maximum parallelism +now depends on the maximum parallelism of the first of the dependently parallel +steps. + +### Failures within one transform + +If processing of an element within a bundle fails, the entire bundle fails. The +elements in the bundle must be retried (otherwise the entire pipeline fails), +although they need not be retried with the same bundling. + +In the following illustration, two elements (in green) were successfully +processed. The third element’s processing failed, and there are three elements +(in yellow) still to be processed. We see that the same elements were retried +and processing successfully completed. Note that as shown, the retry does not +necessarily happen in the same worker thread as the original attempt. + +Because we encountered a failure while processing an element in the input +bundle, we had to reprocess all of the elements in the input bundle. Thus, the +entire output bundle must be thrown away since all of the results it contains +will be recomputed. + +![failure_retry]({{ site.baseurl }}/images/execution_model_failure_retry.svg) + +If the failed transform is a `ParDo`, then the `DoFn` instance is torn down and +abandoned. + +### Coupled failure: Failures between transforms + +If a failure to process an element in `ParDo2` causes `ParDo1` to re-execute, +these two steps are said to be _co-failing_. + +In the following illustration, two `ParDo` transforms are processing elements. +While processing an element, the second `ParDo` fails (which is shown in red). +As a result, the runner must discard and recompute the output of the second +`ParDo`. Because the runner was executing the two `ParDo`s together, the output +bundle from the first `ParDo` must also be thrown away, and the elements in the +input bundle must be retried. These two `ParDo`s are co-failing. + +![bundling_coupled failure]({{ site.baseurl }}/images/execution_model_bundling_coupled_failure.svg) + +All `DoFns` that experience coupled failures are terminated and must be torn +down since they aren’t following the normal `DoFn` lifecycle . + +Executing transforms this way allows a runner to avoid persisting elements +between transforms, saving on persistence costs. + diff --git a/src/documentation/index.md b/src/documentation/index.md index d9cb4ac..40ed972 100644 --- a/src/documentation/index.md +++ b/src/documentation/index.md @@ -11,12 +11,13 @@ redirect_from: This section provides in-depth conceptual information and reference material for the Beam Model, SDKs, and Runners: -## Concepts +## Concepts Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners. * The [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/) introduces all the key Beam concepts. -* Visit [Additional Resources]({{ site.baseurl }}/documentation/resources/) for some of our favorite articles and talks about Beam. +* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/execution-model/) to better understand how pipelines execute. +* Visit [Additional Resources]({{ site.baseurl }}/documentation/resources/) for some of our favorite articles and talks about Beam. ## Pipeline Fundamentals @@ -28,7 +29,7 @@ Learn about the Beam Programming Model and the concepts common to all Beam SDKs Find status and reference information on all of the available Beam SDKs. -* [Java SDK]({{ site.baseurl }}/documentation/sdks/java/) +* [Java SDK]({{ site.baseurl }}/documentation/sdks/java/) * [Python SDK]({{ site.baseurl }}/documentation/sdks/python/) ## Runners @@ -42,7 +43,7 @@ A Beam Runner runs a Beam pipeline on a specific (often distributed) data proces * [FlinkRunner]({{ site.baseurl }}/documentation/runners/flink/): Runs on [Apache Flink](http://flink.apache.org). * [SparkRunner]({{ site.baseurl }}/documentation/runners/spark/): Runs on [Apache Spark](http://spark.apache.org). * [DataflowRunner]({{ site.baseurl }}/documentation/runners/dataflow/): Runs on [Google Cloud Dataflow](https://cloud.google.com/dataflow), a fully managed service within [Google Cloud Platform](https://cloud.google.com/). -* [GearpumpRunner]({{ site.baseurl }}/documentation/runners/gearpump/): Runs on [Apache Gearpump (incubating)](http://gearpump.apache.org). +* [GearpumpRunner]({{ site.baseurl }}/documentation/runners/gearpump/): Runs on [Apache Gearpump (incubating)](http://gearpump.apache.org). ### Choosing a Runner diff --git a/src/images/execution_model_bundling.svg b/src/images/execution_model_bundling.svg new file mode 100644 index 0000000..7a257e3 --- /dev/null +++ b/src/images/execution_model_bundling.svg @@ -0,0 +1,33 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="144.38" width="419.8" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(25.429,-781.45258)"> + <g id="g3135" transform="translate(8.1428566,0)"> + <rect id="rect2987" ry="11.782" height="34.286" width="197.14" y="876.55" x="26.429" fill="#ff7f2a"/> + <text id="text2991" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="40px" y="901.14917" x="88.930664" font-weight="normal" font-style="normal" fill="#000000"><tspan id="tspan2993" y="901.14917" x="88.930664" font-size="20px">ParDo1</tspan></text> + </g> + <rect id="rect3025" height="36.429" width="237.14" y="815.51" x="14.571" fill="#afc6e9"/> + <path id="path3029" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(-14,812.36218)" fill="#59F"/> + <path id="path3029-9" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(94.85714,828.79075)" fill="#59F"/> + <path id="path3029-1" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(7.42857,826.64789)" fill="#59F"/> + <path id="path3029-2" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(40.285713,808.79075)" fill="#59F"/> + <path id="path3029-7" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(40.285713,824.50504)" fill="#59F"/> + <path id="path3029-0" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(75.285713,815.93361)" fill="#59F"/> + <path id="path3029-93" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(20.999998,814.50504)" fill="#59F"/> + <path id="path3029-6" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(108.14286,813.79075)" fill="#59F"/> + <path id="path3029-06" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(136,821.64789)" fill="#59F"/> + <text id="text3153" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" y="841.17542" x="257.14285" font-weight="normal" font-style="normal" fill="#000000"><tspan id="tspan3155" y="841.17542" x="257.14285">Collection</tspan></text> + <rect id="rect3161" height="51.429" width="5" y="808.01" x="110.71" fill="#1c2422"/> + <text id="text3163" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" y="811.64789" x="15.000003" font-weight="normal" font-style="normal" fill="#000000"><tspan id="tspan3165" y="811.64789" x="15.000003">Bundle 1</tspan></text> + <text id="text3163-1" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" y="811.64789" x="128.29799" font-weight="normal" font-style="normal" fill="#000000"><tspan id="tspan3165-8" y="811.64789" x="128.29799">Bundle 2</tspan></text> + </g> +</svg> diff --git a/src/images/execution_model_bundling_coupled_failure.svg b/src/images/execution_model_bundling_coupled_failure.svg new file mode 100644 index 0000000..3c5b015 --- /dev/null +++ b/src/images/execution_model_bundling_coupled_failure.svg @@ -0,0 +1,38 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="303.81" width="589.09" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(20.046875,-752.34875)"> + <path id="path3207" stroke-linejoin="miter" d="m50,805.93,0,196.43h436.43" stroke="#000" stroke-linecap="butt" stroke-width="1px" fill="none"/> + <text id="text3209" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="779.505" x="20" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3211" y="779.505" x="20">Worker</tspan><tspan id="tspan3219" y="799.505" x="20">Thread</tspan></text> + <text id="text3213" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="1020.9336" x="445" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3215" y="1020.9336" x="445">Processing</tspan><tspan id="tspan3217" y="1040.9336" x="445">Time</tspan></text> + <g id="g3140-54" transform="matrix(2.3376624,0,0,1.2745098,24.367395,-85.011242)"> + <rect id="rect3025-7-1-9-8-74" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-54-9" transform="matrix(2.3376624,0,0,1.2745098,152.93886,-85.011242)" fill="#AFA"> + <rect id="rect3025-7-1-9-8-74-5" height="36.429" width="55" y="806.93" x="14.143" fill="#AFA"/> + </g> + <g id="g3140-54-1" transform="matrix(1.6103896,0,0,1.2745098,37.081674,-143.43981)"> + <rect id="rect3025-7-1-9-8-74-7" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-54-9-6" transform="matrix(2.5324676,0,0,1.2745098,112.4695,-143.43981)" fill="#AFA"> + <rect id="rect3025-7-1-9-8-74-5-1" height="36.429" width="55" y="806.93" x="14.143" fill="#AFA"/> + </g> + <path id="path3029-6" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(226.3817,907.58872)" fill="#ff2a2a"/> + <g id="g3140-54-1-9" transform="matrix(2.6753248,0,0,1.2745098,262.54504,-141.92755)"> + <rect id="rect3025-7-1-9-8-74-7-4" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-54-9-6-8" transform="matrix(1.5194805,0,0,1.2745098,423.03483,-141.92755)" fill="#AFA"> + <rect id="rect3025-7-1-9-8-74-5-1-1" height="36.429" width="55" y="806.93" x="14.143" fill="#AFA"/> + </g> + <flowRoot id="flowRoot5183" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" xml:space="preserve" font-size="20px" transform="translate(203.62946,797.53031)" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion5185"><rect id="rect5187" y="94.526" width="125.71" x="192.86" height="30.714"/></flowRegion><flowPara id="flowPara5189" font-size="24px">Retry</flowPara></flowRoot> + </g> +</svg> diff --git a/src/images/execution_model_bundling_gantt.svg b/src/images/execution_model_bundling_gantt.svg new file mode 100644 index 0000000..963ac6d --- /dev/null +++ b/src/images/execution_model_bundling_gantt.svg @@ -0,0 +1,33 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="303.81" width="589.09" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(20.046875,-752.34875)"> + <path id="path3207" stroke-linejoin="miter" d="m50,805.93,0,196.43h436.43" stroke="#000" stroke-linecap="butt" stroke-width="1px" fill="none"/> + <text id="text3209" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="16px" font-style="normal" y="779.505" x="20" font-weight="normal" fill="#000000"><tspan id="tspan3211" y="779.505" x="20">Worker</tspan><tspan id="tspan3219" y="799.505" x="20">Thread</tspan></text> + <text id="text3213" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="16px" font-style="normal" y="1020.9336" x="445" font-weight="normal" fill="#000000"><tspan id="tspan3215" y="1020.9336" x="445">Processing</tspan><tspan id="tspan3217" y="1040.9336" x="445">Time</tspan></text> + <g id="g3140" transform="translate(43.285717,0.78568637)"> + <rect id="rect3025-7-19" height="36.429" width="55" y="900.51" x="100.59" fill="#afc6e9"/> + <rect id="rect3025-7-1-4" height="36.429" width="55" y="900.51" x="160.11" fill="#afc6e9"/> + <rect id="rect3025-7-1-2-7" height="36.429" width="55" y="900.51" x="219.62" fill="#afc6e9"/> + <rect id="rect3025-7-1-9-8" height="36.429" width="55" y="900.51" x="279.14" fill="#afc6e9"/> + <rect id="rect3025-7" height="36.429" width="55" y="956.93" x="22.143" fill="#afc6e9"/> + <rect id="rect3025-7-1" height="36.429" width="55" y="956.93" x="82.571" fill="#afc6e9"/> + <rect id="rect3025-7-1-2" height="36.429" width="55" y="956.93" x="143" fill="#afc6e9"/> + <rect id="rect3025-7-1-9" height="36.429" width="55" y="956.93" x="203.43" fill="#afc6e9"/> + <text id="text3163" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" font-style="normal" y="926.17542" x="375.81967" font-weight="normal" fill="#000000"><tspan id="tspan3165" y="926.17542" x="375.81967">Bundle 1</tspan></text> + <text id="text3163-1" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" font-style="normal" y="982.604" x="260.8978" font-weight="normal" fill="#000000"><tspan id="tspan3165-8" y="982.604" x="260.8978">Bundle 2</tspan></text> + <text id="text3163-1-3" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" font-style="normal" y="982.29639" x="100.64494" font-weight="normal" fill="#000000"><tspan id="tspan3165-8-6" y="982.29639" x="100.64494">ParDo1</tspan></text> + <text id="text3435" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-family="Sans" xml:space="preserve" font-size="20px" font-style="normal" y="925.8678" x="164.21637" font-weight="normal" fill="#000000"><tspan id="tspan3437" y="925.8678" x="164.21637">ParDo1</tspan></text> + <rect id="rect3025-7-19-4" height="36.429" width="55" y="900.51" x="41.071" fill="#afc6e9"/> + </g> + </g> +</svg> diff --git a/src/images/execution_model_bundling_gantt_max.svg b/src/images/execution_model_bundling_gantt_max.svg new file mode 100644 index 0000000..661eaaa --- /dev/null +++ b/src/images/execution_model_bundling_gantt_max.svg @@ -0,0 +1,45 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="303.81" width="589.09" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(20.046875,-752.34875)"> + <path id="path3207" stroke-linejoin="miter" d="m50,805.93,0,196.43h436.43" stroke="#000" stroke-linecap="butt" stroke-width="1px" fill="none"/> + <text id="text3209" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="779.505" x="20" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3211" y="779.505" x="20">Worker</tspan><tspan id="tspan3219" y="799.505" x="20">Thread</tspan></text> + <text id="text3213" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="1020.9336" x="445" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3215" y="1020.9336" x="445">Processing</tspan><tspan id="tspan3217" y="1040.9336" x="445">Time</tspan></text> + <g id="g3140" transform="matrix(1,0,0,0.31372546,43.285717,554.56371)"> + <rect id="rect3025-7-1-9-8" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-0" transform="matrix(1,0,0,0.31372546,43.285717,597.33157)"> + <rect id="rect3025-7-1-9-8-6" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-1" transform="matrix(1,0,0,0.31372546,43.285717,682.86727)"> + <rect id="rect3025-7-1-9-8-5" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-5" transform="matrix(1,0,0,0.31372546,53.285717,618.71549)"> + <rect id="rect3025-7-1-9-8-4" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-7" transform="matrix(1,0,0,0.31372546,55.285717,704.2512)"> + <rect id="rect3025-7-1-9-8-65" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-6" transform="matrix(1,0,0,0.31372546,51.285717,575.94764)"> + <rect id="rect3025-7-1-9-8-9" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-3" transform="matrix(1,0,0,0.31372546,43.285717,640.09942)"> + <rect id="rect3025-7-1-9-8-7" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-4" transform="matrix(1,0,0,0.31372546,85.285717,661.48335)"> + <rect id="rect3025-7-1-9-8-52" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <g id="g3140-54" transform="matrix(1,0,0,0.31372546,43.285717,725.63512)"> + <rect id="rect3025-7-1-9-8-74" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + </g> +</svg> diff --git a/src/images/execution_model_bundling_multi.svg b/src/images/execution_model_bundling_multi.svg new file mode 100644 index 0000000..8d0d09c --- /dev/null +++ b/src/images/execution_model_bundling_multi.svg @@ -0,0 +1,35 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="256.02" width="317.14" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(24.327568,-613.00507)"> + <g id="g3135" transform="translate(9.24386,-180)"> + <rect id="rect2987" ry="11.782" height="34.286" width="197.14" y="876.55" x="26.429" fill="#ff7f2a"/> + <text id="text2991" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" font-size="40px" font-style="normal" y="901.14917" x="88.930664" font-family="Sans" xml:space="preserve" fill="#000000"><tspan id="tspan2993" font-size="20px" y="901.14917" x="88.930664">ParDo1</tspan></text> + </g> + <g id="g3995" transform="translate(1.1010034,0)"> + <rect id="rect3025" height="36.429" width="237.14" y="635.51" x="14.571" fill="#afc6e9"/> + <rect id="rect3161" height="51.429" width="5" y="628.01" x="110.71" fill="#1c2422"/> + <text id="text3163" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" font-size="20px" line-height="125%" xml:space="preserve" font-style="normal" y="661.17542" x="20.000004" font-family="Sans" fill="#000000"><tspan id="tspan3165" y="661.17542" x="20.000004">Bundle 1</tspan></text> + <text id="text3163-1" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" font-size="20px" line-height="125%" xml:space="preserve" font-style="normal" y="661.17542" x="136.15514" font-family="Sans" fill="#000000"><tspan id="tspan3165-8" y="661.17542" x="136.15514">Bundle 2</tspan></text> + </g> + <g id="g3135-4" transform="translate(9.24386,-56.808306)"> + <rect id="rect2987-3" ry="11.782" height="34.286" width="197.14" y="876.55" x="26.429" fill="#ff7f2a"/> + <text id="text2991-0" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" font-size="40px" font-style="normal" y="901.14917" x="88.930664" font-family="Sans" xml:space="preserve" fill="#000000"><tspan id="tspan2993-7" font-size="20px" y="901.14917" x="88.930664">ParDo2</tspan></text> + </g> + <g id="g4003" transform="translate(1.5014038e-5,0)"> + <rect id="rect3025-8" height="36.429" width="237.14" y="758.7" x="15.672" fill="#afc6e9"/> + <rect id="rect3161-9" height="51.429" width="5" y="751.2" x="155.82" fill="#1c2422"/> + <text id="text3163-2" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" font-size="20px" line-height="125%" xml:space="preserve" font-style="normal" y="784.36713" x="41.8153" font-family="Sans" fill="#000000"><tspan id="tspan3165-6" y="784.36713" x="41.8153">Bundle 3</tspan></text> + <text id="text3163-1-6" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" font-size="20px" line-height="125%" xml:space="preserve" font-style="normal" y="784.36713" x="160.82756" font-family="Sans" fill="#000000"><tspan id="tspan3165-8-4" y="784.36713" x="160.82756">Bundle 4</tspan></text> + </g> + </g> +</svg> diff --git a/src/images/execution_model_bundling_multi_gantt.svg b/src/images/execution_model_bundling_multi_gantt.svg new file mode 100644 index 0000000..689fcbc --- /dev/null +++ b/src/images/execution_model_bundling_multi_gantt.svg @@ -0,0 +1,34 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="303.81" width="589.09" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(20.046875,-752.34875)"> + <path id="path3207" stroke-linejoin="miter" d="m50,805.93,0,196.43h436.43" stroke="#000" stroke-linecap="butt" stroke-width="1px" fill="none"/> + <text id="text3209" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="779.505" x="20" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3211" y="779.505" x="20">Worker</tspan><tspan id="tspan3219" y="799.505" x="20">Thread</tspan></text> + <text id="text3213" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="1020.9336" x="445" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3215" y="1020.9336" x="445">Processing</tspan><tspan id="tspan3217" y="1040.9336" x="445">Time</tspan></text> + <g id="g3140-54" transform="matrix(2.3376624,0,0,1.2745098,24.367395,-85.011242)"> + <rect id="rect3025-7-1-9-8-74" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <flowRoot id="flowRoot4030" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" xml:space="preserve" font-size="20px" transform="translate(20.644914,751.7051)" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion4032"><rect id="rect4034" y="202.14" width="107.14" x="65" height="24.286"/></flowRegion><flowPara id="flowPara4036">ParDo1</flowPara></flowRoot> + <g id="g3140-54-9" transform="matrix(2.3376624,0,0,1.2745098,152.93886,-85.011242)" fill="#AFA"> + <rect id="rect3025-7-1-9-8-74-5" height="36.429" width="55" y="806.93" x="14.143" fill="#AFA"/> + </g> + <flowRoot id="flowRoot4030-0" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" xml:space="preserve" font-size="20px" transform="translate(149.21638,751.7051)" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion4032-4"><rect id="rect4034-8" y="202.14" width="107.14" x="65" height="24.286"/></flowRegion><flowPara id="flowPara4036-7">ParDo2</flowPara></flowRoot> + <g id="g3140-54-1" transform="matrix(3.2727274,0,0,1.2745098,13.571526,-143.43981)"> + <rect id="rect3025-7-1-9-8-74-7" height="36.429" width="55" y="806.93" x="14.143" fill="#afc6e9"/> + </g> + <flowRoot id="flowRoot4030-2" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" xml:space="preserve" font-size="20px" transform="translate(48.787789,693.27658)" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion4032-7"><rect id="rect4034-2" y="202.14" width="107.14" x="65" height="24.286"/></flowRegion><flowPara id="flowPara4036-2">ParDo1</flowPara></flowRoot> + <g id="g3140-54-9-6" transform="matrix(2.5324676,0,0,1.2745098,203.89807,-143.43981)" fill="#AFA"> + <rect id="rect3025-7-1-9-8-74-5-1" height="36.429" width="55" y="806.93" x="14.143" fill="#AFA"/> + </g> + <flowRoot id="flowRoot4030-0-0" style="letter-spacing:0px;word-spacing:0px;" line-height="125%" font-weight="normal" xml:space="preserve" font-size="20px" transform="translate(208.36595,693.27658)" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion4032-4-6"><rect id="rect4034-8-1" y="202.14" width="107.14" x="65" height="24.286"/></flowRegion><flowPara id="flowPara4036-7-5">ParDo2</flowPara></flowRoot> + </g> +</svg> diff --git a/src/images/execution_model_failure_retry.svg b/src/images/execution_model_failure_retry.svg new file mode 100644 index 0000000..bc4c097 --- /dev/null +++ b/src/images/execution_model_failure_retry.svg @@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg id="svg2" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg" height="303.81" width="589.09" version="1.1" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/"> + <metadata id="metadata7"> + <rdf:RDF> + <cc:Work rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/> + <dc:title/> + </cc:Work> + </rdf:RDF> + </metadata> + <g id="layer1" transform="translate(20.046875,-752.34875)"> + <path id="path3207" stroke-linejoin="miter" d="m50,805.93,0,196.43h436.43" stroke="#000" stroke-linecap="butt" stroke-width="1px" fill="none"/> + <text id="text3209" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="779.505" x="20" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3211" y="779.505" x="20">Worker</tspan><tspan id="tspan3219" y="799.505" x="20">Thread</tspan></text> + <text id="text3213" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" font-size="16px" font-style="normal" y="1020.9336" x="445" font-family="Sans" line-height="125%" fill="#000000"><tspan id="tspan3215" y="1020.9336" x="445">Processing</tspan><tspan id="tspan3217" y="1040.9336" x="445">Time</tspan></text> + <rect id="rect3025-7-19-4" height="36.429" width="114.29" y="900.8" x="80.357" fill="#afc6e9"/> + <path id="path3029-9" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(48.453122,912.45817)" fill="#4A0"/> + <path id="path3029-0" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(37.453124,899.60097)" fill="#4A0"/> + <path id="path3029-6" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(70.310271,897.45817)" fill="#ff2a2a"/> + <path id="path3029-06" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(98.167409,905.31527)" fill="#ffe680"/> + <path id="path3029-06-4" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(131.38169,914.95814)" fill="#ffe680"/> + <rect id="rect3025-7-19-4-9" height="36.429" width="114.29" y="850.8" x="57.81" fill="#afc6e9"/> + <path id="path3029-06-65" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(113.52455,899.01729)" fill="#ffe680"/> + <g id="g5265"> + <rect id="rect3025-7-19-4-1" height="36.429" width="114.29" y="850.8" x="215.67" fill="#afc6e9"/> + <path id="path3029-9-5" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(192.33482,862.45818)" fill="#4A0"/> + <path id="path3029-0-97" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(169.90625,858.17241)" fill="#4A0"/> + <path id="path3029-6-7" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(211.62054,853.45818)" fill="#4A0"/> + <path id="path3029-06-6" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(233.47768,855.31528)" fill="#4A0"/> + <path id="path3029-06-6-3" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(261.38169,864.0173)" fill="#4A0"/> + <flowRoot id="flowRoot5183" style="letter-spacing:0px;word-spacing:0px;" font-weight="normal" xml:space="preserve" transform="translate(47.200894,756.81605)" line-height="125%" font-size="20px" font-style="normal" font-family="Sans" fill="#000000"><flowRegion id="flowRegion5185"><rect id="rect5187" y="94.526" x="192.86" width="125.71" height="30.714"/></flowRegion><flowPara id="flowPara5189" font-size="24px">Retry</flowPara></flowRoot> + <path id="path3029-9-5-3" d="m57.857,14.286c0,2.3669-1.9188,4.2857-4.2857,4.2857s-4.2857-1.9188-4.2857-4.2857c0.001-2.367,1.919-4.286,4.286-4.286,2.3669,0,4.2857,1.9188,4.2857,4.2857z" transform="translate(183.52455,849.0173)" fill="#4A0"/> + </g> + </g> +</svg> -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" <commits@beam.apache.org>.