This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new a0db495 Publishing website 2019/10/28 10:11:16 at commit 98eb81d a0db495 is described below commit a0db495e8b70607c4bfea8c2083950d14a1179cf Author: jenkins <bui...@apache.org> AuthorDate: Mon Oct 28 10:11:17 2019 +0000 Publishing website 2019/10/28 10:11:16 at commit 98eb81d --- .../io/developing-io-overview/index.html | 3 +- .../documentation/programming-guide/index.html | 3 +- .../images/dofn-sequence-diagram.png | Bin 126448 -> 0 bytes .../images/dofn-sequence-diagram.svg | 94 ++++++++++++++++++ .../images/source-sequence-diagram.png | Bin 73657 -> 0 bytes .../images/source-sequence-diagram.svg | 106 +++++++++++++++++++++ 6 files changed, 204 insertions(+), 2 deletions(-) diff --git a/website/generated-content/documentation/io/developing-io-overview/index.html b/website/generated-content/documentation/io/developing-io-overview/index.html index 858b1e5..f2ad792 100644 --- a/website/generated-content/documentation/io/developing-io-overview/index.html +++ b/website/generated-content/documentation/io/developing-io-overview/index.html @@ -590,7 +590,8 @@ read operations in sorted key order.</p> information to IO developers such as the constraints that apply to the objects or particular cases such as streaming mode.</p> -<p><img src="/images/source-sequence-diagram.png" alt="This is a sequence diagram that shows the lifecycle of the Source" /></p> +<p><!-- The source for the sequence diagram can be found in the the SVG resource. --> +<img src="/images/source-sequence-diagram.svg" alt="This is a sequence diagram that shows the lifecycle of the Source" /></p> <h3 id="using-pardo-and-groupbykey">Using ParDo and GroupByKey</h3> diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index c18446f..350df21 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -1374,7 +1374,8 @@ transforms, including <code class="highlighter-rouge">Filter</code>, <code class apply to the objects or particular cases such as failover or instance reuse. They also give instanciation use cases.</p> -<p><img src="/images/dofn-sequence-diagram.png" alt="This is a sequence diagram that shows the lifecycle of the DoFn" /></p> +<!-- The source for the sequence diagram can be found in the the SVG resource. --> +<p><img src="/images/dofn-sequence-diagram.svg" alt="This is a sequence diagram that shows the lifecycle of the DoFn" /></p> <h4 id="groupbykey">4.2.2. GroupByKey</h4> diff --git a/website/generated-content/images/dofn-sequence-diagram.png b/website/generated-content/images/dofn-sequence-diagram.png deleted file mode 100644 index cf6570d..0000000 Binary files a/website/generated-content/images/dofn-sequence-diagram.png and /dev/null differ diff --git a/website/generated-content/images/dofn-sequence-diagram.svg b/website/generated-content/images/dofn-sequence-diagram.svg new file mode 100644 index 0000000..898b1ae --- /dev/null +++ b/website/generated-content/images/dofn-sequence-diagram.svg @@ -0,0 +1,94 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="863px" preserveAspectRatio="none" style="width:740px;height:863px;" version="1.1" viewBox="0 0 740 863" width="740px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="781.5625" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="49" y="60.8125"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stro [...] +@startuml + +hide footbox +skinparam backgroundColor transparent +skinparam shadowing false +skinparam defaultFontName "Roboto" + +skinparam sequenceArrowThickness 2 + +skinparam note { + BackgroundColor #cee2f2 + BorderColor #cee2f2 +} + +skinparam sequence { + ArrowColor #67666a + + LifeLineBorderColor #6b9fe6 + LifeLineBackgroundColor #6b9fe6 + + GroupBackgroundColor #8ac483 + GroupBorderColor #8ac483 + + ParticipantBackgroundColor #8ac483 + ParticipantBorderColor #8ac483 +} + +participant "User pipeline" as Pipeline +participant DoFn << Serializable >> +note right of DoFn: can have non-transient instance\nvariable state that will be deserialized +note right of DoFn: do not include enclosing class serializable state; use static\nnested DoFn or define as anonymous class in static method +note right of DoFn: no shared (global) static variable access (no sync mechanism) but a beam\nstate (based on engine mechanisms) can be injected to processElement +note right of DoFn: keep as pure function as possible or idempotent side effects\nbecause DoFns can be retried on failed bundles + +participant Runner + +activate Pipeline +Pipeline -> DoFn: **create DoFn ** +DoFn -> Runner: **passed instance or deserialized on workers** + +note right Pipeline: If state variables are known at pipeline construction step\ninitialize state variables by constructor + +group DoFn Lifecycle + Runner -> DoFn: **call setup** + activate Runner + activate DoFn + note right DoFn: reused instance to process other bundles on the same worker + note right DoFn: If state variables do not depend on the main pipeline program and are the\nsame for all DoFn instances initialize them in setup + group For each bundle + Runner -> DoFn: **call startBundle** + group For each element + Runner -> DoFn: **call processElement** + note right DoFn: If state variables are computed by the pipeline\npass it in a PcollectionView as a side input + DoFn - -> Runner: output + end + DoFn <- Runner: call onTimer + DoFn <- Runner: **call finishBundle** + end + DoFn -> Runner: **If DoFn is no more needed: call tearDown** + note right DoFn: Call of teardown is best effort; do not use for side effects +end + +@enduml + +PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019) +(GPL source distribution) +Java Runtime: OpenJDK Runtime Environment +JVM: OpenJDK 64-Bit Server VM +Java Version: 1.8.0_222-b10 +Operating System: Linux +Default Encoding: UTF-8 +Language: en +Country: CA +--></g></svg> diff --git a/website/generated-content/images/source-sequence-diagram.png b/website/generated-content/images/source-sequence-diagram.png deleted file mode 100644 index d544eea..0000000 Binary files a/website/generated-content/images/source-sequence-diagram.png and /dev/null differ diff --git a/website/generated-content/images/source-sequence-diagram.svg b/website/generated-content/images/source-sequence-diagram.svg new file mode 100644 index 0000000..02facd6 --- /dev/null +++ b/website/generated-content/images/source-sequence-diagram.svg @@ -0,0 +1,106 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="538px" preserveAspectRatio="none" style="width:574px;height:538px;" version="1.1" viewBox="0 0 574 538" width="574px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="435.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="250.5" y="82.0469"/><rect fill="#6B9FE6" height="152.1719" style="stroke: #6B9FE6; s [...] +@startuml + +hide footbox +skinparam backgroundColor transparent +skinparam shadowing false +skinparam defaultFontName "Roboto" + +skinparam sequenceArrowThickness 2 + +skinparam note { + BackgroundColor #cee2f2 + BorderColor #cee2f2 +} + +skinparam sequence { + ArrowColor #67666a + + LifeLineBorderColor #6b9fe6 + LifeLineBackgroundColor #6b9fe6 + + GroupBackgroundColor #8ac483 + GroupBorderColor #8ac483 + + ParticipantBackgroundColor #8ac483 + ParticipantBorderColor #8ac483 +} + +participant Runner +participant "Source" << Serializable >> +participant "Reader" +participant Middleware + +Runner -> Source: **create source** +activate Source + +Runner -> Source: get estimated size + +Source -> Middleware: **estimate size** +activate Middleware + +Source - -> Runner: size of data + +Runner -> Runner: compute size / number of executors\n = desired bundle size + +Runner -> Source: split source (desired bundle size) +note right + streaming: split based on number of executors +end note + +Source - -> Runner: list<source> +note right + streaming: runner asks the source for watermark +end note + +Runner -> Source: **for each source create a reader** + +Source -> Reader: **create a reader** +activate Reader + +Reader - -> Runner: **readers** +note right + streaming: + checkpoint +end note + +Runner -> Reader: **for each reader : start reader** + +Runner -> Reader: **read elements until none to read** + +Reader -> Middleware: **get element** + +deactivate Middleware + +Runner -> Reader: **close reader** +deactivate Reader +@enduml + +PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019) +(GPL source distribution) +Java Runtime: OpenJDK Runtime Environment +JVM: OpenJDK 64-Bit Server VM +Java Version: 1.8.0_222-b10 +Operating System: Linux +Default Encoding: UTF-8 +Language: en +Country: CA +--></g></svg>