[beam] branch asf-site updated: Publishing website 2019/11/02 19:41:42 at commit 902dddd
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 10ef080 Publishing website 2019/11/02 19:41:42 at commit 902
10ef080 is described below
commit 10ef080b84bebba55fb77232764000b67b21b9a0
Author: jenkins
AuthorDate: Sat Nov 2 19:41:42 2019 +
Publishing website 2019/11/02 19:41:42 at commit 902
---
.../contribute/become-a-committer/index.html | 2 +
.../contribute/committer-guide/index.html | 2 +
.../contribute/dependencies/index.html | 2 +
.../contribute/feature-branches/index.html | 2 +
.../contribute/get-help/index.html | 2 +
website/generated-content/contribute/index.html| 2 +
.../index.html | 92 --
.../contribute/postcommits-guides/index.html | 2 +
.../postcommits-policies-details/index.html| 2 +
.../contribute/postcommits-policies/index.html | 2 +
.../contribute/precommit-policies/index.html | 2 +
.../contribute/precommit-triage-guide/index.html | 2 +
.../contribute/ptransform-style-guide/index.html | 2 +
.../{get-help => release-blockers}/index.html | 38 ++---
.../contribute/release-guide/index.html| 2 +
.../contribute/runner-guide/index.html | 2 +
16 files changed, 105 insertions(+), 53 deletions(-)
diff --git a/website/generated-content/contribute/become-a-committer/index.html
b/website/generated-content/contribute/become-a-committer/index.html
index e7c8f83..d8ecdfd 100644
--- a/website/generated-content/contribute/become-a-committer/index.html
+++ b/website/generated-content/contribute/become-a-committer/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/committer-guide/index.html
b/website/generated-content/contribute/committer-guide/index.html
index b90ecc9..f6a5d27 100644
--- a/website/generated-content/contribute/committer-guide/index.html
+++ b/website/generated-content/contribute/committer-guide/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/dependencies/index.html
b/website/generated-content/contribute/dependencies/index.html
index da98d55..8dab9ed 100644
--- a/website/generated-content/contribute/dependencies/index.html
+++ b/website/generated-content/contribute/dependencies/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/feature-branches/index.html
b/website/generated-content/contribute/feature-branches/index.html
index 110d13f..8ebf27f 100644
--- a/website/generated-content/contribute/feature-branches/index.html
+++ b/website/generated-content/contribute/feature-branches/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/get-help/index.html
b/website/generated-content/contribute/get-help/index.html
index 2351c63..57ea611 100644
--- a/website/generated-content/contribute/get-help/index.html
+++ b/website/generated-content/contribute/get-help/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/index.html
b/website/generated-content/contribute/index.html
index 0cf3ee0..1e74534 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -188,8 +188,10 @@
Policies
+Jira priorities
Pre-commit test
policies
Post-commit test
policies
+Release blockers
diff --git a/website/generated-content/contribute/precommit-policies/index.html
b/website/generated-content/contribute/jira-priorities/index.html
similarity index 86%
copy from website/generated-content/contribute/precommit-policies/index.html
copy to website/generated-content/contribute/jira-priorities/index.html
index 731f859..dbb9058 100644
--- a/website/generated-content/contribute/precommit-policies/index.html
+++ b/website/generated-content/contribute/jira-priorities/index.html
@@ -28,7 +28,7 @@
- Pre-commit Test Policies
+ Jira Priorities
https://fonts.googleapis.com/css?family=Roboto:100,300,400;
rel="stylesheet">
@@ -42,7 +42,7 @@
- https://beam.apach
[beam] branch asf-site updated: Publishing website 2019/10/28 10:11:16 at commit 98eb81d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a0db495 Publishing website 2019/10/28 10:11:16 at commit 98eb81d a0db495 is described below commit a0db495e8b70607c4bfea8c2083950d14a1179cf Author: jenkins AuthorDate: Mon Oct 28 10:11:17 2019 + Publishing website 2019/10/28 10:11:16 at commit 98eb81d --- .../io/developing-io-overview/index.html | 3 +- .../documentation/programming-guide/index.html | 3 +- .../images/dofn-sequence-diagram.png | Bin 126448 -> 0 bytes .../images/dofn-sequence-diagram.svg | 94 ++ .../images/source-sequence-diagram.png | Bin 73657 -> 0 bytes .../images/source-sequence-diagram.svg | 106 + 6 files changed, 204 insertions(+), 2 deletions(-) diff --git a/website/generated-content/documentation/io/developing-io-overview/index.html b/website/generated-content/documentation/io/developing-io-overview/index.html index 858b1e5..f2ad792 100644 --- a/website/generated-content/documentation/io/developing-io-overview/index.html +++ b/website/generated-content/documentation/io/developing-io-overview/index.html @@ -590,7 +590,8 @@ read operations in sorted key order. information to IO developers such as the constraints that apply to the objects or particular cases such as streaming mode. - + + Using ParDo and GroupByKey diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index c18446f..350df21 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -1374,7 +1374,8 @@ transforms, including Filter, - + + 4.2.2. GroupByKey diff --git a/website/generated-content/images/dofn-sequence-diagram.png b/website/generated-content/images/dofn-sequence-diagram.png deleted file mode 100644 index cf6570d..000 Binary files a/website/generated-content/images/dofn-sequence-diagram.png and /dev/null differ diff --git a/website/generated-content/images/dofn-sequence-diagram.svg b/website/generated-content/images/dofn-sequence-diagram.svg new file mode 100644 index 000..898b1ae --- /dev/null +++ b/website/generated-content/images/dofn-sequence-diagram.svg @@ -0,0 +1,94 @@ + + +http://www.w3.org/2000/svg; xmlns:xlink="http://www.w3.org/1999/xlink; contentScriptType="application/ecmascript" contentStyleType="text/css" height="863px" preserveAspectRatio="none" style="width:740px;height:863px;" version="1.1" viewBox="0 0 740 863" width="740px" zoomAndPan="magnify">> +note right of DoFn: can have non-transient instance\nvariable state that will be deserialized +note right of DoFn: do not include enclosing class serializable state; use static\nnested DoFn or define as anonymous class in static method +note right of DoFn: no shared (global) static variable access (no sync mechanism) but a beam\nstate (based on engine mechanisms) can be injected to processElement +note right of DoFn: keep as pure function as possible or idempotent side effects\nbecause DoFns can be retried on failed bundles + +participant Runner + +activate Pipeline +Pipeline -> DoFn: **create DoFn ** +DoFn -> Runner: **passed instance or deserialized on workers** + +note right Pipeline: If state variables are known at pipeline construction step\ninitialize state variables by constructor + +group DoFn Lifecycle + Runner -> DoFn: **call setup** + activate Runner + activate DoFn + note right DoFn: reused instance to process other bundles on the same worker + note right DoFn: If state variables do not depend on the main pipeline program and are the\nsame for all DoFn instances initialize them in setup + group For each bundle +Runner -> DoFn: **call startBundle** +group For each element + Runner -> DoFn: **call processElement** + note right DoFn: If state variables are computed by the pipeline\npass it in a PcollectionView as a side input + DoFn - -> Runner: output +end +DoFn <- Runner: call onTimer +DoFn <- Runner: **call finishBundle** + end + DoFn -> Runner: **If DoFn is no more needed: call tearDown** + note right DoFn: Call of teardown is best effort; do not use for side effects +end + +@enduml + +PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019) +(GPL source distribution) +Java Runtime: OpenJDK Runtime Environment +JVM: OpenJDK 64-Bit Server VM +Java Version: 1.8.0_222-b10 +Operating System: Linux +Default Encoding: UTF-8 +Language: en +Country:
[beam] branch asf-site updated: Publishing website 2019/10/22 20:18:52 at commit 4fa1501
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f970f73 Publishing website 2019/10/22 20:18:52 at commit 4fa1501
f970f73 is described below
commit f970f7394f62a86871c9234c8cf4d30868bfba3b
Author: jenkins
AuthorDate: Tue Oct 22 20:18:52 2019 +
Publishing website 2019/10/22 20:18:52 at commit 4fa1501
---
.../python/elementwise/filter/index.html | 62 +---
.../python/elementwise/flatmap/index.html | 108 +---
.../transforms/python/elementwise/keys/index.html | 12 +--
.../python/elementwise/kvswap/index.html | 12 +--
.../transforms/python/elementwise/map/index.html | 98 --
.../transforms/python/elementwise/pardo/index.html | 12 +--
.../python/elementwise/partition/index.html| 89 +---
.../transforms/python/elementwise/regex/index.html | 112 +
.../python/elementwise/tostring/index.html | 36 +++
.../python/elementwise/values/index.html | 12 +--
.../python/elementwise/withtimestamps/index.html | 36 +++
11 files changed, 234 insertions(+), 355 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index 1ebcca4..9e927be 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -567,12 +567,10 @@ Then, we apply Filter in multiple ways to
Output PCollection after Filter:
-perennials = [
-{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
-{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
-{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
-]
-
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'}
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'}
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'}
+
[beam] branch asf-site updated: Publishing website 2019/10/22 17:34:00 at commit b3e55a3
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 685cd82 Publishing website 2019/10/22 17:34:00 at commit b3e55a3 685cd82 is described below commit 685cd82cf94b436c287be01aa1a871a52b28c43d Author: jenkins AuthorDate: Tue Oct 22 17:34:00 2019 + Publishing website 2019/10/22 17:34:00 at commit b3e55a3 --- website/generated-content/blog/2019/07/31/beam-2.14.0.html | 1 + website/generated-content/blog/2019/08/22/beam-2.15.0.html | 1 + website/generated-content/blog/2019/10/07/beam-2.16.0.html | 1 + website/generated-content/documentation/sdks/python/index.html | 1 - website/generated-content/feed.xml | 3 +++ 5 files changed, 6 insertions(+), 1 deletion(-) diff --git a/website/generated-content/blog/2019/07/31/beam-2.14.0.html b/website/generated-content/blog/2019/07/31/beam-2.14.0.html index 83c7b11..e78779a 100644 --- a/website/generated-content/blog/2019/07/31/beam-2.14.0.html +++ b/website/generated-content/blog/2019/07/31/beam-2.14.0.html @@ -249,6 +249,7 @@ For more information on changes in 2.14.0, check out the Do NOT use Python MongoDB source in this release. Python MongoDB source https://issues.apache.org/jira/browse/BEAM-5148;>added in this release has a known issue that can result in data loss. See (https://issues.apache.org/jira/browse/BEAM-7866;>BEAM-7866) for details. + Can’t install the Python SDK on macOS 10.15. See (https://issues.apache.org/jira/browse/BEAM-8368;>BEAM-8368) for details. List of Contributors diff --git a/website/generated-content/blog/2019/08/22/beam-2.15.0.html b/website/generated-content/blog/2019/08/22/beam-2.15.0.html index 6469dae..6d47b06 100644 --- a/website/generated-content/blog/2019/08/22/beam-2.15.0.html +++ b/website/generated-content/blog/2019/08/22/beam-2.15.0.html @@ -229,6 +229,7 @@ For more information on changes in 2.15.0, check out the https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. (Regression from 2.14.0) https://issues.apache.org/jira/browse/BEAM-8111;>BEAM-8111 SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0) + (https://issues.apache.org/jira/browse/BEAM-8368;>BEAM-8368) Can’t install the Python SDK on macOS 10.15. Breaking Changes diff --git a/website/generated-content/blog/2019/10/07/beam-2.16.0.html b/website/generated-content/blog/2019/10/07/beam-2.16.0.html index 28cc965..3864832 100644 --- a/website/generated-content/blog/2019/10/07/beam-2.16.0.html +++ b/website/generated-content/blog/2019/10/07/beam-2.16.0.html @@ -245,6 +245,7 @@ For more information on changes in 2.16.0, check out the Given that Python 2 will reach EOL on Jan 1 2020, Python 2 users of Beam will now receive a warning that new releases of Apache Beam will soon support Python 3 only. Filesystems not properly registered using FileIO.write in FlinkRunner. (https://issues.apache.org/jira/browse/BEAM-8303;>BEAM-8303) Performance regression in Java DirectRunner in streaming mode. (https://issues.apache.org/jira/browse/BEAM-8363;>BEAM-8363) + Can’t install the Python SDK on macOS 10.15. (https://issues.apache.org/jira/browse/BEAM-8368;>BEAM-8368) List of Contributors diff --git a/website/generated-content/documentation/sdks/python/index.html b/website/generated-content/documentation/sdks/python/index.html index 36e18c4..538c2cc 100644 --- a/website/generated-content/documentation/sdks/python/index.html +++ b/website/generated-content/documentation/sdks/python/index.html @@ -339,7 +339,6 @@ new I/O connectors. See the D for information about developing new I/O connectors and links to language-specific implementation guidance. -
[beam] branch asf-site updated: Publishing website 2019/10/22 13:41:57 at commit 99484db
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new fd9a26e Publishing website 2019/10/22 13:41:57 at commit 99484db
fd9a26e is described below
commit fd9a26e286e67c8e6a08e22dac5568c74c92f0ac
Author: jenkins
AuthorDate: Tue Oct 22 13:41:57 2019 +
Publishing website 2019/10/22 13:41:57 at commit 99484db
---
website/generated-content/contribute/release-guide/index.html | 2 +-
.../generated-content/documentation/dsls/sql/shell/index.html | 6 +++---
.../generated-content/documentation/runners/flink/index.html | 10 +++---
.../documentation/sdks/java/testing/nexmark/index.html | 6 +++---
4 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 8f4a0fa..9f256d5 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1601,7 +1601,7 @@ Thanks everyone!
Flink Local Runner
-./gradlew
:runners:flink:1.8:runQuickstartJavaFlinkLocal \
+./gradlew
:runners:flink:1.9:runQuickstartJavaFlinkLocal \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY}
\
-Pver=${RELEASE_VERSION}
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html
b/website/generated-content/documentation/dsls/sql/shell/index.html
index fbacb60..af85767 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -324,7 +324,7 @@ limitations under the License.
To use Beam SQL shell, you must first clone the https://github.com/apache/beam;>Beam SDK repository. Then, from the
root of the repository clone, execute the following commands to run the
shell:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' installDist
./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
@@ -416,7 +416,7 @@ No rows affected (0.042 seconds)
Make sure the SQL shell includes the desired runner. Add the
corresponding project id to the -Pbeam.sql.shell.bundled parameter of the
Gradle invocation (https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle;>source
code, https://github.com/apache/beam/blob/master/settings.gradle;>project
ids). For example, use the following command to include Flink runner and
KafkaIO:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' installDist
@@ -445,7 +445,7 @@ No rows affected (0.042 seconds)
You can also build your own standalone package for SQL shell using distZip or distTar tasks. For example:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' distZip
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' distZip
ls ./sdks/java/extensions/sql/shell/build/distributions/
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
diff --git a/website/generated-content/documentation/runners/flink/index.html
b/website/generated-content/documentation/runners/flink/index.html
index 4412aa3..3ab621b 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -319,7 +319,11 @@ period.
Artifact Id
- 2.17.0
+ 2.17.0
+ 1.9.x
+ beam-runners-flink-1.9
+
+
1.8.x
beam-runners-flink-1.8
@@ -492,7 +496,7 @@ download it on the Downloads page.<
To run a pipeline on an embedded Flink cluster:
/span
-1. Start the JobService endpoint: ./gradlew
:runners:flink:1.8:job-server:runShadow
+1. Start the JobService endpoint: ./gradlew
:runners:flink:1.9:job-server:runShadow
@@ -525,7 +529,7 @@ To run on a separate https://ci.apache.org/projects/flink/flink-docs-re
1. Start a Flink cluster which exposes the Rest
interface on localhost:8081 by default.
-2. Start JobService with Flink Rest endpoint:
./gradlew
:runners:flink:1.8:job-server:runShadow -PflinkMasterUrl=localhost:8081.
+2. Start JobService with Flink Rest endpoint:
./gradlew
:runners:flink:1.9:job-server:runShadow -PflinkMasterUrl=localhost:8081.
3. Submit the pipeline as above.
diff --git
a/websi
[beam] branch asf-site updated: Publishing website 2019/10/22 12:21:09 at commit 0f35360
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new f299146 Publishing website 2019/10/22 12:21:09 at commit 0f35360 f299146 is described below commit f29914628aa9e1af4af4af7ba3a98d8065dddb05 Author: jenkins AuthorDate: Tue Oct 22 12:21:09 2019 + Publishing website 2019/10/22 12:21:09 at commit 0f35360 --- .../documentation/runners/flink/index.html| 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 514e0ac..4412aa3 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -481,9 +481,16 @@ If you have a Flink JobManager running on As of now you will need a copy of Apache Beam’s source code. You can -download it on the Downloads page. In the future there will be pre-built Docker images -available. To run a pipeline on an embedded Flink cluster: - +download it on the Downloads page. + +Pre-built Docker images are available at Docker-Hub: +https://hub.docker.com/r/apachebeam/python2.7_sdk;>Python 2.7, +https://hub.docker.com/r/apachebeam/python3.5_sdk;>Python 3.5, +https://hub.docker.com/r/apachebeam/python3.6_sdk;>Python 3.6, +https://hub.docker.com/r/apachebeam/python3.7_sdk;>Python 3.7. + +To run a pipeline on an embedded Flink cluster: +/span 1. Start the JobService endpoint: ./gradlew :runners:flink:1.8:job-server:runShadow @@ -512,7 +519,7 @@ provided with the Flink JobManager address. -To run on a separate https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html;>Flink cluster: +To run on a separate https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html;>Flink cluster: 1. Start a Flink cluster which exposes the Rest interface on localhost:8081 by default.
[beam] branch asf-site updated: Publishing website 2019/10/18 21:47:27 at commit 9231341
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8daf890 Publishing website 2019/10/18 21:47:27 at commit 9231341 8daf890 is described below commit 8daf89033312dc8a470b4c21b63fdf27eb4be0f1 Author: jenkins AuthorDate: Fri Oct 18 21:47:27 2019 + Publishing website 2019/10/18 21:47:27 at commit 9231341 --- website/generated-content/contribute/index.html | 59 +++-- 1 file changed, 45 insertions(+), 14 deletions(-) diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html index 458d89b..a33c827 100644 --- a/website/generated-content/contribute/index.html +++ b/website/generated-content/contribute/index.html @@ -225,6 +225,7 @@ Share your intent Development Setup Make your change + Make reviewer’s job easier When will my change show up in an Apache Beam release? @@ -400,15 +401,24 @@ environment. See the Developers’ wiki for tips, guides, and FAQs on: Make your change - Make your code change. Every source file needs to include the Apache license header. Every new dependency needs to -have an open source license https://www.apache.org/legal/resolved.html#criteria;>compatible with Apache. - Add unit tests for your change - When your change is ready to be reviewed and merged, create a pull request. -Format commit messages and the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, + +Make your code change. Every source file needs to include the Apache license header. Every new dependency needs to +have an open source license https://www.apache.org/legal/resolved.html#criteria;>compatible with Apache. + + +Add unit tests for your change. + + +Use descriptive commit messages that make it easy to identify changes and provide a clear history. + + +When your change is ready to be reviewed and merged, create a pull request. + + +Format commit messages and the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue. -This will automatically link the pull request to the issue. -Use descriptive commit messages that make it easy to identify changes and provide a clear history. -To support efficient and quality review, avoid tiny or out-of-context changes and huge mega-changes. +This will automatically link the pull request to the issue. + The pull request and any changes pushed to it will trigger https://cwiki.apache.org/confluence/display/BEAM/Contribution+Testing+Guide#ContributionTestingGuide-Pre-commit;>pre-commit jobs. If a test fails and appears unrelated to your @@ -419,9 +429,8 @@ PR -There are other trigger phrases for post-commit tests found in -.testinfra/jenkins, but use these sparingly because post-commit -tests consume shared development resources. +Pull request template has a link to a https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md;>catalog of trigger phrases +that start various post-commit tests suites. Use these sparingly because post-commit tests consume shared development resources. Pull requests can only be merged by a Beam committer. @@ -436,9 +445,31 @@ To find a committer for your area, either: If you don’t get any response in 3 business days, email the dev@ mailing list to ask for someone to look at your pull request. - Review feedback typically leads to follow-up changes. You can add these changes as additional “fixup” commits to the -existing PR/branch. This will allow reviewer(s) to track the incremental progress. After review is complete and the -PR accepted, multiple commits should be squashed (see https://cwiki.apache.org/confluence/display/BEAM/Git+Tips;>Git workflow tips). + + +Make reviewer’s job easier + + + +Provide context for your changes in the associated JIRA issue and/or PR description. + + +Avoid huge mega-changes. + + +Review feedback typically leads to follow-up changes. It is easier to review follow-up changes when they are added as additional “fixup” commits to the +existing PR/branch. This allows reviewer(s) to track the incremental progress and focus on new changes, +and keeps comment threads attached to the code. +Please refrain from squashing new commits into reviewed commits before review is completed. +Because squashing reviewed and unreviewed commits often makes it harder to +see the the difference between the review iterations, reviewers may ask you to unsquash new changes. + + +After review is complete and the PR is accepted, fixup commits should be squashed (see https://cwiki.apache.org/confluence/display/BEAM/Git+Tips;>Git workflow tips). +Bea
[beam] branch asf-site updated: Publishing website 2019/10/16 18:28:13 at commit 3330069
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c0e759e Publishing website 2019/10/16 18:28:13 at commit 3330069 c0e759e is described below commit c0e759eba31c6a5e7369295fbacf3f8f6f81244e Author: jenkins AuthorDate: Wed Oct 16 18:28:13 2019 + Publishing website 2019/10/16 18:28:13 at commit 3330069 --- .../generated-content/documentation/patterns/pipeline-options/index.html | 1 - 1 file changed, 1 deletion(-) diff --git a/website/generated-content/documentation/patterns/pipeline-options/index.html b/website/generated-content/documentation/patterns/pipeline-options/index.html index 836e7a4..38461f7 100644 --- a/website/generated-content/documentation/patterns/pipeline-options/index.html +++ b/website/generated-content/documentation/patterns/pipeline-options/index.html @@ -545,7 +545,6 @@ limitations under the License. import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions from apache_beam.utils.value_provider import RuntimeValueProvider -from apache_beam.io import WriteToText class MyOptions(PipelineOptions): @classmethod
[beam] branch asf-site updated: Publishing website 2019/10/15 22:37:25 at commit e4ca3e1
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new e44d646 Publishing website 2019/10/15 22:37:25 at commit e4ca3e1 e44d646 is described below commit e44d6460ab55d5b99796e567915146e507f6d60a Author: jenkins AuthorDate: Tue Oct 15 22:37:25 2019 + Publishing website 2019/10/15 22:37:25 at commit e4ca3e1 --- .../documentation/io/developing-io-python/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/documentation/io/developing-io-python/index.html b/website/generated-content/documentation/io/developing-io-python/index.html index 1f3c00f..fc8fba8 100644 --- a/website/generated-content/documentation/io/developing-io-python/index.html +++ b/website/generated-content/documentation/io/developing-io-python/index.html @@ -834,9 +834,9 @@ numbers = p | 'ProduceNumbers' ReadFromCountingSource(count) class WriteToKVSink(PTransform): - def __init__(self, simplekv, url, final_table_name, **kwargs): + def __init__(self, simplekv, url, final_table_name): self._simplekv = simplekv -super(WriteToKVSink, self).__init__(**kwargs) +super(WriteToKVSink, self).__init__() self._url = url self._final_table_name = final_table_name
[beam] branch asf-site updated: Publishing website 2019/10/15 12:08:45 at commit 14b5ef8
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new ab8eefd Publishing website 2019/10/15 12:08:45 at commit 14b5ef8 ab8eefd is described below commit ab8eefdddeefcc724f5a90fb6a9adef8b065b47a Author: jenkins AuthorDate: Tue Oct 15 12:08:46 2019 + Publishing website 2019/10/15 12:08:45 at commit 14b5ef8 --- website/generated-content/documentation/runtime/environments/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/runtime/environments/index.html b/website/generated-content/documentation/runtime/environments/index.html index c1245ae..f9662c0 100644 --- a/website/generated-content/documentation/runtime/environments/index.html +++ b/website/generated-content/documentation/runtime/environments/index.html @@ -547,7 +547,7 @@ limitations under the License. # Start a Flink job server on localhost:8099 -./gradlew :runners:flink:1.5:job-server:runShadow +./gradlew :runners:flink:1.7:job-server:runShadow # Run a pipeline on the Flink job server python -m apache_beam.examples.wordcount \
[beam] branch asf-site updated: Publishing website 2019/10/11 17:01:24 at commit 713e44d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new ff5f8f1 Publishing website 2019/10/11 17:01:24 at commit 713e44d ff5f8f1 is described below commit ff5f8f1516e8cf06ea45943423835e0b5e28c748 Author: jenkins AuthorDate: Fri Oct 11 17:01:25 2019 + Publishing website 2019/10/11 17:01:24 at commit 713e44d --- .../documentation/dsls/sql/zetasql/overview/index.html| 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/dsls/sql/zetasql/overview/index.html b/website/generated-content/documentation/dsls/sql/zetasql/overview/index.html index 470bb85..a71e182 100644 --- a/website/generated-content/documentation/dsls/sql/zetasql/overview/index.html +++ b/website/generated-content/documentation/dsls/sql/zetasql/overview/index.html @@ -333,11 +333,11 @@ limitations under the License. https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md#kll16-quantile-functions;>KLL16 quantile functionsNo https://github.com/google/zetasql/blob/master/docs/numbering_functions.md;>Numbering functionsNo https://github.com/google/zetasql/blob/master/docs/bit_functions.md;>Bit functionsNo - https://github.com/google/zetasql/blob/master/docs/mathematical_functions.md;>Mathematical functionsSee Beam SQL mathematical functions + https://github.com/google/zetasql/blob/master/docs/mathematical_functions.md;>Mathematical functionsSee mathematical functions https://github.com/google/zetasql/blob/master/docs/navigation_functions.md;>Navigation functionsNo - https://github.com/google/zetasql/blob/master/docs/aggregate_analytic_functions.md;>Aggregate analytic functionsSee Beam SQL aggregate functions + https://github.com/google/zetasql/blob/master/docs/aggregate_analytic_functions.md;>Aggregate analytic functionsSee aggregate functions https://github.com/google/zetasql/blob/master/docs/hash_functions.md;>Hash functionsNo - https://github.com/google/zetasql/blob/master/docs/string_functions.md;>String functionsSee Beam SQL string functions + https://github.com/google/zetasql/blob/master/docs/string_functions.md;>String functionsSee string functions https://github.com/google/zetasql/blob/master/docs/json_functions.md;>JSON functionsNo https://github.com/google/zetasql/blob/master/docs/array_functions.md;>Array functionsNo https://github.com/google/zetasql/blob/master/docs/date_functions.md;>Date functionsNo @@ -348,7 +348,7 @@ limitations under the License. https://github.com/google/zetasql/blob/master/docs/security_functions.md;>Security functionsNo https://github.com/google/zetasql/blob/master/docs/net_functions.md;>Net functionsNo https://github.com/google/zetasql/blob/master/docs/operators.md;>Operator precedenceYes - Conditional expressionsSee Beam SQL conditional expressions + Conditional expressionsSee conditional expressions https://github.com/google/zetasql/blob/master/docs/expression_subqueries.md;>Expression subqueriesNo https://github.com/google/zetasql/blob/master/docs/debugging_functions.md;>Debugging functionsNo
[beam] branch asf-site updated: Publishing website 2019/10/10 21:24:41 at commit 71a4690
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a7d5e2d Publishing website 2019/10/10 21:24:41 at commit 71a4690 a7d5e2d is described below commit a7d5e2d80530eda0219b8bcf5976dcacd41824c1 Author: jenkins AuthorDate: Thu Oct 10 21:24:42 2019 + Publishing website 2019/10/10 21:24:41 at commit 71a4690 --- website/generated-content/documentation/programming-guide/index.html | 2 ++ website/generated-content/get-started/quickstart-py/index.html | 2 ++ website/generated-content/roadmap/python-sdk/index.html | 2 +- 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index d9bdfd2..c934c19 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -556,6 +556,8 @@ how to implement Beam concepts in your pipelines. +The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 (https://issues.apache.org/jira/browse/BEAM-8371;>BEAM-8371). For best results, use Beam with Python 3. + 1. Overview To use Beam, you need to first create a driver program using the classes in one diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html index 61014a9..94776a7 100644 --- a/website/generated-content/get-started/quickstart-py/index.html +++ b/website/generated-content/get-started/quickstart-py/index.html @@ -268,6 +268,8 @@ limitations under the License. Next Steps +The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 (https://issues.apache.org/jira/browse/BEAM-8371;>BEAM-8371). For best results, use Beam with Python 3. + Set up your environment Check your Python version diff --git a/website/generated-content/roadmap/python-sdk/index.html b/website/generated-content/roadmap/python-sdk/index.html index 8b8fc20..5fd11c6 100644 --- a/website/generated-content/roadmap/python-sdk/index.html +++ b/website/generated-content/roadmap/python-sdk/index.html @@ -249,7 +249,7 @@ limitations under the License. Python 3 Support -Apache Beam first offered Python 3.5 support with the 2.11.0 SDK release and added Python 3.6, Python 3.7 support with the 2.14.0 version. However, we continue to polish some https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050;>rough edges and strengthen Beam’s Python 3 offering: +Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050;>improve the experience for Python 3 users and phase out Python 2 support (https://issues.apache.org/jira/browse/BEAM-8371;>BEAM-8371): https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245view=detail;>Kanban Board
[beam] branch asf-site updated: Publishing website 2019/10/10 14:22:57 at commit 67ef076
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c0d98df Publishing website 2019/10/10 14:22:57 at commit 67ef076 c0d98df is described below commit c0d98dfd34beed06879921620607f6ec68dceb6d Author: jenkins AuthorDate: Thu Oct 10 14:22:57 2019 + Publishing website 2019/10/10 14:22:57 at commit 67ef076 --- website/generated-content/roadmap/portability/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index f1991a1..d06dbe9 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -427,7 +427,7 @@ performs work on the machine the job originated from, not the worker nodes. each worker node. DOCKER (default): User code is executed within a container started on each worker node. This requires docker to be installed on worker nodes. For more information, see -https://github.com/apache/beam/blob/master/sdks/CONTAINERS.md;>here. +here. environment_config configures the environment depending on the value of environment_type.
[beam] branch asf-site updated: Publishing website 2019/10/09 21:17:46 at commit 603d68a
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 559edfa Publishing website 2019/10/09 21:17:46 at commit 603d68a
559edfa is described below
commit 559edfac7340426b92425925b4efb53e1f68d159
Author: jenkins
AuthorDate: Wed Oct 9 21:17:46 2019 +
Publishing website 2019/10/09 21:17:46 at commit 603d68a
---
.../get-started/mobile-gaming-example/index.html | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/website/generated-content/get-started/mobile-gaming-example/index.html
b/website/generated-content/get-started/mobile-gaming-example/index.html
index ee6e589..23c0d20 100644
--- a/website/generated-content/get-started/mobile-gaming-example/index.html
+++ b/website/generated-content/get-started/mobile-gaming-example/index.html
@@ -445,7 +445,7 @@ looks more like what is depicted by the red squiggly line
above the ideal line.<
}
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the user_score
pipeline."""
parser = argparse.ArgumentParser()
@@ -466,7 +466,7 @@ looks more like what is depicted by the red squiggly line
above the ideal line.<
# We use the save_main_session option because one or more
DoFn's in this
# workflow rely on global context (e.g., a module imported
at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as
p:
def format_user_score_sums(user_score):
@@ -700,7 +700,7 @@ logical windows based on when those scores occurred in
event time.
| 'ExtractAndSumScore'
ExtractAndSumScore('team'))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score
pipeline."""
parser = argparse.ArgumentParser()
@@ -751,7 +751,7 @@ logical windows based on when those scores occurred in
event time.
# We use the save_main_session option because one or more
DoFn's in this
# workflow rely on global context (e.g., a module imported
at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as
p:
(p #
pylint: disable=expression-not-assigned
[beam] branch asf-site updated: Publishing website 2019/10/08 18:24:02 at commit 7587aa4
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new db2f788 Publishing website 2019/10/08 18:24:02 at commit 7587aa4 db2f788 is described below commit db2f788362b9496560faa161163366df804f645a Author: jenkins AuthorDate: Tue Oct 8 18:24:03 2019 + Publishing website 2019/10/08 18:24:02 at commit 7587aa4 --- .../sdks/python-dependencies/index.html| 42 ++ 1 file changed, 42 insertions(+) diff --git a/website/generated-content/documentation/sdks/python-dependencies/index.html b/website/generated-content/documentation/sdks/python-dependencies/index.html index 03bb339..b7a1681 100644 --- a/website/generated-content/documentation/sdks/python-dependencies/index.html +++ b/website/generated-content/documentation/sdks/python-dependencies/index.html @@ -290,6 +290,48 @@ the listed versions that will be in scope during execution. To see the compile and runtime dependencies for your Beam SDK version, expand the relevant section below. +2.16.0 + +Beam SDK for Python 2.16.0 has the following compile and + runtime dependencies. + + + PackageVersion + avro-python3=1.8.1,2.0.0; python_version = "3.0" + avro=1.8.1,2.0.0; python_version "3.0" + cachetools=3.1.0,4 + crcmod=1.7,2.0 + dill=0.3.0,0.3.1 + fastavro=0.21.4,0.22 + funcsigs=1.0.2,2; python_version "3.0" + future=0.16.0,1.0.0 + futures=3.2.0,4.0.0; python_version "3.0" + google-apitools=0.5.28,0.5.29 + google-cloud-bigquery=1.6.0,1.18.0 + google-cloud-bigtable=0.31.1,1.1.0 + google-cloud-core=0.28.1,2 + google-cloud-datastore=1.7.1,1.8.0 + google-cloud-pubsub=0.39.0,1.1.0 + googledatastore=7.0.1,7.1; python_version "3.0" + grpcio=1.12.1,2 + hdfs=2.1.0,3.0.0 + httplib2=0.8,=0.12.0 + mock=1.0.1,3.0.0 + oauth2client=2.0.1,4 + proto-google-cloud-datastore-v1=0.90.0,=0.90.4; python_version "3.0" + protobuf=3.5.0.post1,4 + pyarrow=0.11.1,0.15.0; python_version = "3.0" or platform_system != "Windows" + pydot=1.2.0,2 + pymongo=3.8.0,4.0.0 + python-dateutil=2.8.0,3 + pytz=2018.3 + pyvcf=0.6.8,0.7.0; python_version "3.0" + pyyaml=3.12,4.0.0 + typing=3.6.0,3.7.0; python_version "3.5.0" + + + + 2.15.0 Beam SDK for Python 2.15.0 has the following compile and
[beam] branch asf-site updated: Publishing website 2019/10/08 00:38:00 at commit 7327e45
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8861f6a Publishing website 2019/10/08 00:38:00 at commit 7327e45 8861f6a is described below commit 8861f6a3c52d5e3936480af9fbfa3b941554c412 Author: jenkins AuthorDate: Tue Oct 8 00:38:00 2019 + Publishing website 2019/10/08 00:38:00 at commit 7327e45 --- .../2019/10/07/beam-2.16.0.html} | 311 - website/generated-content/blog/index.html | 32 +++ website/generated-content/feed.xml | 187 +++-- website/generated-content/index.html | 10 +- 4 files changed, 254 insertions(+), 286 deletions(-) diff --git a/website/generated-content/index.html b/website/generated-content/blog/2019/10/07/beam-2.16.0.html similarity index 57% copy from website/generated-content/index.html copy to website/generated-content/blog/2019/10/07/beam-2.16.0.html index 394af82..28cc965 100644 --- a/website/generated-content/index.html +++ b/website/generated-content/blog/2019/10/07/beam-2.16.0.html @@ -28,9 +28,8 @@ - Apache Beam - + Apache Beam 2.16.0 + https://fonts.googleapis.com/css?family=Roboto:100,300,400; rel="stylesheet"> https://code.jquery.com/jquery-2.2.4.min.js"</a>;> @@ -39,7 +38,7 @@ - https://beam.apache.org/; data-proofer-ignore> + https://beam.apache.org/blog/2019/10/07/beam-2.16.0.html; data-proofer-ignore> https://beam.apache.org/feed.xml;> https://use.fontawesome.com/releases/v5.4.1/css/all.css; integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous"> @@ -53,7 +52,7 @@ - + - https://github.com/apache/beam/edit/master/website/src/index.md; data-proofer-ignore> + https://github.com/apache/beam/edit/master/website/src/_posts/2019-10-07-beam-2.16.0.md; data-proofer-ignore> @@ -151,6 +150,31 @@ + + + +http://schema.org/BlogPosting;> + + +Apache Beam 2.16.0 +Oct 7, 2019 • + Mark Liu + + + + + + - - - - - - -Apache Beam: An advanced unified programming model - - -Implement batch and streaming data processing jobs that run on any execution engine. - - -Learn more -Try Beam -Download Beam SDK 2.16.0 - - -Java Quickstart -Python Quickstart -Go Quickstart - - - - - - -The latest from the blog - - - - - Google Summer of Code '19 - Sep 4, 2019 - - - - Apache Beam 2.15.0 - Aug 22, 2019 - - - - Apache Beam 2.14.0 - Jul 31, 2019 - - - - - - - - - - -All about Apache Beam - - - - - -Unified - - -Use a single programming model for both batch and streaming use cases. - - - - - -Portable - - -Execute pipelines on multiple execution environments. - - - - - -Extensible - - -Write and share new SDKs, IO connectors, and transformation libraries. - - - - - +We are happy to present the new 2.16.0 release of Beam. This release includes both improvements and new functionality. +See the download page for this release. +For more information on changes in 2.16.0, check out the +https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345494;>detailed release notes. - - - - - +Highlights - - -Works with - - - - - http://apex.apache.org;> - - - - http://flink.apache.org;> - - - - http://spark.apache.org/;> - - - - https://cloud.google.com/dataflow/;> - - - - http://gearpump.apache.org/;> - - - - http://samza.apache.org/;> - - - - + + Customizable Docker container images released and supported by Beam portable runners on Python 2.7, 3.5, 3.6, 3.7. (https://issues.apache.org/jira/browse/BEAM-7907;>BEAM-7907) + Integration improvements for Python Streaming on Dataflow including service features like autoscaling, drain, update, streaming engine and counter updates. + - - - - Testimonials - - - - - -
[beam] branch asf-site updated: Publishing website 2019/10/07 22:57:23 at commit 6a3c008
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new b6bb83a Publishing website 2019/10/07 22:57:23 at commit 6a3c008 b6bb83a is described below commit b6bb83abfd8d6b6183e172f341a032c6a2257a81 Author: jenkins AuthorDate: Mon Oct 7 22:57:23 2019 + Publishing website 2019/10/07 22:57:23 at commit 6a3c008 --- website/generated-content/get-started/downloads/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html index dbd5660..6c4a5d3 100644 --- a/website/generated-content/get-started/downloads/index.html +++ b/website/generated-content/get-started/downloads/index.html @@ -216,7 +216,7 @@ API stability Releases - 2.16.0 (2019-10-01) + 2.16.0 (2019-10-07) 2.15.0 (2019-08-22) 2.14.0 (2019-08-01) @@ -331,7 +331,7 @@ versions denoted 0.x.y. Releases -2.16.0 (2019-10-01) +2.16.0 (2019-10-07) Official http://www.apache.org/dyn/closer.cgi/beam/2.16.0/apache-beam-2.16.0-source-release.zip;>source code download. https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.sha512;>SHA-512. https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.asc;>signature.
[beam] branch asf-site updated: Publishing website 2019/10/07 22:19:40 at commit 4733374
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new b92b8c0 Publishing website 2019/10/07 22:19:40 at commit 4733374 b92b8c0 is described below commit b92b8c01f2379dcd908fff4323f4e870509de8c7 Author: jenkins AuthorDate: Mon Oct 7 22:19:40 2019 + Publishing website 2019/10/07 22:19:40 at commit 4733374 --- website/generated-content/.htaccess| 2 +- .../blog/2016/10/20/test-stream.html | 2 +- .../blog/2017/08/16/splittable-do-fn.html | 10 +- .../08/20/review-input-streaming-connectors.html | 22 +++--- .../contribute/ptransform-style-guide/index.html | 4 ++-- .../sql/calcite/aggregate-functions/index.html | 4 ++-- .../dsls/sql/calcite/data-types/index.html | 4 ++-- .../dsls/sql/calcite/lexical/index.html| 4 ++-- .../dsls/sql/calcite/overview/index.html | 4 ++-- .../dsls/sql/calcite/query-syntax/index.html | 4 ++-- .../dsls/sql/calcite/scalar-functions/index.html | 4 ++-- .../extensions/create-external-table/index.html| 4 ++-- .../dsls/sql/extensions/joins/index.html | 4 ++-- .../dsls/sql/extensions/set/index.html | 4 ++-- .../extensions/user-defined-functions/index.html | 4 ++-- .../extensions/windowing-and-triggering/index.html | 4 ++-- .../documentation/dsls/sql/overview/index.html | 8 .../documentation/dsls/sql/shell/index.html| 4 ++-- .../documentation/dsls/sql/walkthrough/index.html | 10 +- .../io/built-in/google-bigquery/index.html | 14 +++--- .../documentation/io/built-in/parquet/index.html | 6 +++--- .../io/developing-io-python/index.html | 6 +++--- .../pipelines/test-your-pipeline/index.html| 4 ++-- .../documentation/programming-guide/index.html | 16 .../documentation/runners/dataflow/index.html | 6 +++--- .../documentation/runners/direct/index.html| 14 +++--- .../documentation/runners/flink/index.html | 4 ++-- .../documentation/runners/gearpump/index.html | 4 ++-- .../documentation/runners/jstorm/index.html| 2 +- .../documentation/runners/mapreduce/index.html | 2 +- .../documentation/runners/samza/index.html | 2 +- .../documentation/runners/spark/index.html | 2 +- .../sdks/feature-comparison/index.html | 4 ++-- .../documentation/sdks/go/index.html | 4 ++-- .../sdks/java-dependencies/index.html | 4 ++-- .../documentation/sdks/java-extensions/index.html | 4 ++-- .../documentation/sdks/java-thirdparty/index.html | 4 ++-- .../documentation/sdks/java/euphoria/index.html| 6 +++--- .../documentation/sdks/java/index.html | 4 ++-- .../sdks/java/testing/nexmark/index.html | 8 .../sdks/python-dependencies/index.html| 4 ++-- .../sdks/python-pipeline-dependencies/index.html | 4 ++-- .../documentation/sdks/python-streaming/index.html | 4 ++-- .../sdks/python-type-safety/index.html | 4 ++-- .../documentation/sdks/python/index.html | 4 ++-- .../get-started/downloads/index.html | 18 +- .../get-started/quickstart-java/index.html | 4 ++-- .../get-started/wordcount-example/index.html | 4 ++-- website/generated-content/index.html | 4 ++-- 49 files changed, 142 insertions(+), 134 deletions(-) diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess index f3bf7b7..20d4586 100644 --- a/website/generated-content/.htaccess +++ b/website/generated-content/.htaccess @@ -21,4 +21,4 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301] # The following redirect maintains the previously supported URLs. RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2; # Keep this updated to point to the current release. -RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.15.0$2; +RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.16.0$2; diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html index b8f0e03..0ef0abc 100644 --- a/website/generated-content/blog/2016/10/20/test-stream.html +++ b/website/generated-content/blog/2016/10/20/test-stream.html @@ -240,7 +240,7 @@ be controlled within a test. Writing Deterministic Tests to Emulate Nondeterminism The Beam testing infrastructure provides the -https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/testing/PAssert.html;>PAssert +htt
[beam] branch asf-site updated: Publishing website 2019/10/04 17:11:14 at commit 8e75935
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a568015 Publishing website 2019/10/04 17:11:14 at commit 8e75935 a568015 is described below commit a568015f315dbf3423ac96a68c04f8ba63817c7f Author: jenkins AuthorDate: Fri Oct 4 17:11:15 2019 + Publishing website 2019/10/04 17:11:14 at commit 8e75935 --- .../python/elementwise/filter/index.html | 52 --- .../python/elementwise/flatmap/index.html | 76 +- 2 files changed, 30 insertions(+), 98 deletions(-) diff --git a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html index 3f8bd3f..3b5eb48 100644 --- a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html +++ b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html @@ -513,13 +513,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>https://beam.apache.org/images/logos/sdks/python.png; width="32px" height="32px" alt="Pydoc" /> Pydoc - @@ -597,13 +595,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - - + https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png; width="32px" height="32px" alt="Run code now" /> Run code now - - - + https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px" height="32px" alt="View source code" /> View source code - @@ -690,13 +684,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - - + https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png; width="32px" height="32px" alt="Run code now" /> Run code now - - - + https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px" height="32px" alt="View source code" /> View source code - @@ -788,13 +778,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - - + https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png; width="32px" height="32px" alt="Run code now" /> Run code now - - - + https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px" height="32px" alt="View source code" /> View source code - @@ -889,13 +875,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - - + https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png; width="32px" height="32px" alt="Run code now" /> Run code now - - - + https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px" height="32px" alt="View source code" /> View source code - @@ -993,13 +975,11 @@ See the License for the specific language governing permissions and limitations under the License. --> - - + https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentatio
[beam] branch asf-site updated: Publishing website 2019/10/04 16:46:34 at commit 37e8926
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8b73df7 Publishing website 2019/10/04 16:46:34 at commit 37e8926 8b73df7 is described below commit 8b73df77765e8e735ff311eca177c38577f114fc Author: jenkins AuthorDate: Fri Oct 4 16:46:34 2019 + Publishing website 2019/10/04 16:46:34 at commit 37e8926 --- .../documentation/programming-guide/index.html | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index f775a5e..cee3367 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -3139,14 +3139,14 @@ windows. The simplest form of windowing is using fixed time windows: given a timestamped PCollection which might be continuously updating, each window -might capture (for example) all elements with timestamps that fall into a five -minute interval. +might capture (for example) all elements with timestamps that fall into a 30 +second interval. A fixed time window represents a consistent duration, non overlapping time -interval in the data stream. Consider windows with a five-minute duration: all +interval in the data stream. Consider windows with a 30 second duration: all of the elements in your unbounded PCollection with timestamp values from -0:00:00 up to (but not including) 0:05:00 belong to the first window, elements -with timestamp values from 0:05:00 up to (but not including) 0:10:00 belong to +0:00:00 up to (but not including) 0:00:30 belong to the first window, elements +with timestamp values from 0:00:30 up to (but not including) 0:01:00 belong to the second window, and so on. @@ -3157,15 +3157,15 @@ the second window, and so on. A sliding time window also represents time intervals in the data stream; however, sliding time windows can overlap. For example, each window might -capture five minutes worth of data, but a new window starts every ten seconds. +capture 60 seconds worth of data, but a new window starts every 30 seconds. The frequency with which sliding windows begin is called the period. -Therefore, our example would have a window duration of five minutes and a -period of ten seconds. +Therefore, our example would have a window duration of 60 seconds and a +period of 30 seconds. Because multiple windows overlap, most elements in a data set will belong to more than one window. This kind of windowing is useful for taking running averages of data; using sliding time windows, you can compute a running average -of the past five minutes’ worth of data, updated every ten seconds, in our +of the past 60 seconds’ worth of data, updated every 30 seconds, in our example.
[beam] branch asf-site updated: Publishing website 2019/10/04 16:32:02 at commit 63344d5
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 57761e0 Publishing website 2019/10/04 16:32:02 at commit 63344d5 57761e0 is described below commit 57761e0132de153f63c4ba1794bfae0f85dedf7c Author: jenkins AuthorDate: Fri Oct 4 16:32:03 2019 + Publishing website 2019/10/04 16:32:02 at commit 63344d5 --- .../documentation/runners/capability-matrix/index.html| 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/runners/capability-matrix/index.html b/website/generated-content/documentation/runners/capability-matrix/index.html index 36a7f80..bfac9427 100644 --- a/website/generated-content/documentation/runners/capability-matrix/index.html +++ b/website/generated-content/documentation/runners/capability-matrix/index.html @@ -2108,7 +2108,7 @@ limitations under the License. - - +~ @@ -8962,7 +8962,7 @@ limitations under the License. - -YesGauge metrics are not supported. All other metric types are supported. +PartiallyGauge metrics are not supported. All other metric types are supported.
[beam] branch asf-site updated: Publishing website 2019/10/04 16:29:56 at commit daa36e7
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 7d29d39 Publishing website 2019/10/04 16:29:56 at commit daa36e7 7d29d39 is described below commit 7d29d3964bcac7cd332e68b12a47b1f3cc7a6346 Author: jenkins AuthorDate: Fri Oct 4 16:29:57 2019 + Publishing website 2019/10/04 16:29:56 at commit daa36e7 --- website/generated-content/roadmap/portability/index.html | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index 5ec2aed..f1991a1 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -396,6 +396,8 @@ See the https://s.apache.org/apache-beam-portability-support-table;>Portability support table for details. +Prerequisites: https://docs.docker.com/compose/install/;>Docker, https://docs.python-guide.org/starting/install3/linux/;>Python, https://openjdk.java.net/install/;>Java 8 + Running Python wordcount on Flink The Beam Flink runner can run Python pipelines in batch and streaming modes.
[beam] branch asf-site updated: Publishing website 2019/09/26 01:00:43 at commit 7120ceb
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c8cabba Publishing website 2019/09/26 01:00:43 at commit 7120ceb
c8cabba is described below
commit c8cabbadd2569b905714a18b30438eb50f519120
Author: jenkins
AuthorDate: Thu Sep 26 01:00:43 2019 +
Publishing website 2019/09/26 01:00:43 at commit 7120ceb
---
.../python/elementwise/filter/index.html | 427 ---
.../python/elementwise/flatmap/index.html | 606 ++---
2 files changed, 881 insertions(+), 152 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index 99ce805..3f8bd3f 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -485,14 +485,42 @@ limitations under the License.
localStorage.setItem('language', 'language-py')
+
+
+
+
+
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>
- https://beam.apache.org/images/logos/sdks/python.png;
width="32px" height="32px" alt="Pydoc" />
- Pydoc
-
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>https://beam.apache.org/images/logos/sdks/python.png; width="32px"
height="32px" alt="Pydoc" /> Pydoc
+
+
Given a predicate, filter out all elements that don’t satisfy that
predicate.
@@ -541,23 +569,64 @@ Then, we apply Filter in multiple ways to
-
+
+
+
+
+
+
-https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
- https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run in Colab" />
- Run code now
-
+https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run code now" /> Run code now
+
+
+
+
+
-https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="32px" height="32px" alt="View source code" />
- View source code
-
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px"
height="32px" alt="View source code" /> View source code
+
+
Example 2: Filtering with
a lambda function
@@ -593,23 +662,64 @@ Then, we apply Filter in multiple ways to
-
+
+
+
+
+
+
-https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
- https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run code now" />
- Run code now
-
+https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run code now" /> Run code now
+
+
+
+
-https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="32px" height="32px" alt="View source code" />
- View source code
-
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>https://www.tensorflow.org/images/GitHub-Mark-32px.png; width="32px"
height="32px" alt="View source code" /> View source code
+
+
Example 3: Filtering with
multiple arguments
@@ -650,23 +760,64 @@ They are passed as additional positional arguments or
keyword arguments to the f
-
+
+
+
+
+
+
-https://colab.research.google.com/github/apache/beam/blob/master/examples/note
[beam] branch asf-site updated: Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 345c0dd Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
345c0dd is described below
commit 345c0ddcb93eae5002b0e616b576c9a96c8a0ca4
Author: jenkins
AuthorDate: Tue Sep 24 09:29:33 2019 +
Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
---
.../contribute/release-guide/index.html| 2 +-
.../documentation/dsls/sql/shell/index.html| 6 ++---
.../documentation/runners/flink/index.html | 29 +++---
.../sdks/java/testing/nexmark/index.html | 6 ++---
4 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 40403b8..8f4a0fa 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1601,7 +1601,7 @@ Thanks everyone!
Flink Local Runner
-./gradlew
:runners:flink:1.5:runQuickstartJavaFlinkLocal \
+./gradlew
:runners:flink:1.8:runQuickstartJavaFlinkLocal \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY}
\
-Pver=${RELEASE_VERSION}
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html
b/website/generated-content/documentation/dsls/sql/shell/index.html
index ec1fb7a..bed0830 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -301,7 +301,7 @@ limitations under the License.
To use Beam SQL shell, you must first clone the https://github.com/apache/beam;>Beam SDK repository. Then, from the
root of the repository clone, execute the following commands to run the
shell:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
@@ -393,7 +393,7 @@ No rows affected (0.042 seconds)
Make sure the SQL shell includes the desired runner. Add the
corresponding project id to the -Pbeam.sql.shell.bundled parameter of the
Gradle invocation (https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle;>source
code, https://github.com/apache/beam/blob/master/settings.gradle;>project
ids). For example, use the following command to include Flink runner and
KafkaIO:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
@@ -422,7 +422,7 @@ No rows affected (0.042 seconds)
You can also build your own standalone package for SQL shell using distZip or distTar tasks. For example:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' distZip
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' distZip
ls ./sdks/java/extensions/sql/shell/build/distributions/
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
diff --git a/website/generated-content/documentation/runners/flink/index.html
b/website/generated-content/documentation/runners/flink/index.html
index b3ad611..921907c 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -319,12 +319,33 @@ period.
Artifact Id
- =2.13.0
+ 2.17.0
1.8.x
beam-runners-flink-1.8
- =2.10.0
+ 1.7.x
+ beam-runners-flink-1.7
+
+
+ 2.13.0 - 2.16.0
+ 1.8.x
+ beam-runners-flink-1.8
+
+
+ 1.7.x
+ beam-runners-flink-1.7
+
+
+ 1.6.x
+ beam-runners-flink-1.6
+
+
+ 1.5.x
+ beam-runners-flink_2.11
+
+
+ 2.10.0 - 2.16.0
1.7.x
beam-runners-flink-1.7
@@ -464,7 +485,7 @@ download it on the Downloads page. In the
available. To run a pipeline on an embedded Flink cluster:
-1. Start the JobService endpoint: ./gradlew
:runners:flink:1.5:job-server:runShadow
+1. Start the JobService endpoint: ./gradlew
:runners:flink:1.8:job-server:runShadow
@@ -497,7 +518,7 @@ To run on a separate https://ci.apache.org/projects/flink/flink-docs-re
1. Start a Flink cluster which exposes the Rest
interface on localhost:8081 by default.
-2. Start JobService with Flink Rest endpoint:
./gradlew
:runners:flink:1.5:job-server:runShadow -Pflink
[beam] branch asf-site updated: Publishing website 2019/09/24 00:16:54 at commit 113461a
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e18160c Publishing website 2019/09/24 00:16:54 at commit 113461a
e18160c is described below
commit e18160cfccadba882c97e3277d87f2547df4862a
Author: jenkins
AuthorDate: Tue Sep 24 00:16:54 2019 +
Publishing website 2019/09/24 00:16:54 at commit 113461a
---
.../documentation/runners/direct/index.html| 76 ++
.../documentation/sdks/python-streaming/index.html | 7 +-
2 files changed, 77 insertions(+), 6 deletions(-)
diff --git a/website/generated-content/documentation/runners/direct/index.html
b/website/generated-content/documentation/runners/direct/index.html
index 994d5df..8187175 100644
--- a/website/generated-content/documentation/runners/direct/index.html
+++ b/website/generated-content/documentation/runners/direct/index.html
@@ -208,6 +208,7 @@
Memory considerations
Streaming execution
+ Execution Mode
@@ -295,6 +296,81 @@ interface for defaults and additional pipeline
configuration options.
If your pipeline uses an unbounded data source or sink, you must set the
streaming option to true.
+Execution Mode
+
+Python https://beam.apache.org/contribute/runner-guide/#the-fn-api;>FnApiRunner
supports multi-threading and multi-processing mode.
+
+Setting parallelism
+
+Number of threads or subprocesses is defined by setting the direct_num_workers option. There are several
ways to set this option.
+
+
+ Passing through CLI when executing a pipeline.
+python
wordcount.py --input xx --output xx --direct_num_workers 2
+
+
+
+ Setting with PipelineOptions.
+from
apache_beam.options.pipeline_options import PipelineOptions
+pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+
+
+
+ Adding to existing PipelineOptions.
+from
apache_beam.options.pipeline_options import DirectOptions
+pipeline_options = PipelineOptions(xxx)
+pipeline_options.view_as(DirectOptions).direct_num_workers = 2
+
+
+
+
+
+Running with multi-threading
mode
+
+import argparse
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC)))
+
+
+
+Running with multi-processing
mode
+
+import argparse
+import sys
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.SUBPROCESS_SDK,
+ payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
+% sys.executable.encode('ascii'
+
+
diff --git
a/website/generated-content/documentation/sdks/python-streaming/index.html
b/website/generated-content/documentation/sdks/python-streaming/index.html
index 4531074..342cd83 100644
--- a/website/generated-content/documentation/sdks/python-streaming/index.html
+++ b/website/generated-content/documentation/sdks/python-streaming/index.html
@@ -451,7 +451,7 @@ about executing streaming pipelines:
Custom source API
Splittable DoFn API
Handling of late data
- User-defined custom WindowFn
+ User-defined custom merging WindowFn (with fnapi)
DataflowRunner specific features
@@ -460,12 +460,7 @@ about executing streaming pipelines:
Dataflow specific features with Python streaming execution.
- Streaming autoscaling
- Updating existing pipelines
Cloud Dataflow Templates
- Some monitoring features, such as msec counters, display data, metrics,
and
-element counts for transforms. However, logging, watermarks, and element
-counts for sources are supported.
[beam] branch asf-site updated: Publishing website 2019/09/23 12:12:49 at commit cb30e25
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 785f59d Publishing website 2019/09/23 12:12:49 at commit cb30e25 785f59d is described below commit 785f59d631eabd7ac113f9c3aab79a8770c8e1f9 Author: jenkins AuthorDate: Mon Sep 23 12:12:50 2019 + Publishing website 2019/09/23 12:12:49 at commit cb30e25 --- .../documentation/runners/flink/index.html | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 886409c..b3ad611 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -256,8 +256,8 @@ which is useful for testing pipelines. It is important to understand that the Flink Runner comes in two flavors: - A legacy Runner which supports only Java (and other JVM-based languages) - A portable Runner which supports Java/Python/Go + The original classic Runner which supports only Java (and other JVM-based languages) + The newer portable Runner which supports Java/Python/Go You may ask why there are two Runners? @@ -267,8 +267,8 @@ which is useful for testing pipelines. architecture of the Runners had to be changed significantly to support executing pipelines written in other languages. -If your applications only use Java, then you should currently go with the legacy -Runner. Eventually, the portable Runner will replace the legacy Runner because +If your applications only use Java, then you should currently go with the classic +Runner. Eventually, the portable Runner will replace the classic Runner because it contains the generalized framework for executing Java, Python, Go, and more languages in the future. @@ -276,14 +276,14 @@ languages in the future. portable Runner. For more information on portability, please visit the Portability page. -Consequently, this guide is split into two parts to document the legacy and +Consequently, this guide is split into two parts to document the classic and the portable functionality of the Flink Runner. Please use the switcher below to select the appropriate Runner: Adapt for: -Legacy (Java) +Classic (Java) Portable (Java/Python/Go) @@ -820,7 +820,7 @@ Many sources like PubSubIO rely on their Capability The Beam Capability Matrix documents the -capabilities of the legacy Flink Runner. +capabilities of the classic Flink Runner. The https://s.apache.org/apache-beam-portability-support-table;>Portable Capability Matrix documents
[beam] branch asf-site updated: Publishing website 2019/09/20 15:10:43 at commit bb6f9ed
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new f50c31f Publishing website 2019/09/20 15:10:43 at commit bb6f9ed f50c31f is described below commit f50c31fe1bca4949f5f3edd70ff080dc54ddec09 Author: jenkins AuthorDate: Fri Sep 20 15:10:43 2019 + Publishing website 2019/09/20 15:10:43 at commit bb6f9ed --- .../documentation/io/testing/index.html| 368 ++--- 1 file changed, 23 insertions(+), 345 deletions(-) diff --git a/website/generated-content/documentation/io/testing/index.html b/website/generated-content/documentation/io/testing/index.html index d840b62..acce2e8 100644 --- a/website/generated-content/documentation/io/testing/index.html +++ b/website/generated-content/documentation/io/testing/index.html @@ -466,9 +466,11 @@ Goals Integration tests, data stores, and Kubernetes - Running integration tests + Running integration tests on your machine + Running Integration Tests on Pull Requests Performance testing dashboard Implementing Integration Tests + Small Scale and Large Scale Integration Tests @@ -623,120 +625,23 @@ limitations under the License. However, when working locally, there is no requirement to use Kubernetes. All of the test infrastructure allows you to pass in connection info, so developers can use their preferred hosting infrastructure for local development. -Running integration tests +Running integration tests on your machine -The high level steps for running an integration test are: +You can always run the IO integration tests on your own machine. The high level steps for running an integration test are: Set up the data store corresponding to the test being run. Run the test, passing it connection info from the just created data store. Clean up the data store. -Since setting up data stores and running the tests involves a number of steps, and we wish to time these tests when running performance benchmarks, we use PerfKit Benchmarker to manage the process end to end. With a single command, you can go from an empty Kubernetes cluster to a running integration test. - -However, PerfKit Benchmarker is not required for running integration tests. Therefore, we have listed the steps for both using PerfKit Benchmarker, and manually running the tests below. - -Using PerfKit Benchmarker - -Prerequisites: - - https://github.com/GoogleCloudPlatform/PerfKitBenchmarker;>Install PerfKit Benchmarker - Have a running Kubernetes cluster you can connect to locally using kubectl. We recommend using Google Kubernetes Engine - it’s proven working for all the use cases we tested. - - -You won’t need to invoke PerfKit Benchmarker directly. Run ./gradlew performanceTest task in project’s root directory, passing kubernetes scripts of your choice (located in .test_infra/kubernetes directory). It will setup PerfKitBenchmarker for you. - -Example run with the Direct runner: -./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTest [...] - - - -Example run with the Google Cloud Dataflow runner: -./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/. [...] - - - -Example run with the HDFS filesystem and Cloud Dataflow runner: - -./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=10", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/file-based-io-tests/ -DintegrationTest=org.apache.beam.sdk.io.text.TextIOIT -DkubernetesScripts=".test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml,.test-infra/kubernetes [...] - - - -NOTE: When using Direct runner along with HDFS cluster, please set export HADOOP_USER_NAME=root before runnning performanceTest task. - -Parameter descriptions: - - - - -
[beam] branch asf-site updated: Publishing website 2019/09/20 01:10:50 at commit ec3e770
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 4760ec3 Publishing website 2019/09/20 01:10:50 at commit ec3e770 4760ec3 is described below commit 4760ec39e2bf818a8cac94126076e0057f13eff4 Author: jenkins AuthorDate: Fri Sep 20 01:10:50 2019 + Publishing website 2019/09/20 01:10:50 at commit ec3e770 --- website/generated-content/contribute/release-guide/index.html | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html index 9385dad..40403b8 100644 --- a/website/generated-content/contribute/release-guide/index.html +++ b/website/generated-content/contribute/release-guide/index.html @@ -996,6 +996,8 @@ to find environment issues this way without having to wait until the full build The list of release-blocking issues is available at the https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel;>version status page. Triage each unresolved issue with one of the following resolutions: +The release manager should triage what does and does not block a release. An issue should not block the release if the problem exists in the current released version or is a bug in new functionality that does not exist in the current released version. It should be a blocker if the bug is a regression between the currently released version and the release in progress and has no easy workaround. + For all JIRA issues:
[beam] branch asf-site updated: Publishing website 2019/09/19 23:24:57 at commit 6a81cdc
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new e783406 Publishing website 2019/09/19 23:24:57 at commit 6a81cdc e783406 is described below commit e783406ddab6743cd7c060251032248e325b15b0 Author: jenkins AuthorDate: Thu Sep 19 23:24:57 2019 + Publishing website 2019/09/19 23:24:57 at commit 6a81cdc --- .../documentation/runners/flink/index.html | 24 ++ .../documentation/runners/spark/index.html | 25 ++ .../roadmap/portability/index.html | 53 ++ 3 files changed, 65 insertions(+), 37 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 4363dd1..886409c 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -461,13 +461,10 @@ If you have a Flink JobManager running on As of now you will need a copy of Apache Beam’s source code. You can download it on the Downloads page. In the future there will be pre-built Docker images -available. +available. To run a pipeline on an embedded Flink cluster: -1. Only required once: Build the SDK harness container (optionally replace py35 with the Python version of your choice): ./gradlew :sdks:python:container:py35:docker - - -2. Start the JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow +1. Start the JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow @@ -477,13 +474,17 @@ To execute the job on a Flink cluster, the Beam JobService needs to be provided with the Flink JobManager address. -3. Submit the Python pipeline to the above endpoint by using the PortableRunner and job_endpoint set to localhost:8099 (this is the default address of the JobService). For example: +2. Submit the Python pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService), and environment_type set to LOOPBACK. For example: import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions -options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"]) +options = PipelineOptions([ +"--runner=PortableRunner", +"--job_endpoint=localhost:8099", +"--environment_type=LOOPBACK" +]) with beam.Pipeline(options) as p: ... @@ -500,6 +501,8 @@ To run on a separate https://ci.apache.org/projects/flink/flink-docs-re 3. Submit the pipeline as above. +Note however that environment_type=LOOPBACK is only intended for local testing. +See here for details. As of Beam 2.15.0, steps 2 and 3 can be automated in Python by using the FlinkRunner, @@ -509,7 +512,12 @@ plus the optional flink_version and import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions -options = PipelineOptions(["--runner=FlinkRunner", "--flink_version=1.8", "--flink_master_url=localhost:8081"]) +options = PipelineOptions([ +"--runner=FlinkRunner", +"--flink_version=1.8", +"--flink_master_url=localhost:8081", +"--environment_type=LOOPBACK" +]) with beam.Pipeline(options) as p: ... diff --git a/website/generated-content/documentation/runners/spark/index.html b/website/generated-content/documentation/runners/spark/index.html index 9171417..f90ea91 100644 --- a/website/generated-content/documentation/runners/spark/index.html +++ b/website/generated-content/documentation/runners/spark/index.html @@ -382,10 +382,7 @@ download it on the Downloads page. In the available. -1. Only required once: Build the SDK harness container (optionally replace py35 with the Python version of your choice): ./gradlew :sdks:python:container:py35:docker - - -2. Start the JobService endpoint: ./gradlew :runners:spark:job-server:runShadow +1. Start the JobService endpoint: ./gradlew :runners:spark:job-server:runShadow @@ -395,16 +392,19 @@ job. To execute the job on a Spark cluster, the Beam JobService needs to be provided with the Spark master address. -3. Submit the Python pipeline to the above endpoint by using the PortableRunner and job_endpoint set to localhost:8099 (this is the default address of the JobService). For example: +2. Submit the Python pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService), and environment_type set to LOOPBACK. For example: import apache_beam as beam from apache_beam.options.pipeline_options import Pipel
[beam] branch asf-site updated: Publishing website 2019/09/19 20:57:32 at commit 2adc2d1
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 63968ee Publishing website 2019/09/19 20:57:32 at commit 2adc2d1
63968ee is described below
commit 63968eec9c816cc1ae5be9df4473590d9fbd561d
Author: jenkins
AuthorDate: Thu Sep 19 20:57:32 2019 +
Publishing website 2019/09/19 20:57:32 at commit 2adc2d1
---
.../python/elementwise/filter/index.html | 72
.../python/elementwise/flatmap/index.html | 199 +++--
2 files changed, 176 insertions(+), 95 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index ad7d59e..00d256c 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -484,15 +484,15 @@ limitations under the License.
localStorage.setItem('language', 'language-py')
-
+
https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>
- https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
+ https://beam.apache.org/images/logos/sdks/python.png;
width="32px" height="32px" alt="Pydoc" />
Pydoc
-
+
Given a predicate, filter out all elements that don’t satisfy that
predicate.
May also be used to filter based on an inequality with a given value based
@@ -540,24 +540,24 @@ Then, we apply Filter in multiple ways to
-
+
https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
- https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run in Colab" />
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run in Colab" />
Run code now
-
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View source code" />
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="32px" height="32px" alt="View source code" />
View source code
-
+
Example 2: Filtering with
a lambda function
@@ -592,24 +592,24 @@ Then, we apply Filter in multiple ways to
-
+
https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
- https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run code now" />
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run code now" />
Run code now
-
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View source code" />
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="32px" height="32px" alt="View source code" />
View source code
-
+
Example 3: Filtering with
multiple arguments
@@ -649,24 +649,24 @@ They are passed as additional positional arguments or
keyword arguments to the f
-
+
https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
- https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run in Colab" />
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="32px" height="32px" alt="Run in Colab" />
Run code now
-
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View so
[beam] branch asf-site updated: Publishing website 2019/09/19 17:18:12 at commit 3d6afca
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c218ab3 Publishing website 2019/09/19 17:18:12 at commit 3d6afca c218ab3 is described below commit c218ab31ff69d98025ae3c53f00253f4f372949f Author: jenkins AuthorDate: Thu Sep 19 17:18:13 2019 + Publishing website 2019/09/19 17:18:12 at commit 3d6afca --- website/generated-content/documentation/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/index.html b/website/generated-content/documentation/index.html index b542313..2dde771 100644 --- a/website/generated-content/documentation/index.html +++ b/website/generated-content/documentation/index.html @@ -485,7 +485,7 @@ limitations under the License. Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners. - Read the Programming Guide which introduces all the key Beam concepts. + Read the Programming Guide, which introduces all the key Beam concepts. Learn about Beam’s execution model to better understand how pipelines execute. Visit Learning Resources for some of our favorite articles and talks about Beam.
[beam] branch asf-site updated: Publishing website 2019/09/19 17:06:43 at commit eb76a71
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 275bf63 Publishing website 2019/09/19 17:06:43 at commit eb76a71 275bf63 is described below commit 275bf63f73970724a7a0e5c32f1fe84cf932a906 Author: jenkins AuthorDate: Thu Sep 19 17:06:44 2019 + Publishing website 2019/09/19 17:06:43 at commit eb76a71 --- .../documentation/runners/capability-matrix/index.html| 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/runners/capability-matrix/index.html b/website/generated-content/documentation/runners/capability-matrix/index.html index 449ecca..36a7f80 100644 --- a/website/generated-content/documentation/runners/capability-matrix/index.html +++ b/website/generated-content/documentation/runners/capability-matrix/index.html @@ -2108,7 +2108,7 @@ limitations under the License. - -~ + @@ -8962,7 +8962,7 @@ limitations under the License. - -Partially: In batch mode, Dataflow supports committed and attempted Counters and Distributions.Gauge metrics are not supported in batch mode. Metrics are not yet supported at all in streaming mode, but this support is coming soon ([BEAM-2059](https://issues.apache.org/jira/browse/BEAM-2059)). +YesGauge metrics are not supported. All other metric types are supported.
[beam] branch asf-site updated: Publishing website 2019/09/17 19:06:59 at commit 6d56040
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 497c18c Publishing website 2019/09/17 19:06:59 at commit 6d56040
497c18c is described below
commit 497c18ce7f7aa06d5c45ff8c9499180a6bd17f88
Author: jenkins
AuthorDate: Tue Sep 17 19:06:59 2019 +
Publishing website 2019/09/17 19:06:59 at commit 6d56040
---
.../contribute/release-guide/index.html| 111 -
1 file changed, 110 insertions(+), 1 deletion(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 803566f..9385dad 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -271,6 +271,7 @@
Deploy
artifacts to Maven Central Repository
Deploy Python artifacts
to PyPI
+ Deploy SDK docker
images to DockerHub
Git tag
Merge website pull
request
Mark the version as
released in JIRA
@@ -327,6 +328,7 @@ limitations under the License.
Submit
your GPG public key into MIT PGP Public Key Server
Website development setup
Register to PyPI
+ Login to DockerHub
Create a new version in
JIRA
@@ -364,6 +366,7 @@ limitations under the License.
Build and stage
Java artifacts with Gradle
Stage source release on
dist.apache.org
Stage python binaries
on dist.apache.org
+ Stage SDK images on
hub.docker.com
Build and stage python
wheels
@@ -395,6 +398,7 @@ limitations under the License.
Deploy source release
to dist.apache.org
+ Deploy SDK docker
images to DockerHub
Git tag
Merge website pull request
Mark the version as
released in JIRA
@@ -472,6 +476,7 @@ please follow https://help.github.com/articles/creating-a-personal-acce
GPG pass phrase (see the next section);
Apache ID and Password;
GitHub ID and Password.
+ DockerHub ID and Password. (You should be a member of maintainer team;
email at dev@ if you are not.)
One-time setup instructions
@@ -633,6 +638,32 @@ Automation Reliability.
Release manager needs to have an account with PyPI. If you need one, https://pypi.python.org/account/register/;>register at PyPI. You also
need to be a maintainer (or an owner) of the https://pypi.python.org/pypi/apache-beam;>apache-beam package in
order to push a new release. Ask on the mailing list for assistance.
+Login to DockerHub
+Run following command manually. It will ask you to input your DockerHub ID
and password if
+authorization info cannot be found from ~/.docker/config.json file.
+docker login
docker.io
+
+
+After successful login, authorization info will be stored at
~/.docker/config.json file. For example,
+"https://index.docker.io/v1/": {
+ "auth": "aGFubmFoamlhbmc6cmtkdGpmZ2hrMTIxMw=="
+}
+
+
+Release managers should have push permission; please ask for help at
dev@.
+From: Release
Manager
+To: d...@beam.apache.org
+Subject: DockerHub Push Permission
+
+Hi DockerHub Admins
+
+I need push permission to proceed with release, can you please add me to
maintainer team?
+My docker hub ID is: xxx
+
+Thanks,
+Release Manager
+
+
Create a new version in JIRA
When contributors resolve an issue in JIRA, they are tagging it with a
release that will contain their changes. With the release currently underway,
new issues should be resolved against a subsequent future release. Therefore,
you should create a release item for this subsequent release, as follows:
@@ -685,7 +716,8 @@ Automation Reliability.
Change and commit version number in release branch:
https://github.com/apache/beam/blob/release-2.6.0/sdks/python/apache_beam/version.py#L21;>version.py,
-https://github.com/apache/beam/blob/release-2.6.0/runners/google-cloud-dataflow-java/build.gradle#L39;>build.gradle
+https://github.com/apache/beam/blob/release-2.6.0/runners/google-cloud-dataflow-java/build.gradle#L39;>build.gradle,
+https://github.com/apache/beam/blob/release-2.16.0/gradle.properties#L27;>gradle.properties
@@ -759,6 +791,7 @@ git commit -m "Moving to
${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branc
DEV=${RELEASE}.dev
sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py
+sed -i -e "s/${DEV}/${RELEASE}/g" gradle.properties
sed -i -e "s/'beam-master-.*'/'beam-${RELEASE}'/g"
runners/google-cloud-dataflow-java/build.gradle
@@ -1063,6 +1096,7 @@ When prompted for a description, enter “Apache Beam,
version X, release candid
Stage source
[beam] branch asf-site updated: Publishing website 2019/09/17 15:42:15 at commit 511caa3
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c103160 Publishing website 2019/09/17 15:42:15 at commit 511caa3 c103160 is described below commit c103160c56f6ca0d6102ba4ff216e9e8ad940154 Author: jenkins AuthorDate: Tue Sep 17 15:42:15 2019 + Publishing website 2019/09/17 15:42:15 at commit 511caa3 --- website/generated-content/documentation/programming-guide/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index 0ac9b06..e0dd36b 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -2064,7 +2064,7 @@ deterministic, and your transforms’ behavior more predictable and easier to de to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other -data that can be read from within the ParDo transform’s DoFn while procesing +data that can be read from within the ParDo transform’s DoFn while processing each element. Side inputs are useful if your ParDo needs to inject additional data when
[beam] branch asf-site updated: Publishing website 2019/09/16 15:55:04 at commit d93e4e0
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0a47b92 Publishing website 2019/09/16 15:55:04 at commit d93e4e0 0a47b92 is described below commit 0a47b9221378a5bfc7b8b5cc33660fc38def79b2 Author: jenkins AuthorDate: Mon Sep 16 15:55:04 2019 + Publishing website 2019/09/16 15:55:04 at commit d93e4e0 --- website/generated-content/documentation/execution-model/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/documentation/execution-model/index.html b/website/generated-content/documentation/execution-model/index.html index e9db33f..4d52a9a 100644 --- a/website/generated-content/documentation/execution-model/index.html +++ b/website/generated-content/documentation/execution-model/index.html @@ -502,7 +502,7 @@ and for other reasons such as persistence. A runner may decide to transfer elements between transforms in a variety of ways, such as: - + Routing elements to a worker for processing as part of a grouping operation. This may involve serializing elements and grouping or sorting them by their key. @@ -514,7 +514,7 @@ the ParDo. Passing elements between transforms that are running on the same worker. This may allow the runner to avoid serializing elements; instead, the runner can just pass the elements in memory. - + Some situations where the runner may serialize and persist elements are:
[beam] branch asf-site updated: Publishing website 2019/09/15 21:58:03 at commit 43cea184
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c39046c Publishing website 2019/09/15 21:58:03 at commit 43cea184 c39046c is described below commit c39046c60e3c3b8e334239d378017f1ce297c81b Author: jenkins AuthorDate: Sun Sep 15 21:58:04 2019 + Publishing website 2019/09/15 21:58:03 at commit 43cea184 --- website/generated-content/documentation/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/index.html b/website/generated-content/documentation/index.html index c44ed3e..b542313 100644 --- a/website/generated-content/documentation/index.html +++ b/website/generated-content/documentation/index.html @@ -485,7 +485,7 @@ limitations under the License. Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners. - The Programming Guide introduces all the key Beam concepts. + Read the Programming Guide which introduces all the key Beam concepts. Learn about Beam’s execution model to better understand how pipelines execute. Visit Learning Resources for some of our favorite articles and talks about Beam.
[beam] branch asf-site updated: Publishing website 2019/09/15 21:11:41 at commit a06be47
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8ba0129 Publishing website 2019/09/15 21:11:41 at commit a06be47 8ba0129 is described below commit 8ba0129112ae7ba723bdc9d411bbdbdbfb5f1e2a Author: jenkins AuthorDate: Sun Sep 15 21:11:41 2019 + Publishing website 2019/09/15 21:11:41 at commit a06be47 --- website/generated-content/contribute/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/contribute/index.html b/website/generated-content/contribute/index.html index ce284d3..458d89b 100644 --- a/website/generated-content/contribute/index.html +++ b/website/generated-content/contribute/index.html @@ -327,9 +327,9 @@ a place for notes. Later, your pull request will be linked to the issue as well. Assign the issue to yourself. To get the permission to do so, email the dev@ mailing list to introduce yourself and to be added as a contributor in the Beam issue tracker including your -ASF Jira Username. For example https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E;>this welcome email +ASF Jira Username. For example https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E;>this welcome email. If your change is large or it is your first change, it is a good idea to -discuss it on the dev@ mailing list +discuss it on the dev@ mailing list. For large changes create a design doc (https://s.apache.org/beam-design-doc-template;>template, https://s.apache.org/beam-design-docs;>examples) and email it to the dev@ mailing list.
[beam] branch asf-site updated: Publishing website 2019/09/14 01:27:42 at commit 6b5084b
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 5e90a30 Publishing website 2019/09/14 01:27:42 at commit 6b5084b
5e90a30 is described below
commit 5e90a30063c801e04c3c3a9392f7f0f2dafb95be
Author: jenkins
AuthorDate: Sat Sep 14 01:27:43 2019 +
Publishing website 2019/09/14 01:27:42 at commit 6b5084b
---
.../python/elementwise/filter/index.html | 114 +++--
1 file changed, 84 insertions(+), 30 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index be50340..ad7d59e 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -530,9 +530,9 @@ Then, we apply Filter in multiple ways to
-Output PCollection after Filter:
+Output PCollection after Filter:
-perennials = [
+perennials = [
{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
@@ -540,11 +540,20 @@ Then, we apply Filter in multiple ways to
-
+
+
+https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run in Colab" />
+ Run code now
+
+
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View source code" />
+ View source code
@@ -573,9 +582,9 @@ Then, we apply Filter in multiple ways to
-Output PCollection after Filter:
+Output PCollection after Filter:
-perennials = [
+perennials = [
{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
@@ -583,11 +592,20 @@ Then, we apply Filter in multiple ways to
-
+
+
+https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run code now" />
+ Run code now
+
+
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View source code" />
+ View source code
@@ -621,9 +639,9 @@ They are passed as additional positional arguments or
keyword arguments to the f
-Output PCollection after Filter:
+Output PCollection after Filter:
-perennials = [
+perennials = [
{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
@@ -631,11 +649,20 @@ They are passed as additional positional arguments or
keyword arguments to the f
-
+
+
+https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb;>
+ https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png;
width="20px" height="20px" alt="Run in Colab" />
+ Run code now
+
+
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
- https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View source code" />
+ View s
[beam] branch asf-site updated: Publishing website 2019/09/11 23:12:59 at commit 15eb089
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new e4003c6 Publishing website 2019/09/11 23:12:59 at commit 15eb089 e4003c6 is described below commit e4003c654e4d55350759d3c335fddb9e09cc6e99 Author: jenkins AuthorDate: Wed Sep 11 23:13:00 2019 + Publishing website 2019/09/11 23:12:59 at commit 15eb089 --- .../get-started/quickstart-java/index.html | 12 +- .../get-started/wordcount-example/index.html | 43 ++ 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/website/generated-content/get-started/quickstart-java/index.html b/website/generated-content/get-started/quickstart-java/index.html index cfbbf5c..3c7b228 100644 --- a/website/generated-content/get-started/quickstart-java/index.html +++ b/website/generated-content/get-started/quickstart-java/index.html @@ -395,13 +395,9 @@ $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ -$ mvn package -Pjet-runner -DskipTests +$ mvn package -Pjet-runner $ java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts - -Please note that the Jet Runner will be present in released versions of Beam -only starting with 2.14.0. Until then SNAPSHOT versions need to be used when -generating the above Maven archetype for the examples code. @@ -456,13 +452,9 @@ PS java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples. -PS mvn package -P jet-runner -DskipTests +PS mvn package -P jet-runner PS java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount ` --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml --output=counts - -Please note that the Jet Runner will be present in released versions of Beam -only starting with 2.14.0. Until then SNAPSHOT versions need to be used when -generating the above Maven archetype for the examples code. diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html index 6ef8d37..6048ef4 100644 --- a/website/generated-content/get-started/wordcount-example/index.html +++ b/website/generated-content/get-started/wordcount-example/index.html @@ -662,6 +662,11 @@ You can monitor the running job by visiting the Flink dashboard at http://fl +$ mvn package -P jet-runner java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ + --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts + + + To view the full code in Java, see https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java;>WordCount. @@ -708,6 +713,10 @@ python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespea +This runner is not yet available for the Python SDK. + + + To view the full code in Python, see https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py;>wordcount.py. @@ -755,6 +764,10 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ +This runner is not yet available for the Go SDK. + + + To view the full code in Go, see https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go;>wordcount.go. @@ -1003,6 +1016,11 @@ You can monitor the running job by visiting the Flink dashboard at http://fl +$ mvn package -P jet-runner java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.DebuggingWordCount \ + --runner=JetRunner --jetLocalMode=3 --output=counts + + + To view the full code in Java, see https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java;>DebuggingWordCount. @@ -1046,6 +1064,10 @@ python -m apache_beam.examples.wordcount_debugging --input gs://dataflow-samples +This runner is not yet available for the Python SDK. + + + To view the full code in Python, see https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_debugging.py;>wordcount_debugging.py. @@ -1093,6 +1115,10 @@ $ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ +This runner is not yet available for the Go SDK. + + + To view the full code in Go, see https://github.com/apache/beam/blob/master/sdks/go/examples/debugging_wordcount/debugging_wordcount.go;>debugging_wordcount.go. @@ -1359,6 +1385,11 @@ You can monitor the running job by visiting the Flink dashboard at http://fl +$ mvn package -P jet-runner java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples
[beam] branch asf-site updated: Publishing website 2019/09/11 21:24:34 at commit daea77d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 12e2f56 Publishing website 2019/09/11 21:24:34 at commit daea77d 12e2f56 is described below commit 12e2f56255c25c564816d623a950181cc0a7df23 Author: jenkins AuthorDate: Wed Sep 11 21:24:34 2019 + Publishing website 2019/09/11 21:24:34 at commit daea77d --- website/generated-content/documentation/runners/flink/index.html | 2 +- website/generated-content/documentation/runners/spark/index.html | 2 +- website/generated-content/roadmap/portability/index.html | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index f22e6e8..4363dd1 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -464,7 +464,7 @@ download it on the Downloads page. In the available. -1. Only required once: Build the SDK harness container: ./gradlew :sdks:python:container:docker +1. Only required once: Build the SDK harness container (optionally replace py35 with the Python version of your choice): ./gradlew :sdks:python:container:py35:docker 2. Start the JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow diff --git a/website/generated-content/documentation/runners/spark/index.html b/website/generated-content/documentation/runners/spark/index.html index 603b31c..9171417 100644 --- a/website/generated-content/documentation/runners/spark/index.html +++ b/website/generated-content/documentation/runners/spark/index.html @@ -382,7 +382,7 @@ download it on the Downloads page. In the available. -1. Only required once: Build the SDK harness container: ./gradlew :sdks:python:container:docker +1. Only required once: Build the SDK harness container (optionally replace py35 with the Python version of your choice): ./gradlew :sdks:python:container:py35:docker 2. Start the JobService endpoint: ./gradlew :runners:spark:job-server:runShadow diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index cde03f4..b8a4b53 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -400,7 +400,7 @@ for details. To run a basic Python wordcount (in batch mode) with embedded Flink: - Run once to build the SDK harness container: ./gradlew :sdks:python:container:docker + Run once to build the SDK harness container (optionally replace py35 with the Python version of your choice): ./gradlew :sdks:python:container:py35:docker Start the Flink portable JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK
[beam] branch asf-site updated: Publishing website 2019/09/11 17:16:48 at commit cd7b025
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7b72d47 Publishing website 2019/09/11 17:16:48 at commit cd7b025
7b72d47 is described below
commit 7b72d47a5c320600341af29b4c75235ceb1d86fb
Author: jenkins
AuthorDate: Wed Sep 11 17:16:49 2019 +
Publishing website 2019/09/11 17:16:48 at commit cd7b025
---
.../python/elementwise/tostring/index.html | 181 -
1 file changed, 178 insertions(+), 3 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
b/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
index d974a91..9eb696d 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/tostring/index.html
@@ -447,7 +447,13 @@
- Examples
+ Examples
+
+ Example 1: Key-value
pairs to string
+ Example 2: Elements to
string
+ Example 3: Iterables to
string
+
+
Related transforms
@@ -470,13 +476,182 @@ limitations under the License.
-->
ToString
-Transforms every element in an input collection a string.
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString;>
+ https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
+ Pydoc
+
+
+
+
+Transforms every element in an input collection to a string.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+Any non-string element can be converted to a string using standard Python
functions and methods.
+Many I/O transforms, such as
+https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText;>textio.WriteToText,
+expect their input elements to be strings.
+
+Example 1: Key-value pairs to
string
+
+The following example converts a (key,
value) pair into a string delimited by ','.
+You can specify a different delimiter using the delimiter argument.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' beam.Create([
+ ('', 'Strawberry'),
+ ('凌', 'Carrot'),
+ ('', 'Eggplant'),
+ ('', 'Tomato'),
+ ('凜', 'Potato'),
+ ])
+ | 'To string' beam.ToString.Kvs()
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after ToString:
+
+plants = [
+',Strawberry',
+'凌,Carrot',
+',Eggplant',
+',Tomato',
+'凜,Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2: Elements to string
+
+The following example converts a dictionary into a string.
+The string output will be equivalent to str(element).
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plant_lists = (
+ pipeline
+ | 'Garden plants' beam.Create([
+ ['', 'Strawberry',
'perennial'],
+ ['凌', 'Carrot',
'biennial'],
+ ['', 'Eggplant',
'perennial'],
+ ['', 'Tomato',
'annual'],
+ ['凜', 'Potato',
'perennial'],
+ ])
+ | 'To string' beam.ToString.Element()
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after ToString:
+
+plant_lists = [
+"['', 'Strawberry', 'perennial']",
+"['凌', 'Carrot', 'biennial']",
+"['', 'Eggplant', 'perennial']",
+"['', 'Tomato', 'annual']",
+"['凜', 'Potato', 'perennial']",
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 3: Iterables to string
+
+The following example converts an iterable, in this case a list of strings,
+into a string delimited by ','.
+You can specify a different delimiter using the delimiter argument.
+The string output will be equivalent to iterable.join(delimiter).
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants_csv = (
+ pipeline
+ | 'Garden plants' beam.Create([
+ ['', 'Strawberry',
'perennia
[beam] branch asf-site updated: Publishing website 2019/09/11 17:15:01 at commit 40ccf08
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 67daf41 Publishing website 2019/09/11 17:15:01 at commit 40ccf08
67daf41 is described below
commit 67daf41daf5cf82751ee998261f2234c5a22dde4
Author: jenkins
AuthorDate: Wed Sep 11 17:15:01 2019 +
Publishing website 2019/09/11 17:15:01 at commit 40ccf08
---
.../transforms/python/elementwise/pardo/index.html | 306 -
1 file changed, 293 insertions(+), 13 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
index cac7dd6..eb7ac6d 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/pardo/index.html
@@ -447,7 +447,13 @@
- Examples
+ Examples
+
+ Example 1: ParDo with
a simple DoFn
+ Example 2: ParDo
with timestamp and window information
+ Example 3: ParDo with
DoFn methods
+
+
Related transforms
@@ -470,30 +476,304 @@ limitations under the License.
-->
ParDo
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-A transform for generic parallel processing. A ParDo transform considers each
-element in the input PCollection,
performs some processing function
-(your user code) on that element, and emits zero or more elements to
-an output PCollection.
+
+
+A transform for generic parallel processing.
+A ParDo transform considers each
element in the input PCollection,
+performs some processing function (your user code) on that element,
+and emits zero or more elements to an output PCollection.
-See more information in the Beam Programming Guide.
+See more information in the
+Beam Programming
Guide.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we explore how to create custom DoFns and access
+the timestamp and windowing information.
+
+Example 1: ParDo with a simple
DoFn
+
+The following example defines a simple DoFn class called SplitWords
+which stores the delimiter as an object
field.
+The process method is called once per
element,
+and it can yield zero or more output elements.
+
+import apache_beam as beam
+
+class SplitWords(beam.DoFn):
+ def __init__(self, delimiter=','):
+self.delimiter = delimiter
+
+ def process(self, text):
+for word in text.split(self.delimiter):
+ yield word
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ 'Strawberry,凌Carrot,Eggplant',
+ 'Tomato,凜Potato',
+ ])
+ | 'Split words' beam.ParDo(SplitWords(','))
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after ParDo:
+
+plants = [
+'Strawberry',
+'凌Carrot',
+'Eggplant',
+'Tomato',
+'凜Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2:
ParDo with timestamp and window information
+
+In this example, we add new parameters to the process method to bind parameter values at
runtime.
+
+
+ https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam;>beam.DoFn.TimestampParam
+binds the timestamp information as an
+https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp;>apache_beam.utils.timestamp.Timestamp
+object.
+ https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam;>beam.DoFn.WindowParam
+binds the window information as the appropriate
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html;>apache_beam.transforms.window.*Window
+object.
+
+
+import apache_beam as beam
+
+class AnalyzeElement(beam.DoFn):
+ def process(self, elem, timestamp=beam.DoFn.TimestampParam,
window=beamyield '\n'.join([
+'# timestamp',
+'type(t
[beam] branch asf-site updated: Publishing website 2019/09/11 16:47:26 at commit f0b7f8b
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 2788666 Publishing website 2019/09/11 16:47:26 at commit f0b7f8b 2788666 is described below commit 2788666a338a448e63ec1aa374e6a6f70e9f20c3 Author: jenkins AuthorDate: Wed Sep 11 16:47:27 2019 + Publishing website 2019/09/11 16:47:26 at commit f0b7f8b --- website/generated-content/get-started/quickstart-go/index.html | 2 +- website/generated-content/get-started/wordcount-example/index.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/get-started/quickstart-go/index.html b/website/generated-content/get-started/quickstart-go/index.html index 4c5b011..f3ed1dc 100644 --- a/website/generated-content/get-started/quickstart-go/index.html +++ b/website/generated-content/get-started/quickstart-go/index.html @@ -286,7 +286,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ --project your-gcp-project \ --temp_location gs://your-gcs-bucket/tmp/ \ --staging_location gs://your-gcs-bucket/binaries/ \ - --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +--worker_harness_container_image=apachebeam/go_sdk:latest diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html index 37f38f7..6ef8d37 100644 --- a/website/generated-content/get-started/wordcount-example/index.html +++ b/website/generated-content/get-started/wordcount-example/index.html @@ -743,7 +743,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ --project your-gcp-project \ --temp_location gs://your-gcs-bucket/tmp/ \ --staging_location gs://your-gcs-bucket/binaries/ \ - --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515 +--worker_harness_container_image=apachebeam/go_sdk:latest
[beam] branch asf-site updated: Publishing website 2019/09/10 21:44:14 at commit f1d48fa
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8afbe46 Publishing website 2019/09/10 21:44:14 at commit f1d48fa 8afbe46 is described below commit 8afbe4603a8087c0e44791fd917374b7bca389e8 Author: jenkins AuthorDate: Tue Sep 10 21:44:14 2019 + Publishing website 2019/09/10 21:44:14 at commit f1d48fa --- .../contribute/release-guide/index.html| 62 ++ 1 file changed, 41 insertions(+), 21 deletions(-) diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html index a41e884..803566f 100644 --- a/website/generated-content/contribute/release-guide/index.html +++ b/website/generated-content/contribute/release-guide/index.html @@ -344,7 +344,8 @@ limitations under the License. 3. Verify release branch Run automation script (verify_release_build.sh) - (Alternative) Run all commands manually + Verify the build succeeds + (Alternative) Run release build manually (locally) Create release-blocking issues in JIRA @@ -820,30 +821,52 @@ There are 2 ways to perform this verification, either running automation script( Script: https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh;>verify_release_build.sh - -Usage - -./beam/release/src/main/scripts/verify_release_build.sh + Usage + + Create a personal access token from your Github account. See instruction https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line;>here. +It’ll be used by the script for accessing Github API. + Update required configurations listed in RELEASE_BUILD_CONFIGS in https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config + Then run +cd beam/release/src/main/scripts ./verify_release_build.sh - + + + Trigger beam_Release_Gradle_Build and all PostCommit Jenkins jobs from PR (which is created by previous step). +To do so, only add one trigger phrase per comment. See JOB_TRIGGER_PHRASES in https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L43;>verify_release_build.sh +for full list of phrases. + - Tasks included + Tasks included in the script - Install pip, virtualenv, cython and /usr/bin/time with your agreements. - Run gradle release build against release branch. + Installs hub with your agreement and setup local git repo; + Create a test PR against release branch; + + +Jenkins job beam_Release_Gradle_Build basically run ./gradlew build -PisRelease. +This only verifies that everything builds with unit tests passing. + +Verify the build succeeds + + Tasks you need to do manually to verify the build succeed: Check the build result. If build failed, scan log will contain all failures. You should stabilize the release branch until release build succeeded. - The script will output a set of Jenkins phrases to enter in the created PR + The script will output a set of Jenkins phrases to enter in the created PR. -(Alternative) Run all commands manually +There are some projects that don’t produce the artifacts, e.g. beam-test-tools, you may be able to +ignore failures there. + +To triage the failures and narrow things down you may want to look at settings.gradle and run the build only for the +projects you’re interested at the moment, e.g. ./gradlew :runners:java-fn-execution. + +(Alternative) Run release build manually (locally) Pre-installation for python build @@ -875,14 +898,6 @@ There are 2 ways to perform this verification, either running automation script( - -Make sure your time alias to /usr/bin/time, if not: - - sudo apt-get install time - alias time='/usr/bin/time' - - - @@ -906,6 +921,10 @@ There are 2 ways to perform this verification, either running automation script( ./gradlew build -PisRelease --no-parallel --scan --stacktrace --continue + +To speed things up locally you might want to omit --no-parallel. You can also omit --continue +if you want build fails after the first error instead of continuing, it may be easier and faster +to find environment issues this way without having to wait until the full build completes. @@ -1429,9 +1448,10 @@ Thanks everyone! Usage - First update https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config with correct config value (e.g. release version, rc number). + First update required
[beam] branch asf-site updated: Publishing website 2019/09/10 18:19:21 at commit 740ddc6
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0dee528 Publishing website 2019/09/10 18:19:21 at commit 740ddc6 0dee528 is described below commit 0dee5284332ed887926691c82f96d789df66fd1d Author: jenkins AuthorDate: Tue Sep 10 18:19:21 2019 + Publishing website 2019/09/10 18:19:21 at commit 740ddc6 --- .../get-started/mobile-gaming-example/index.html | 16 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/website/generated-content/get-started/mobile-gaming-example/index.html b/website/generated-content/get-started/mobile-gaming-example/index.html index 9af4960..ee6e589 100644 --- a/website/generated-content/get-started/mobile-gaming-example/index.html +++ b/website/generated-content/get-started/mobile-gaming-example/index.html @@ -410,7 +410,9 @@ looks more like what is depicted by the red squiggly line above the ideal line.< extracted. """ def __init__(self, field): -super(ExtractAndSumScore, self).__init__() +# TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3. +# super(ExtractAndSumScore, self).__init__() +beam.PTransform.__init__(self) self.field = field def expand(self, pcoll): @@ -658,7 +660,9 @@ logical windows based on when those scores occurred in event time. class HourlyTeamScore(beam.PTransform): def __init__(self, start_min, stop_min, window_duration): -super(HourlyTeamScore, self).__init__() +# TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3. +# super(HourlyTeamScore, self).__init__() +beam.PTransform.__init__(self) self.start_timestamp = str2timestamp(start_min) self.stop_timestamp = str2timestamp(stop_min) self.window_duration_in_seconds = window_duration * 60 @@ -859,7 +863,9 @@ ten minutes after data is received. global windowing. Get periodic updates on all users' running scores. """ def __init__(self, allowed_lateness): -super(CalculateUserScores, self).__init__() +# TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3. +# super(CalculateUserScores, self).__init__() +beam.PTransform.__init__(self) self.allowed_lateness_seconds = allowed_lateness * 60 def expand(self, pcoll): @@ -947,7 +953,9 @@ late results. default. """ def __init__(self, team_window_duration, allowed_lateness): -super(CalculateTeamScores, self).__init__() +# TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3. +# super(CalculateTeamScores, self).__init__() +beam.PTransform.__init__(self) self.team_window_duration = team_window_duration * 60 self.allowed_lateness_seconds = allowed_lateness * 60
[beam] branch asf-site updated: Publishing website 2019/09/06 15:52:26 at commit 8b7a3e3
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 766d92d Publishing website 2019/09/06 15:52:26 at commit 8b7a3e3 766d92d is described below commit 766d92d82f28eeeca3e673086573e45c80e9b1fd Author: jenkins AuthorDate: Fri Sep 6 15:52:27 2019 + Publishing website 2019/09/06 15:52:26 at commit 8b7a3e3 --- .../generated-content/documentation/io/built-in/hcatalog/index.html | 4 ++-- .../documentation/runners/capability-matrix/index.html| 2 +- website/generated-content/documentation/sdks/java/euphoria/index.html | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/website/generated-content/documentation/io/built-in/hcatalog/index.html b/website/generated-content/documentation/io/built-in/hcatalog/index.html index 773ffb8..ddf88e0 100644 --- a/website/generated-content/documentation/io/built-in/hcatalog/index.html +++ b/website/generated-content/documentation/io/built-in/hcatalog/index.html @@ -28,7 +28,7 @@ - Apache HCatalog InputFormat IO + Apache HCatalog I/O connector https://fonts.googleapis.com/css?family=Roboto:100,300,400; rel="stylesheet"> @@ -519,7 +519,7 @@ The destination table should exist beforehand as the transform will not create a Using older versions of HCatalog (1.x) -HCatalogIO is build for Apache HCatalog versions 2 and up and will not work out of the box for older versions of HCatalog. +HCatalogIO is built for Apache HCatalog versions 2 and up and will not work out of the box for older versions of HCatalog. The following illustrates a workaround to work with Hive 1.1. Include the following Hive 1.2 jars in the über jar you build. diff --git a/website/generated-content/documentation/runners/capability-matrix/index.html b/website/generated-content/documentation/runners/capability-matrix/index.html index 56a004c..449ecca 100644 --- a/website/generated-content/documentation/runners/capability-matrix/index.html +++ b/website/generated-content/documentation/runners/capability-matrix/index.html @@ -13549,7 +13549,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -Partially: APIs and semantics for draining a pipeline are under discussion. This would cause incomplete aggregations to be emitted regardless of trigger and tagged with metadata indicating it is incomplated. +Partially: APIs and semantics for draining a pipeline are under discussion. This would cause incomplete aggregations to be emitted regardless of trigger and tagged with metadata indicating it is incompleted. diff --git a/website/generated-content/documentation/sdks/java/euphoria/index.html b/website/generated-content/documentation/sdks/java/euphoria/index.html index 3c1f842..39e332d 100644 --- a/website/generated-content/documentation/sdks/java/euphoria/index.html +++ b/website/generated-content/documentation/sdks/java/euphoria/index.html @@ -830,7 +830,7 @@ the API as a high level DSL over Beam Java SDK and share our effort with the com Translation -Euphoria API is build on top of Beam Java SDK. The API is transparently translated into Beam’s PTransforms in background. +Euphoria API is built on top of Beam Java SDK. The API is transparently translated into Beam’s PTransforms in background. The fact that Euphoria API is translated to Beam Java SDK give us option to fine tune the translation itself. Translation of an Operator is realized through implementations of OperatorTranslator. Euphoria uses TranslationProvider to decide which translator should be used. User of Euphoria API can supply its own OperatorTranslator through TranslationProvider by extending EuphoriaOptions.
[beam] branch asf-site updated: Publishing website 2019/09/05 22:27:36 at commit 6f88601
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6db6c24 Publishing website 2019/09/05 22:27:36 at commit 6f88601
6db6c24 is described below
commit 6db6c24596b15fd2d31936b99a9a016569360cda
Author: jenkins
AuthorDate: Thu Sep 5 22:27:36 2019 +
Publishing website 2019/09/05 22:27:36 at commit 6f88601
---
.../transforms/python/elementwise/regex/index.html | 551 -
1 file changed, 548 insertions(+), 3 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
b/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
index f6febb1..f6e65a4 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/regex/index.html
@@ -447,7 +447,19 @@
- Examples
+ Examples
+
+ Example 1: Regex match
+ Example 2: Regex
match with all groups
+ Example 3:
Regex match into key-value pairs
+ Example 4: Regex find
+ Example 5: Regex find
all
+ Example 6: Regex
find as key-value pairs
+ Example 7: Regex replace
all
+ Example 8: Regex replace
first
+ Example 9: Regex split
+
+
Related transforms
@@ -470,16 +482,549 @@ limitations under the License.
-->
Regex
-Filters input string elements based on a regex. May also transform them
based on the matching groups.
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex;>
+ https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
+ Pydoc
+
+
+
+
+Filters input string elements based on a regex. May also transform them based
on the matching groups.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we create a pipeline with a PCollection of text strings.
+Then, we use the Regex transform to
search, replace, and split through the text elements using
+https://docs.python.org/3/library/re.html;>regular
expressions.
+
+You can use tools to help you create and test your regular expressions,
such as
+https://regex101.com/;>regex101.
+Make sure to specify the Python flavor at the left side bar.
+
+Lets look at the
+https://regex101.com/r/Z7hTTj/3;>regular expression (?Picon[^\s,]+), *(\w+), *(\w+)
+for example.
+It matches anything that is not a whitespace \s ([
\t\n\r\f\v]) or comma ,
+until a comma is found and stores that in the named group icon,
+this can match even utf-8 strings.
+Then it matches any number of whitespaces, followed by at least one word
character
+\w ([a-zA-Z0-9_]), which is stored in the second
group for the name.
+It does the same with the third group for the duration.
+
+
+ Note: To avoid unexpected string escaping in your regular
expressions,
+it is recommended to use
+https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals;>raw
strings
+such as r'raw-string' instead of 'escaped-string'.
+
+
+Example 1: Regex match
+
+Regex.matches keeps only the
elements that match the regular expression,
+returning the matched group.
+The argument group is set to 0 (the entire match) by default,
+but can be set to a group number like 3, or to a named group like 'icon'.
+
+Regex.matches starts to match the
regular expression at the beginning of the string.
+To match until the end of the string, add '$' at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+Regex.find(regex).
+
+import apache_beam as beam
+
+# Matches a named group 'icon', and then two comma-separated
groups.
+regex = r'(?Picon[^\s,]+), *(\w+),
*(\w+)'
+with beam.Pipeline()
as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' beam.Create([
+ ', Strawberry, perennial',
+ '凌, Carrot, biennial ignoring trailing
words',
+ ', Eggplant, perennial',
+ ', Tomato, annual',
+ '凜, Potato, perennial',
+ '# , invalid, format',
+ 'invalid, , format',
+ ])
+ | 'Parse plants' beam.Regex.matches(regex)
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after Regex.matches:
+
+plants_matches = [
+', Strawberry, perennial',
+'凌, Carrot, biennial',
+', Eggplant, perennial',
+', Tomato, annual',
+'凜, Potato, perennial',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py;>
+ https
[beam] branch asf-site updated: Publishing website 2019/09/05 12:43:13 at commit 8869fce
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a8f93eb Publishing website 2019/09/05 12:43:13 at commit 8869fce a8f93eb is described below commit a8f93eb1b978b10a6c1e3836a452eeddffc9a472 Author: jenkins AuthorDate: Thu Sep 5 12:43:14 2019 + Publishing website 2019/09/05 12:43:13 at commit 8869fce --- website/generated-content/documentation/runners/flink/index.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 10c8041..f22e6e8 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -472,8 +472,8 @@ available. The JobService is the central instance where you submit your Beam pipeline to. -The JobService will create a Flink job for the pipeline and execute the job -job. To execute the job on a Flink cluster, the Beam JobService needs to be +The JobService will create a Flink job for the pipeline and execute the job. +To execute the job on a Flink cluster, the Beam JobService needs to be provided with the Flink JobManager address.
[beam] branch asf-site updated: Publishing website 2019/09/04 17:00:12 at commit d251702
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new da2ed38 Publishing website 2019/09/04 17:00:12 at commit d251702
da2ed38 is described below
commit da2ed38e5fe9402d76d691f1f733a91e9fbc49d0
Author: jenkins
AuthorDate: Wed Sep 4 17:00:13 2019 +
Publishing website 2019/09/04 17:00:12 at commit d251702
---
.../{index.html => blog/2019/09/04/gsoc-19.html} | 299 +++--
website/generated-content/blog/index.html | 32 +++
website/generated-content/feed.xml | 162 ++-
website/generated-content/index.html | 10 +-
4 files changed, 232 insertions(+), 271 deletions(-)
diff --git a/website/generated-content/index.html
b/website/generated-content/blog/2019/09/04/gsoc-19.html
similarity index 53%
copy from website/generated-content/index.html
copy to website/generated-content/blog/2019/09/04/gsoc-19.html
index 57b28e2..f651080 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/blog/2019/09/04/gsoc-19.html
@@ -28,9 +28,8 @@
- Apache Beam
-
+ Google Summer of Code 19
+
https://fonts.googleapis.com/css?family=Roboto:100,300,400;
rel="stylesheet">
https://code.jquery.com/jquery-2.2.4.min.js"</a>;>
@@ -39,7 +38,7 @@
- https://beam.apache.org/; data-proofer-ignore>
+ https://beam.apache.org/blog/2019/09/04/gsoc-19.html; data-proofer-ignore>
https://beam.apache.org/feed.xml;>
https://use.fontawesome.com/releases/v5.4.1/css/all.css;
integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz"
crossorigin="anonymous">
@@ -53,7 +52,7 @@
-
+
- https://github.com/apache/beam/edit/master/website/src/index.md;
data-proofer-ignore>
+ https://github.com/apache/beam/edit/master/website/src/_posts/2019-09-04-gsoc-19.md;
data-proofer-ignore>
@@ -151,6 +150,31 @@
+
+
+
+http://schema.org/BlogPosting;>
+
+
+Google Summer of Code
'19
+Sep 4, 2019 •
+ Tanay Tummalapalli [https://twitter.com/ttanay100;>@ttanay100]
+
+
+
+
+
+
-
-
-
-
-
-
-Apache Beam: An advanced unified programming model
-
-
-Implement batch and streaming data processing jobs that run on any
execution engine.
-
-
-Learn more
-Try Beam
-Download Beam SDK 2.15.0
-
-
-Java
Quickstart
-Python
Quickstart
-Go
Quickstart
-
-
-
-
-
-
-The latest from the blog
-
-
-
-
- Apache Beam
2.15.0
- Aug 22, 2019
-
-
-
- Apache Beam
2.14.0
- Jul 31, 2019
-
-
-
- Looping timers in
Apache Beam
- Jun 11, 2019
-
-
-
-
-
-
-
-
-
-
-All about Apache Beam
-
-
-
-
-
-Unified
-
-
-Use a single programming model for both batch and streaming use cases.
-
-
-
-
-
-Portable
-
-
-Execute pipelines on multiple execution environments.
-
-
-
-
-
-Extensible
-
-
-Write and share new SDKs, IO connectors, and transformation libraries.
-
-
-
-
-
+Google Summer of Code was an amazing learning experience for me.
+I contributed to open source, learned about Apache Beam’s internals and worked
with the best engineers in the world.
-
-
-
-
-
+
-
-
-Works with
-
-
-
-
- http://apex.apache.org;>
-
-
-
- http://flink.apache.org;>
-
-
-
- http://spark.apache.org/;>
-
-
-
- https://cloud.google.com/dataflow/;>
-
-
-
- http://gearpump.apache.org/;>
-
-
-
- http://samza.apache.org/;>
-
-
-
-
+Motivation
+Two of my friends had participated in GSoC in 2018. I was intrigued by
their experience.
+The idea of working on open-source software that could potentially be used by
developers across the world, while being mentored by the best people in a field
was exciting!
+So, I decided to give Google Summer of Code a shot this year.
-
-
-
- Testimonials
-
-
-
-
-
- A framework that delivers the flexibility and advanced function
[beam] branch asf-site updated: Publishing website 2019/09/03 21:31:54 at commit 8f01a44
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0c45eed Publishing website 2019/09/03 21:31:54 at commit 8f01a44 0c45eed is described below commit 0c45eed65f1d19b7c25d8ca879e539e90784894e Author: jenkins AuthorDate: Tue Sep 3 21:31:54 2019 + Publishing website 2019/09/03 21:31:54 at commit 8f01a44 --- .../contribute/release-guide/index.html| 426 + 1 file changed, 267 insertions(+), 159 deletions(-) diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html index b9889ac..a41e884 100644 --- a/website/generated-content/contribute/release-guide/index.html +++ b/website/generated-content/contribute/release-guide/index.html @@ -218,45 +218,56 @@ - Introduction - Overview - Decide to release + Introduction + Overview + Decide to release Checklist to proceed to the next step - Prepare for the release + 1. Prepare for the release + Accounts One-time setup instructions Create a new version in JIRA - Create a release branch in apache/beam repository + + + 2. Create a release branch in apache/beam repository + Start a snapshot build - Verify release branch - Update and Verify Javadoc - Triage release-blocking issues in JIRA + + + 3. Verify release branch + 4. Triage release-blocking issues in JIRA + Review Release Notes in JIRA - Checklist to proceed to the next step + Review cherry-picks - Build a release candidate + 5. Build a release candidate - Run build_release_candidate.sh to create RC - Run all steps manually - Write the Beam blog post and create a pull request + Checklist before proceeding + Run build_release_candidate.sh to create a release candidate + (Alternative) Run all steps manually + Build and stage python wheels - Vote on the release candidate + 6. Prepare documents - Run validation tests - Checklist to proceed to the finalization step + Update and Verify Javadoc + Build the Pydoc API reference + Propose pull requests for website updates + Blog post - Fix any issues + 7. Vote and validate release candidate - Checklist to proceed to the next step + Run validation tests + Fix any issues + Checklist to proceed to the next step - Finalize the release + 8. Finalize the release Deploy artifacts to Maven Central Repository Deploy Python artifacts to PyPI @@ -264,10 +275,10 @@ Merge website pull request Mark the version as released in JIRA Recordkeeping with ASF - Checklist to proceed to the next step + Checklist to proceed to the next step - Promote the release + 9. Promote the release Apache mailing lists Social media @@ -298,13 +309,14 @@ limitations under the License. Apache Beam Release Guide - Introduction - Overview - Decide to release + Introduction + Overview + Decide to release Checklist to proceed to the next step - Prepare for the release + 1. Prepare for the release + Accounts One-time setup instructions GPG Key Use preparation_before_release.sh to setup GPG @@ -318,59 +330,65 @@ limitations under the License. Create a new version in JIRA - Create a release branch in apache/beam repository - Use cut_release_branch.sh to cut a release branch - Run all steps manually - - + + + 2. Create a release branch in apache/beam repository + Use cut_release_branch.sh to cut a release branch + (Alternative) Run all steps manually Start a snapshot build Run start_snapshot_build.sh to trigger build - Do all operations manually - - - Verify release branch - Run automation script (verify_release_build.sh) - Run all commands manually - Create release-blocking issues in JIRA + (Alternative) Do all operations manually - Update and Verify Javadoc - Triage release-blocking issues in JIRA + + + 3. Verify release branch + Run automation script (verify_release_build.sh) + (Alternative) Run all commands manually + Create release-blocking issues in JIRA + + + 4. Triage release-blocking issues in JIRA Review Release Notes in JIRA - Checklist to proceed to the next step + Review cherry-picks
[beam] branch asf-site updated: Publishing website 2019/08/29 00:36:04 at commit afeba4c
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new d1214a1 Publishing website 2019/08/29 00:36:04 at commit afeba4c d1214a1 is described below commit d1214a1a4c5782461c41aa6fe3a8d471f9539d64 Author: jenkins AuthorDate: Thu Aug 29 00:36:04 2019 + Publishing website 2019/08/29 00:36:04 at commit afeba4c --- website/generated-content/blog/2019/08/22/beam-2.15.0.html | 1 + website/generated-content/feed.xml | 1 + 2 files changed, 2 insertions(+) diff --git a/website/generated-content/blog/2019/08/22/beam-2.15.0.html b/website/generated-content/blog/2019/08/22/beam-2.15.0.html index 2616294..6469dae 100644 --- a/website/generated-content/blog/2019/08/22/beam-2.15.0.html +++ b/website/generated-content/blog/2019/08/22/beam-2.15.0.html @@ -228,6 +228,7 @@ For more information on changes in 2.15.0, check out the https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. (Regression from 2.14.0) + https://issues.apache.org/jira/browse/BEAM-8111;>BEAM-8111 SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0) Breaking Changes diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index c392aa8..ce1f99f 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -75,6 +75,7 @@ For more information on changes in 2.15.0, check out the ul lia href=https://issues.apache.org/jira/browse/BEAM-7616BEAM-7616/a; urlopen calls may get stuck. (Regression from 2.14.0)/li + lia href=https://issues.apache.org/jira/browse/BEAM-8111BEAM-8111/a; SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0)/li /ul h3 id=breaking-changesBreaking Changes/h3
[beam] branch asf-site updated: Publishing website 2019/08/27 20:54:13 at commit f0618c5
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 368132a Publishing website 2019/08/27 20:54:13 at commit f0618c5 368132a is described below commit 368132af170bddc4a7613aa623c277dbce38daf7 Author: jenkins AuthorDate: Tue Aug 27 20:54:13 2019 + Publishing website 2019/08/27 20:54:13 at commit f0618c5 --- website/generated-content/documentation/io/built-in/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/documentation/io/built-in/index.html b/website/generated-content/documentation/io/built-in/index.html index a7e709a..13e69a6 100644 --- a/website/generated-content/documentation/io/built-in/index.html +++ b/website/generated-content/documentation/io/built-in/index.html @@ -533,6 +533,7 @@ limitations under the License. Google BigQuery https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore;>Google Cloud Datastore https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigtableio.py;>Google Cloud Bigtable (Write) +https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/mongodbio.py;>MongoDB
[beam] branch asf-site updated: Publishing website 2019/08/27 14:59:40 at commit 3d4ebaf
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 1eb4cc4 Publishing website 2019/08/27 14:59:40 at commit 3d4ebaf 1eb4cc4 is described below commit 1eb4cc4ca70ecd496178535b4358bbe5364af0bc Author: jenkins AuthorDate: Tue Aug 27 14:59:41 2019 + Publishing website 2019/08/27 14:59:40 at commit 3d4ebaf --- .../release-guide/validate-candidate/index.html| 5 +- .../release-guide/verify-branch/index.html | 64 ++ 2 files changed, 43 insertions(+), 26 deletions(-) diff --git a/website/generated-content/contribute/release-guide/validate-candidate/index.html b/website/generated-content/contribute/release-guide/validate-candidate/index.html index d5f40b7..be68125 100644 --- a/website/generated-content/contribute/release-guide/validate-candidate/index.html +++ b/website/generated-content/contribute/release-guide/validate-candidate/index.html @@ -352,9 +352,10 @@ In case of script failure, you can still run all of them manually. Usage - First update https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config with correct config value (e.g. release version, rc number). + First update required configurations listed in RC_VALIDATE_CONFIGS in +https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config Then run -./beam/release/src/main/scripts/run_rc_validation.sh +cd release/src/main/scripts ./run_rc_validation.sh diff --git a/website/generated-content/contribute/release-guide/verify-branch/index.html b/website/generated-content/contribute/release-guide/verify-branch/index.html index fd409d5..73c2db4 100644 --- a/website/generated-content/contribute/release-guide/verify-branch/index.html +++ b/website/generated-content/contribute/release-guide/verify-branch/index.html @@ -238,7 +238,7 @@ Verify the build succeeds - (Alternative) Run all commands manually + (Alternative) Run release build locally Create release-blocking issues in JIRA Checklist to proceed to the next step @@ -263,40 +263,43 @@ limitations under the License. Verify the release branch - - After the release branch is cut you need to make sure it builds and has no significant issues that would block the creation of the release candidate. + + Run verify_release_build.sh Script: https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh;>verify_release_build.sh - -Usage - -./beam/release/src/main/scripts/verify_release_build.sh + Usage + + Create a personal access token from your Github account. See instruction https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line;>here. +It’ll be used by the script for accessing Github API. + Update required configurations listed in RELEASE_BUILD_CONFIGS in https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config + Then run +cd release/src/main/scripts ./verify_release_build.sh - + + + Trigger Jenkins beam_Release_Gradle_Build and all PostCommit jobs from PR that’s created from previous step. +To do so, only add one trigger phrase per comment. See JOB_TRIGGER_PHRASES in https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L43;>verify_release_build.sh +for full list of phrases. + - -The script does the following: + The script does the following: - Installs pip, virtualenv, cython and /usr/bin/time with your agreements; - Runs gradle release build against release branch; + Installs hub with your agreement and setup local git repo; + Create a test PR against release branch; -Basically it does ./gradlew build -PisRelease from the root of Beam repository. +Jenkins job beam_Release_Gradle_Build basically run ./gradlew build -PisRelease. This only verifies that everything builds with unit tests passing. -To speed things up locally you might want to omit --no-parallel. -You might want to omit --continue if you want the script to fail after the first error instead of continuing, -it may be easier and faster to find environment issues this way without having to wait until the full build completes. - -There are some projects that don’t produce the artifacts, e.g. beam-test-tools`, you may be able to +There are some projects that don’t produce the artifacts, e.g. beam-test-tools, you may be able to ignore failures there. To triage the failures and narrow things down you may want to look at settings.gradle and run the build only for the @@ -309,18 +312,27 @@ projects you’re intere
[beam] branch asf-site updated: Publishing website 2019/08/23 19:47:04 at commit 279c55f
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new dba2a69 Publishing website 2019/08/23 19:47:04 at commit 279c55f dba2a69 is described below commit dba2a6945e6a8883d28fffd39583887e719b8743 Author: jenkins AuthorDate: Fri Aug 23 19:47:05 2019 + Publishing website 2019/08/23 19:47:04 at commit 279c55f --- .../blog/2019/08/22/beam-2.15.0.html| 21 +++-- website/generated-content/feed.xml | 21 +++-- 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/website/generated-content/blog/2019/08/22/beam-2.15.0.html b/website/generated-content/blog/2019/08/22/beam-2.15.0.html index 9169592..2616294 100644 --- a/website/generated-content/blog/2019/08/22/beam-2.15.0.html +++ b/website/generated-content/blog/2019/08/22/beam-2.15.0.html @@ -197,21 +197,25 @@ For more information on changes in 2.15.0, check out the Vendored Guava was upgraded to version 26.0. + Support multi-process execution on the FnApiRunner for Python. (https://issues.apache.org/jira/browse/BEAM-3645;>BEAM-3645) I/Os - Add AvroIO.sink for IndexedRecord (FileIO compatible). + Add AvroIO.sink for IndexedRecord (FileIO compatible). (https://issues.apache.org/jira/browse/BEAM-6480;>BEAM-6480) + Add support for writing to BigQuery clustered tables. (https://issues.apache.org/jira/browse/BEAM-5191;>BEAM-5191) New Features / Improvements - Support ParquetTable in SQL. - Add hot key detection to Dataflow Runner. - Enable throughput collection in BigQuery performance tests. - Support schemas in the JDBC sink. + Support ParquetTable in SQL. (https://issues.apache.org/jira/browse/BEAM-7728;>BEAM-7728) + Add hot key detection to Dataflow Runner. (https://issues.apache.org/jira/browse/BEAM-7820;>BEAM-7820) + Support schemas in the JDBC sink. (https://issues.apache.org/jira/browse/BEAM-6675;>BEAM-6675) + Report GCS throttling time to Dataflow autoscaler for better autoscaling. (https://issues.apache.org/jira/browse/BEAM-7667;>BEAM-7667) + Support transform_name_mapping option in Python SDK for --update use. (https://issues.apache.org/jira/browse/BEAM-7761;>BEAM-7761) + Dependency: Upgrade Jackson databind to version 2.9.9.3 (https://issues.apache.org/jira/browse/BEAM-7880;>BEAM-7880) Bugfixes @@ -223,7 +227,12 @@ For more information on changes in 2.15.0, check out the Known Issues - https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. + https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. (Regression from 2.14.0) + + +Breaking Changes + + --region flag will be a required flag in the future for Dataflow. A warning is added to warn for this future change. (https://issues.apache.org/jira/browse/BEAM-7833;>BEAM-7833) List of Contributors diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index 48ff680..c392aa8 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -44,21 +44,25 @@ For more information on changes in 2.15.0, check out the ul liVendored Guava was upgraded to version 26.0./li + liSupport multi-process execution on the FnApiRunner for Python. (a href=https://issues.apache.org/jira/browse/BEAM-3645BEAM-3645/a;)/li /ul h3 id=iosI/Os/h3 ul - liAdd AvroIO.sink for IndexedRecord (FileIO compatible)./li + liAdd AvroIO.sink for IndexedRecord (FileIO compatible). (a href=https://issues.apache.org/jira/browse/BEAM-6480BEAM-6480/a;)/li + liAdd support for writing to BigQuery clustered tables. (a href=https://issues.apache.org/jira/browse/BEAM-5191BEAM-5191/a;)/li /ul h3 id=new-features--improvementsNew Features / Improvements/h3 ul - liSupport ParquetTable in SQL./li - liAdd hot key detection to Dataflow Runner./li - liEnable throughput collection in BigQuery performance tests./li - liSupport schemas in the JDBC sink./li + liSupport ParquetTable in SQL. (a href=https://issues.apache.org/jira/browse/BEAM-7728BEAM-7728/a;)/li + liAdd hot key detection to Dataflow Runner. (a href=https://issues.apache.org/jira/browse/BEAM-7820BEAM-7820/a;)/li + liSupport schemas in the JDBC sink. (a href=https://issues.apache.org/jira/browse/BEAM-6675BEAM-6675/a;)/li + liReport GCS throttling time to Dataflow autoscaler for better autoscaling. (a href=https://issues.apache.org/jira/browse/BEAM-7667BEAM-7667/a;)/li + liSupport transform_name_mapping option in Python SDK for code class=highlighter-rouge--update/code use. (a href=https://issues.apache.org/jira/browse/BEAM-7761BEAM-7761/a;)/li + liDependency: Upgrade Jackson databind to version 2.9.9.3 (a href=https://issues.apache.org/jira/browse/BEA
[beam] branch asf-site updated: Publishing website 2019/08/23 16:01:42 at commit dcb389d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new e8fdb9e Publishing website 2019/08/23 16:01:42 at commit dcb389d e8fdb9e is described below commit e8fdb9e609342b76ad0533a5163362c10232a598 Author: jenkins AuthorDate: Fri Aug 23 16:01:43 2019 + Publishing website 2019/08/23 16:01:42 at commit dcb389d --- .../sdks/python-dependencies/index.html| 39 ++ 1 file changed, 39 insertions(+) diff --git a/website/generated-content/documentation/sdks/python-dependencies/index.html b/website/generated-content/documentation/sdks/python-dependencies/index.html index 2753af7..7c35d28 100644 --- a/website/generated-content/documentation/sdks/python-dependencies/index.html +++ b/website/generated-content/documentation/sdks/python-dependencies/index.html @@ -290,6 +290,45 @@ the listed versions that will be in scope during execution. To see the compile and runtime dependencies for your Beam SDK version, expand the relevant section below. +2.15.0 + +Beam SDK for Python 2.15.0 has the following compile and + runtime dependencies. + + PackageVersion + avro-python3=1.8.1,2.0.0; python_version = "3.0" + avro=1.8.1,2.0.0; python_version "3.0" + cachetools=3.1.0,4 + crcmod=1.7,2.0 + dill=0.2.9,0.2.10 + fastavro=0.21.4,0.22 + future=0.16.0,1.0.0 + futures=3.2.0,4.0.0; python_version "3.0" + google-apitools=0.5.28,0.5.29 + google-cloud-bigquery=1.6.0,1.18.0 + google-cloud-bigtable=0.31.1,0.33.0 + google-cloud-core=0.28.1,2 + google-cloud-datastore=1.7.1,1.8.0 + google-cloud-pubsub=0.39.0,0.40.0 + googledatastore=7.0.1,7.1; python_version "3.0" + grpcio=1.8,2 + hdfs=2.1.0,3.0.0 + httplib2=0.8,=0.12.0 + mock=1.0.1,3.0.0 + oauth2client=2.0.1,4 + proto-google-cloud-datastore-v1=0.90.0,=0.90.4; python_version "3.0" + protobuf=3.5.0.post1,4 + pyarrow=0.11.1,0.15.0; python_version = "3.0" or platform_system != "Windows" + pydot=1.2.0,2 + pymongo=3.8.0,4.0.0 + pytz=2018.3 + pyvcf=0.6.8,0.7.0; python_version "3.0" + pyyaml=3.12,4.0.0 + typing=3.6.0,3.7.0; python_version "3.5.0" + + + + 2.14.0 Beam SDK for Python 2.14.0 has the following compile and
[beam] branch asf-site updated: Publishing website 2019/08/23 07:52:19 at commit b830a6c
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new d7a97e3 Publishing website 2019/08/23 07:52:19 at commit b830a6c d7a97e3 is described below commit d7a97e3e3f896b21f9fd03c88757465b6f0554e0 Author: jenkins AuthorDate: Fri Aug 23 07:52:20 2019 + Publishing website 2019/08/23 07:52:19 at commit b830a6c --- website/generated-content/get-started/downloads/index.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html index 3d36f28..3e12893 100644 --- a/website/generated-content/get-started/downloads/index.html +++ b/website/generated-content/get-started/downloads/index.html @@ -216,9 +216,9 @@ API stability Releases - 2.15.0 (2019-08-22) - 2.14.0 (2019-08-01) + 2.15.0 (2019-08-22) + 2.14.0 (2019-08-01) 2.13.0 (2019-05-21) 2.12.0 (2019-04-25) 2.11.0 (2019-02-26) @@ -337,7 +337,7 @@ versions denoted 0.x.y. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345489;>Release notes. -2.14.0 (2019-08-01) +2.14.0 (2019-08-01) Official http://www.apache.org/dyn/closer.cgi/beam/2.14.0/apache-beam-2.14.0-source-release.zip;>source code download. https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.sha512;>SHA-512. https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.asc;>signature.
[beam] branch asf-site updated: Publishing website 2019/08/23 07:27:37 at commit 0020398
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new cd2f457 Publishing website 2019/08/23 07:27:37 at commit 0020398 cd2f457 is described below commit cd2f45724fead770f8bdc52a87ff904bab0bfeca Author: jenkins AuthorDate: Fri Aug 23 07:27:37 2019 + Publishing website 2019/08/23 07:27:37 at commit 0020398 --- .../generated-content/2019/08/22/beam-2.15.0.html | 83 -- .../2019/08/22/beam-2.15.0.html} | 288 +++-- website/generated-content/blog/index.html | 25 +- website/generated-content/feed.xml | 33 +-- website/generated-content/index.html | 4 +- 5 files changed, 122 insertions(+), 311 deletions(-) diff --git a/website/generated-content/2019/08/22/beam-2.15.0.html b/website/generated-content/2019/08/22/beam-2.15.0.html deleted file mode 100644 index 975787f..000 --- a/website/generated-content/2019/08/22/beam-2.15.0.html +++ /dev/null @@ -1,83 +0,0 @@ - - -layout: post -title: “Apache Beam 2.15.0” -date: 2019-08-22 00:00:01 -0800 -Date above corrected but keep the old URL: -permalink: /blog/2019/08/22/beam-2.15.0.html -excerpt_separator: -categories: blog -authors: -- yifanzou - - - - -We are happy to present the new 2.15.0 release of Beam. This release includes both improvements and new functionality. -See the download page for this release. -For more information on changes in 2.15.0, check out the -https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345489;>detailed release notes. - -Highlights - - - Vendored Guava was upgraded to version 26.0. - - -I/Os - - - Add AvroIO.sink for IndexedRecord (FileIO compatible). - - -New Features / Improvements - - - Support ParquetTable in SQL. - Add hot key detection to Dataflow Runner. - Enable throughput collection in BigQuery performance tests. - Support schemas in the JDBC sink. - - -Bugfixes - - - Various bug fixes and performance improvements. - - -Known Issues - - - https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. - - -List of Contributors - -According to git shortlog, the following people contributed to the 2.15.0 release. Thank you to all contributors! - -Ahmet Altay, Alexey Romanenko, Alex Goos, Alireza Samadian, Andrew Pilloud, Ankur Goenka, -Anton Kedin, Aryan Naraghi, Bartok Jozsef, bmv126, B M VISHWAS, Boyuan Zhang, -Brian Hulette, brucearctor, Cade Markegard, Cam Mach, Chad Dombrova, -Chaim Turkel, Chamikara Jayalath, Charith Ellawala, Claire McGinty, Craig Chambers, -Daniel Oliveira, David Cavazos, David Moravek, Dominic Mitchell, Dustin Rhodes, -Etienne Chauchot, Filipe Regadas, Gleb Kanterov, Gunnar Schulze, Hannah Jiang, -Heejong Lee, Henry Suryawirawan, Ismaël Mejía, Ivo Galic, Jan Lukavský, -Jawad, Juta, Juta Staes, Kai Jiang, Kamil Wasilewski, Kasia Kucharczyk, -Kenneth Jung, Kenneth Knowles, Kyle Weaver, Lily Li, Logan HAUSPIE, lostluck, -Łukasz Gajowy, Luke Cwik, Mark Liu, Matt Helm, Maximilian Michels, -Michael Luckey, Mikhail Gryzykhin, Neville Li, Nicholas Rucci, pabloem, -Pablo Estrada, Paul King, Paul Suganthan, Raheel Khan, Rakesh Kumar, -Reza Rokni, Robert Bradshaw, Robert Burke, rosetn, Rui Wang, Ryan Skraba, RyanSkraba, -Sahith Nallapareddy, Sam Rohde, Sam Whittle, Steve Niemitz, Tanay Tummalapalli, Thomas Weise, -Tianyang Hu, ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Wout Scheepers, -yanzhi, Yekut, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, Yunqing Zhou diff --git a/website/generated-content/index.html b/website/generated-content/blog/2019/08/22/beam-2.15.0.html similarity index 59% copy from website/generated-content/index.html copy to website/generated-content/blog/2019/08/22/beam-2.15.0.html index 0e54f26..9169592 100644 --- a/website/generated-content/index.html +++ b/website/generated-content/blog/2019/08/22/beam-2.15.0.html @@ -28,9 +28,8 @@ - Apache Beam - + Apache Beam 2.15.0 + https://fonts.googleapis.com/css?family=Roboto:100,300,400; rel="stylesheet"> https://code.jquery.com/jquery-2.2.4.min.js"</a>;> @@ -39,7 +38,7 @@ - https://beam.apache.org/; data-proofer-ignore> + https://beam.apache.org/blog/2019/08/22/beam-2.15.0.html; data-proofer-ignore> https://beam.apache.org/feed.xml;> https://use.fontawesome.com/releases/v5.4.1/css/all.css; integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous"> @@ -53,7 +52,7 @@ - + - https://github.com/apache/beam/edit/master/website/src/index.md; data-proofer-ignore> + https://github.com/apache/beam/edit/master
[beam] branch asf-site updated: Publishing website 2019/08/23 06:44:58 at commit 93f02f9
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c85832e Publishing website 2019/08/23 06:44:58 at commit 93f02f9 c85832e is described below commit c85832e9692c34ed38f2862132321dae94a03fcc Author: jenkins AuthorDate: Fri Aug 23 06:44:59 2019 + Publishing website 2019/08/23 06:44:58 at commit 93f02f9 --- .../generated-content/2019/08/22/beam-2.15.0.html | 83 + website/generated-content/blog/index.html | 15 +++ website/generated-content/feed.xml | 133 ++--- website/generated-content/index.html | 10 +- 4 files changed, 196 insertions(+), 45 deletions(-) diff --git a/website/generated-content/2019/08/22/beam-2.15.0.html b/website/generated-content/2019/08/22/beam-2.15.0.html new file mode 100644 index 000..975787f --- /dev/null +++ b/website/generated-content/2019/08/22/beam-2.15.0.html @@ -0,0 +1,83 @@ + + +layout: post +title: “Apache Beam 2.15.0” +date: 2019-08-22 00:00:01 -0800 +Date above corrected but keep the old URL: +permalink: /blog/2019/08/22/beam-2.15.0.html +excerpt_separator: +categories: blog +authors: +- yifanzou + + + + +We are happy to present the new 2.15.0 release of Beam. This release includes both improvements and new functionality. +See the download page for this release. +For more information on changes in 2.15.0, check out the +https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345489;>detailed release notes. + +Highlights + + + Vendored Guava was upgraded to version 26.0. + + +I/Os + + + Add AvroIO.sink for IndexedRecord (FileIO compatible). + + +New Features / Improvements + + + Support ParquetTable in SQL. + Add hot key detection to Dataflow Runner. + Enable throughput collection in BigQuery performance tests. + Support schemas in the JDBC sink. + + +Bugfixes + + + Various bug fixes and performance improvements. + + +Known Issues + + + https://issues.apache.org/jira/browse/BEAM-7616;>BEAM-7616 urlopen calls may get stuck. + + +List of Contributors + +According to git shortlog, the following people contributed to the 2.15.0 release. Thank you to all contributors! + +Ahmet Altay, Alexey Romanenko, Alex Goos, Alireza Samadian, Andrew Pilloud, Ankur Goenka, +Anton Kedin, Aryan Naraghi, Bartok Jozsef, bmv126, B M VISHWAS, Boyuan Zhang, +Brian Hulette, brucearctor, Cade Markegard, Cam Mach, Chad Dombrova, +Chaim Turkel, Chamikara Jayalath, Charith Ellawala, Claire McGinty, Craig Chambers, +Daniel Oliveira, David Cavazos, David Moravek, Dominic Mitchell, Dustin Rhodes, +Etienne Chauchot, Filipe Regadas, Gleb Kanterov, Gunnar Schulze, Hannah Jiang, +Heejong Lee, Henry Suryawirawan, Ismaël Mejía, Ivo Galic, Jan Lukavský, +Jawad, Juta, Juta Staes, Kai Jiang, Kamil Wasilewski, Kasia Kucharczyk, +Kenneth Jung, Kenneth Knowles, Kyle Weaver, Lily Li, Logan HAUSPIE, lostluck, +Łukasz Gajowy, Luke Cwik, Mark Liu, Matt Helm, Maximilian Michels, +Michael Luckey, Mikhail Gryzykhin, Neville Li, Nicholas Rucci, pabloem, +Pablo Estrada, Paul King, Paul Suganthan, Raheel Khan, Rakesh Kumar, +Reza Rokni, Robert Bradshaw, Robert Burke, rosetn, Rui Wang, Ryan Skraba, RyanSkraba, +Sahith Nallapareddy, Sam Rohde, Sam Whittle, Steve Niemitz, Tanay Tummalapalli, Thomas Weise, +Tianyang Hu, ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Wout Scheepers, +yanzhi, Yekut, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, Yunqing Zhou diff --git a/website/generated-content/blog/index.html b/website/generated-content/blog/index.html index e2f259b..b6d4f5c 100644 --- a/website/generated-content/blog/index.html +++ b/website/generated-content/blog/index.html @@ -169,6 +169,21 @@ limitations under the License. This is the blog for the Apache Beam project. This blog contains news and updates for the project. +Beam 2.15.0 +Aug 22, 2019 + + + + + + + +Read more + + + + + Apache Beam 2.14.0 Jul 31, 2019 • Anton Kedin Ahmet Altay diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index 810f038..0c155d9 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -20,6 +20,99 @@ Jekyll v3.2.0 +Beam 2.15.0 +hr / + +playout: post +title: “Apache Beam 2.15.0” +date: 2019-08-22 00:00:01 -0800/p +h1 id=date-above-corrected-but-keep-the-old-urlDate above corrected but keep the old URL:/h1 +ppermalink: /blog/2019/08/22/beam-2.15.0.html +excerpt_separator: !--more-- +categories: blog +authors: +- yifanzou/p + +hr / +!-- +Licensed under the Apache License, Version 2.0 (the License); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +
[beam] branch asf-site updated: Publishing website 2019/08/23 05:58:13 at commit ba7e16c
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 9ef131d Publishing website 2019/08/23 05:58:13 at commit ba7e16c 9ef131d is described below commit 9ef131df514c5052e32f57b929d6ad611fa57825 Author: jenkins AuthorDate: Fri Aug 23 05:58:14 2019 + Publishing website 2019/08/23 05:58:13 at commit ba7e16c --- website/generated-content/.htaccess| 2 +- .../blog/2016/10/20/test-stream.html | 2 +- .../blog/2017/08/16/splittable-do-fn.html | 10 +- .../08/20/review-input-streaming-connectors.html | 22 +++--- .../contribute/ptransform-style-guide/index.html | 4 ++-- .../sql/calcite/aggregate-functions/index.html | 4 ++-- .../dsls/sql/calcite/data-types/index.html | 4 ++-- .../dsls/sql/calcite/lexical/index.html| 4 ++-- .../dsls/sql/calcite/overview/index.html | 4 ++-- .../dsls/sql/calcite/query-syntax/index.html | 4 ++-- .../dsls/sql/calcite/scalar-functions/index.html | 4 ++-- .../extensions/create-external-table/index.html| 4 ++-- .../dsls/sql/extensions/joins/index.html | 4 ++-- .../dsls/sql/extensions/set/index.html | 4 ++-- .../extensions/user-defined-functions/index.html | 4 ++-- .../extensions/windowing-and-triggering/index.html | 4 ++-- .../documentation/dsls/sql/overview/index.html | 8 .../documentation/dsls/sql/shell/index.html| 4 ++-- .../documentation/dsls/sql/walkthrough/index.html | 10 +- .../io/built-in/google-bigquery/index.html | 14 +++--- .../documentation/io/built-in/parquet/index.html | 6 +++--- .../io/developing-io-python/index.html | 6 +++--- .../pipelines/test-your-pipeline/index.html| 4 ++-- .../documentation/programming-guide/index.html | 16 .../documentation/runners/dataflow/index.html | 6 +++--- .../documentation/runners/direct/index.html| 14 +++--- .../documentation/runners/flink/index.html | 4 ++-- .../documentation/runners/gearpump/index.html | 4 ++-- .../documentation/runners/jstorm/index.html| 2 +- .../documentation/runners/mapreduce/index.html | 2 +- .../documentation/runners/samza/index.html | 2 +- .../documentation/runners/spark/index.html | 2 +- .../sdks/feature-comparison/index.html | 4 ++-- .../documentation/sdks/go/index.html | 4 ++-- .../sdks/java-dependencies/index.html | 4 ++-- .../documentation/sdks/java-extensions/index.html | 4 ++-- .../documentation/sdks/java-thirdparty/index.html | 4 ++-- .../documentation/sdks/java/euphoria/index.html| 6 +++--- .../documentation/sdks/java/index.html | 4 ++-- .../sdks/java/testing/nexmark/index.html | 8 .../sdks/python-dependencies/index.html| 4 ++-- .../sdks/python-pipeline-dependencies/index.html | 4 ++-- .../documentation/sdks/python-streaming/index.html | 4 ++-- .../sdks/python-type-safety/index.html | 4 ++-- .../documentation/sdks/python/index.html | 4 ++-- .../get-started/downloads/index.html | 18 +- .../get-started/quickstart-java/index.html | 4 ++-- .../get-started/wordcount-example/index.html | 4 ++-- website/generated-content/index.html | 4 ++-- 49 files changed, 142 insertions(+), 134 deletions(-) diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess index 48b5559..f3bf7b7 100644 --- a/website/generated-content/.htaccess +++ b/website/generated-content/.htaccess @@ -21,4 +21,4 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301] # The following redirect maintains the previously supported URLs. RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2; # Keep this updated to point to the current release. -RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.14.0$2; +RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.15.0$2; diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html index bc8bd6f..b8f0e03 100644 --- a/website/generated-content/blog/2016/10/20/test-stream.html +++ b/website/generated-content/blog/2016/10/20/test-stream.html @@ -240,7 +240,7 @@ be controlled within a test. Writing Deterministic Tests to Emulate Nondeterminism The Beam testing infrastructure provides the -https://beam.apache.org/releases/javadoc/2.14.0/org/apache/beam/sdk/testing/PAssert.html;>PAssert +htt
[beam] branch asf-site updated: Publishing website 2019/08/21 17:03:48 at commit 5994182
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0bac031 Publishing website 2019/08/21 17:03:48 at commit 5994182
0bac031 is described below
commit 0bac031a938306c364b3382f2d91cf2a6b9f0e25
Author: jenkins
AuthorDate: Wed Aug 21 17:03:49 2019 +
Publishing website 2019/08/21 17:03:48 at commit 5994182
---
.../documentation/programming-guide/index.html | 50 +-
1 file changed, 48 insertions(+), 2 deletions(-)
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 45d2a2f..ad20d7d 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -2321,20 +2321,37 @@ together.
-4.5.3. Accessing
additional parameters in your DoFn
+4.5.3. Accessing additional parameters in your
DoFn
In addition to the element and the OutputReceiver, Beam will populate other
parameters to your DoFn’s @ProcessElement method.
Any combination of these parameters can be added to your process method in any
order.
+In addition to the element, Beam will populate other
parameters to your DoFn’s process method.
+Any combination of these parameters can be added to your process method in any
order.
+
Timestamp:
To access the timestamp of an input element, add a parameter annotated with
@Timestamp of type Instant. For example:
+Timestamp:
+To access the timestamp of an input element, add a keyword parameter default
to DoFn.TimestampParam. For example:
+
.of(new DoFnString,
String() {
public void processElement(@Element String word, @Timestamp Instant timestamp) {
}})
+import apache_beam as beam
+
+class ProcessRecord(beam.DoFn):
+
+ def process(self, element, timestamp=beam.DoFn.TimestampParam):
+ # access timestamp of element.
+ pass
+
+
+
+
Window:
To access the window an input element falls into, add a parameter of the type
of the window used for the input PCollection.
If the parameter is a window type (a subclass of BoundedWindow) that does not match the input
PCollection, then an error
@@ -2342,11 +2359,17 @@ will be raised. If an element falls in multiple windows
(for example, this will
@ProcessElement method will be invoked
multiple time for the element, once for each window. For example, when fixed
windows
are being used, the window is of type IntervalWindow.
+Window:
+To access the window an input element falls into, add a keyword parameter
default to DoFn.WindowParam.
+If an element falls in multiple windows (for example, this will happen when
using SlidingWindows), then the
+process method will be invoked multiple
time for the element, once for each window.
+
.of(new DoFnString,
String() {
public void processElement(@Element String word, IntervalWindow window) {
}})
+
import apache_beam as beam
class ProcessRecord(beam.DoFn):
@@ -2357,16 +2380,33 @@ are being used, the window is of type IntervalWi
-PaneInfo:
+
+PaneInfo:
When triggers are used, Beam provides a PaneInfo object that contains information
about the current firing. Using PaneInfo
you can determine whether this is an early or a late firing, and how many
times this window has already fired for this key.
+PaneInfo:
+When triggers are used, Beam provides a DoFn.PaneInfoParam object that contains
information about the current firing. Using DoFn.PaneInfoParam
+you can determine whether this is an early or a late firing, and how many
times this window has already fired for this key.
+This feature implementation in python sdk is not fully completed, see more at
https://issues.apache.org/jira/browse/BEAM-3759;>BEAM-3759.
+
.of(new DoFnString,
String() {
public void processElement(@Element String word, PaneInfo
paneInfo) {
}})
+import apache_beam as beam
+
+class ProcessRecord(beam.DoFn):
+
+ def process(self, element, pane_info=beam.DoFn.PaneInfoParam):
+ # access pane info e.g pane_info.is_first,
pane_info.is_last, pane_info.timing
+ pass
+
+
+
+
PipelineOptions:
The PipelineOptions for the current
pipeline can always be accessed in a process method by adding it as a
parameter:
.of(new DoFnString,
String() {
@@ -2380,6 +2420,12 @@ The PipelineOptions for the current pipel
a parameter of type TimeDomain which
tells whether the timer is based on event time or processing time.
Timers are explained in more detail in the
Timely (and Stateful)
Processing with Apache Beam blog post.
+
+Timer and State:
+In addition to aforementioned parameters, user defined Timer and State
parameters can be used in a Stateful DoFn.
+Timers and States are explained in more detail in the
+Timely (and State
[beam] branch asf-site updated: Publishing website 2019/08/20 18:32:00 at commit c0f456f
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 0ecee9e Publishing website 2019/08/20 18:32:00 at commit c0f456f 0ecee9e is described below commit 0ecee9e52e9b10a9d9e7d8735641e05d14db355d Author: jenkins AuthorDate: Tue Aug 20 18:32:01 2019 + Publishing website 2019/08/20 18:32:00 at commit c0f456f --- website/generated-content/blog/2019/07/31/beam-2.14.0.html | 2 +- website/generated-content/feed.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/blog/2019/07/31/beam-2.14.0.html b/website/generated-content/blog/2019/07/31/beam-2.14.0.html index e25780e..83c7b11 100644 --- a/website/generated-content/blog/2019/07/31/beam-2.14.0.html +++ b/website/generated-content/blog/2019/07/31/beam-2.14.0.html @@ -205,7 +205,7 @@ For more information on changes in 2.14.0, check out the Schema support added to BigQuery reads. (Java) (https://issues.apache.org/jira/browse/BEAM-6673;>BEAM-6673) - Schema support added to JDBC source and sink. (Java) (https://issues.apache.org/jira/browse/BEAM-6674;>BEAM-6674, https://issues.apache.org/jira/browse/BEAM-6675;>BEAM-6675) + Schema support added to JDBC source. (Java) (https://issues.apache.org/jira/browse/BEAM-6674;>BEAM-6674) BigQuery support for bytes is fixed. (Python 3) (https://issues.apache.org/jira/browse/BEAM-6769;>BEAM-6769) Added DynamoDB IO. (Java) (https://issues.apache.org/jira/browse/BEAM-7043;>BEAM-7043) Added support unbounded reads with HCatalogIO (Java) (https://issues.apache.org/jira/browse/BEAM-7450;>BEAM-7450) diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index 77793c5..810f038 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -52,7 +52,7 @@ For more information on changes in 2.14.0, check out the ul liSchema support added to BigQuery reads. (Java) (a href=https://issues.apache.org/jira/browse/BEAM-6673BEAM-6673/a;)/li - liSchema support added to JDBC source and sink. (Java) (a href=https://issues.apache.org/jira/browse/BEAM-6674BEAM-6674/a;, a href=https://issues.apache.org/jira/browse/BEAM-6675BEAM-6675/a;)/li + liSchema support added to JDBC source. (Java) (a href=https://issues.apache.org/jira/browse/BEAM-6674BEAM-6674/a;)/li liBigQuery support for code class=highlighter-rougebytes/code is fixed. (Python 3) (a href=https://issues.apache.org/jira/browse/BEAM-6769BEAM-6769/a;)/li liAdded DynamoDB IO. (Java) (a href=https://issues.apache.org/jira/browse/BEAM-7043BEAM-7043/a;)/li liAdded support unbounded reads with HCatalogIO (Java) (a href=https://issues.apache.org/jira/browse/BEAM-7450BEAM-7450/a;)/li
[beam] branch asf-site updated: Publishing website 2019/08/20 08:19:16 at commit 5b6a8b6
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new b5674b9 Publishing website 2019/08/20 08:19:16 at commit 5b6a8b6 b5674b9 is described below commit b5674b9764dce1703cc24f2ed5213e1a2671cdc8 Author: jenkins AuthorDate: Tue Aug 20 08:19:16 2019 + Publishing website 2019/08/20 08:19:16 at commit 5b6a8b6 --- .../documentation/execution-model/index.html | 10 ++ website/generated-content/documentation/index.html | 10 ++ .../io/built-in/google-bigquery/index.html | 10 ++ .../documentation/io/built-in/hadoop/index.html| 10 ++ .../documentation/io/built-in/hcatalog/index.html | 10 ++ .../documentation/io/built-in/index.html | 10 ++ .../documentation/io/built-in/parquet/index.html | 10 ++ .../documentation/io/developing-io-java/index.html | 10 ++ .../io/developing-io-overview/index.html | 10 ++ .../io/developing-io-python/index.html | 10 ++ .../documentation/io/testing/index.html| 10 ++ .../patterns/custom-io-patterns/index.html | 10 ++ .../patterns/file-processing-patterns/index.html | 10 ++ .../documentation/patterns/overview/index.html | 10 ++ .../patterns/pipeline-option-patterns/index.html | 10 ++ .../patterns/side-input-patterns/index.html| 10 ++ .../pipelines/create-your-pipeline/index.html | 10 ++ .../pipelines/design-your-pipeline/index.html | 10 ++ .../pipelines/test-your-pipeline/index.html| 10 ++ .../documentation/programming-guide/index.html | 164 + .../resources/learning-resources/index.html| 10 ++ .../resources/videos-and-podcasts/index.html | 10 ++ .../documentation/runners/jstorm/index.html| 10 ++ .../aggregation/approximatequantiles/index.html| 10 ++ .../java/aggregation/approximateunique/index.html | 10 ++ .../java/aggregation/cogroupbykey/index.html | 10 ++ .../transforms/java/aggregation/combine/index.html | 10 ++ .../java/aggregation/combinewithcontext/index.html | 10 ++ .../transforms/java/aggregation/count/index.html | 10 ++ .../java/aggregation/distinct/index.html | 10 ++ .../java/aggregation/groupbykey/index.html | 10 ++ .../java/aggregation/groupintobatches/index.html | 10 ++ .../transforms/java/aggregation/latest/index.html | 10 ++ .../transforms/java/aggregation/max/index.html | 10 ++ .../transforms/java/aggregation/mean/index.html| 10 ++ .../transforms/java/aggregation/min/index.html | 10 ++ .../transforms/java/aggregation/sample/index.html | 10 ++ .../transforms/java/aggregation/sum/index.html | 10 ++ .../transforms/java/aggregation/top/index.html | 10 ++ .../transforms/java/elementwise/filter/index.html | 10 ++ .../java/elementwise/flatmapelements/index.html| 10 ++ .../transforms/java/elementwise/keys/index.html| 10 ++ .../transforms/java/elementwise/kvswap/index.html | 10 ++ .../java/elementwise/mapelements/index.html| 10 ++ .../transforms/java/elementwise/pardo/index.html | 10 ++ .../java/elementwise/partition/index.html | 10 ++ .../transforms/java/elementwise/regex/index.html | 10 ++ .../transforms/java/elementwise/reify/index.html | 10 ++ .../java/elementwise/tostring/index.html | 10 ++ .../transforms/java/elementwise/values/index.html | 10 ++ .../java/elementwise/withkeys/index.html | 10 ++ .../java/elementwise/withtimestamps/index.html | 10 ++ .../transforms/java/other/create/index.html| 10 ++ .../transforms/java/other/flatten/index.html | 10 ++ .../transforms/java/other/passert/index.html | 10 ++ .../transforms/java/other/view/index.html | 10 ++ .../transforms/java/other/window/index.html| 10 ++ .../transforms/java/overview/index.html| 10 ++ .../aggregation/approximatequantiles/index.html| 10 ++ .../aggregation/approximateunique/index.html | 10 ++ .../python/aggregation/cogroupbykey/index.html | 10 ++ .../python/aggregation/combineglobally/index.html | 10 ++ .../aggregation/combinewithcontext/index.html | 10 ++ .../transforms/python/aggregation/count/index.html | 10 ++ .../python/aggregation/distinct/index.html | 10 ++ .../python/aggregation/groupbykey/index.html | 10 ++ .../python/aggregation/groupintobatches/index.html | 10 ++ .../python/aggregation/latest/index.html | 10 ++ .../transforms/python/aggregation/max/index.html | 10 ++ .../transforms/python/aggregation/mean/index.html | 10 ++ .../transforms/python/aggregation/min/index.html | 10 ++ .../python/aggregation/sample/index.html | 10 ++ .../transforms/python/aggregation/sum/index.html | 10
[beam] branch asf-site updated: Publishing website 2019/08/19 12:24:49 at commit 0467ac4
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 24752a2 Publishing website 2019/08/19 12:24:49 at commit 0467ac4
24752a2 is described below
commit 24752a2954f0af41c6521f45b6e8fb0882a558c8
Author: jenkins
AuthorDate: Mon Aug 19 12:24:49 2019 +
Publishing website 2019/08/19 12:24:49 at commit 0467ac4
---
.../documentation/io/built-in/index.html | 4 +-
.../io/built-in/{ => parquet}/index.html | 253 ++---
2 files changed, 128 insertions(+), 129 deletions(-)
diff --git a/website/generated-content/documentation/io/built-in/index.html
b/website/generated-content/documentation/io/built-in/index.html
index 431a3fe..766f0c8 100644
--- a/website/generated-content/documentation/io/built-in/index.html
+++ b/website/generated-content/documentation/io/built-in/index.html
@@ -478,7 +478,7 @@ limitations under the License.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java;>TFRecordIO
https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java;>XmlIO
https://github.com/apache/beam/blob/master/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java;>TikaIO
-https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java;>ParquetIO
+ParquetIO
https://github.com/apache/beam/tree/master/sdks/java/io/kinesis;>Amazon
Kinesis
@@ -512,7 +512,7 @@ limitations under the License.
Beam Python supports Apache HDFS, Google Cloud Storage, and local
filesystems.
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py;>avroio
-https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py;>parquetio
+parquetio.py
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py;>textio
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/tfrecordio.py;>tfrecordio
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/vcfio.py;>vcfio
diff --git a/website/generated-content/documentation/io/built-in/index.html
b/website/generated-content/documentation/io/built-in/parquet/index.html
similarity index 67%
copy from website/generated-content/documentation/io/built-in/index.html
copy to website/generated-content/documentation/io/built-in/parquet/index.html
index 431a3fe..cb60328 100644
--- a/website/generated-content/documentation/io/built-in/index.html
+++ b/website/generated-content/documentation/io/built-in/parquet/index.html
@@ -28,7 +28,7 @@
- Built-in I/O Transforms
+ Apache Parquet I/O connector
https://fonts.googleapis.com/css?family=Roboto:100,300,400;
rel="stylesheet">
@@ -39,7 +39,7 @@
- https://beam.apache.org/documentation/io/built-in/; data-proofer-ignore>
+ https://beam.apache.org/documentation/io/built-in/parquet/;
data-proofer-ignore>
https://beam.apache.org/feed.xml;>
https://use.fontawesome.com/releases/v5.4.1/css/all.css;
integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz"
crossorigin="anonymous">
@@ -141,7 +141,7 @@
GitHub links will not resolve until the markdown source is
available on the master branch.
New pages would fail validation during development / PR test
automation.
-->
- https://github.com/apache/beam/edit/master/website/src/documentation/io/built-in.md;
data-proofer-ignore>
+ https://github.com/apache/beam/edit/master/website/src/documentation/io/built-in-parquet.md;
data-proofer-ignore>
@@ -436,6 +436,9 @@
+
+ Before you start
+
@@ -455,130 +458,126 @@ See the License for the specific language governing
permissions and
limitations under the License.
-->
-Built-in I/O Transforms
-
-This table contains the currently available I/O transforms.
-
-Consult the Programming Guide I/O
section for general usage instructions, and see the javadoc/pydoc for the
particular I/O transforms.
-
-
-
- Language
- File-based
- Messaging
- Database
-
-
- Java
-
-Beam Java supports Apache HDFS, Amazon S3, Google Cloud Storage, and
local filesystems.
-https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java;>FileIO
(general-purpose reading, writing, and matching of files)
-https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
-https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java
[beam] branch asf-site updated: Publishing website 2019/08/16 23:40:06 at commit 96abacb
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8118da8 Publishing website 2019/08/16 23:40:06 at commit 96abacb
8118da8 is described below
commit 8118da80d30c65547d0286d324c9445fd39a5389
Author: jenkins
AuthorDate: Fri Aug 16 23:40:06 2019 +
Publishing website 2019/08/16 23:40:06 at commit 96abacb
---
.../python/elementwise/partition/index.html| 264 -
1 file changed, 258 insertions(+), 6 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
b/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
index 1dbe6a3..508a4f0 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/partition/index.html
@@ -437,7 +437,13 @@
- Examples
+ Examples
+
+ Example 1: Partition
with a function
+ Example 2:
Partition with a lambda function
+ Example 3:
Partition with multiple arguments
+
+
Related transforms
@@ -460,11 +466,18 @@ limitations under the License.
-->
Partition
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
Separates elements in a collection into multiple output
@@ -478,11 +491,240 @@ You cannot determine the number of partitions in
mid-pipeline
See more information in the Beam Programming
Guide.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we create a pipeline with a PCollection of produce with their icon, name,
and duration.
+Then, we apply Partition in multiple
ways to split the PCollection into
multiple PCollections.
+
+Partition accepts a function that
receives the number of partitions,
+and returns the index of the desired partition for the element.
+The number of partitions passed must be a positive integer,
+and it must return an integer in the range 0 to num_partitions-1.
+
+Example 1: Partition with a
function
+
+In the following example, we have a known list of durations.
+We partition the PCollection into one
PCollection for every duration type.
+
+import apache_beam as beam
+
+durations = ['annual', 'biennial', 'perennial']
+
+def by_duration(plant, num_partitions):
+ return durations.index(plant['duration'])
+
+with beam.Pipeline()
as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ {'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' beam.Partition(by_duration, len(durations))
+ )
+ _ = (
+ annuals
+ | 'Annuals' beam.Map(lambda x: print('annual: '
+ str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' beam.Map(lambda x: print('biennial:
' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' beam.Map(lambda x: print('perennial:
' + str(x)))
+ )
+
+
+
+Output PCollections:
+
+annuals = [
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+]
+biennials = [
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+]
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2: Partition with
a lambda function
+
+We can also use lambda functions to simplify Example 1.
+
+import apache_beam as beam
+
+durations = ['annual', 'biennial', 'perennial']
+
+with beam.Pipeline()
as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ {'icon': '', 'name': 'Strawberry
[beam] branch asf-site updated: Publishing website 2019/08/16 19:34:21 at commit 442fbcd
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1d609a8 Publishing website 2019/08/16 19:34:21 at commit 442fbcd
1d609a8 is described below
commit 1d609a8e59a6b4eb254fb797ffd5bc1353cc
Author: jenkins
AuthorDate: Fri Aug 16 19:34:21 2019 +
Publishing website 2019/08/16 19:34:21 at commit 442fbcd
---
.../contribute/design-documents/index.html | 523 +
1 file changed, 8 insertions(+), 515 deletions(-)
diff --git a/website/generated-content/contribute/design-documents/index.html
b/website/generated-content/contribute/design-documents/index.html
index 9a39c42..a05af45 100644
--- a/website/generated-content/contribute/design-documents/index.html
+++ b/website/generated-content/contribute/design-documents/index.html
@@ -1,517 +1,10 @@
-
-
-
-
-
-
-
-
-
- Beam Design Documents
-
- https://fonts.googleapis.com/css?family=Roboto:100,300,400;
rel="stylesheet">
-
- https://code.jquery.com/jquery-2.2.4.min.js"</a>;>
-
-
-
-
-
- https://beam.apache.org/contribute/design-documents/; data-proofer-ignore>
-
- https://beam.apache.org/feed.xml;>
- https://use.fontawesome.com/releases/v5.4.1/css/all.css;
integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz"
crossorigin="anonymous">
-
-
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
-
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-ga('create', 'UA-73650088-1', 'auto');
-ga('send', 'pageview');
-
-
-
-
-
-
-
-
-
-Toggle navigation
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Get Started
-
-
- Documentation
-
-
- Languages
-
-
- RUNNERS
-
-
- Roadmap
-
-
- Contribute
-
-
- Community
-
-Blog
-
-
-
-
-
- (function() {
-var cx = '012923275103528129024:4emlchv9wzi';
-var gcse = document.createElement('script');
-gcse.type = 'text/javascript';
-gcse.async = true;
-gcse.src = '<a rel="nofollow" href="https://cse.google.com/cse.js?cx=">https://cse.google.com/cse.js?cx=</a>' + cx;
-var s = document.getElementsByTagName('script')[0];
-s.parentNode.insertBefore(gcse, s);
- })();
-
-
-
-
-
- https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache
Logo" style="height:20px;">
-
-http://www.apache.org/;>ASF Homepage
-http://www.apache.org/licenses/;>License
-http://www.apache.org/security/;>Security
-http://www.apache.org/foundation/thanks.html;>Thanks
-http://www.apache.org/foundation/sponsorship.html;>Sponsorship
-https://www.apache.org/foundation/policies/conduct;>Code of
Conduct
-
-
-
-
- https://github.com/apache/beam/edit/master/website/src/contribute/design-documents.md;
data-proofer-ignore>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Contribute
-Get started contributing
-Get Help
-
- Technical Docs
-
-
-https://cwiki.apache.org/confluence/display/BEAM/Contribution+Testing+Guide;>Testing
guide
-Pre-commit Slowness
Triage
-PTransform style
guide
-Runner authoring guide
-Design documents
-Dependencies guide
-Feature branches
-
-
-
- Policies
-
-Pre-commit test
policies
-Post-commit test
policies
-
-
-
- Committers
-
-Become a committer
-Committer guide
-
- Release guide
-
-
-Overview
-Prepare
-Create
Branch
-Verify
Branch
-Build
Candidate
-Triage
JIRAs
-Prepare
Docs
-Validate
Candidate
-Finalize
-Post-release
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Documents by category
-
- Project Incubation (2016)
- Beam Model
- IO / Filesystem
- Metrics
- Runners
- SQL / Schema
- Portability
- Build / Testing
- Python
- Go
-
-
[beam] branch asf-site updated: Publishing website 2019/08/16 10:33:48 at commit b134e67
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c80924b Publishing website 2019/08/16 10:33:48 at commit b134e67 c80924b is described below commit c80924b1cd205a3f746b9c02a4c228597e0a37aa Author: jenkins AuthorDate: Fri Aug 16 10:33:49 2019 + Publishing website 2019/08/16 10:33:48 at commit b134e67 --- website/generated-content/roadmap/portability/index.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index 01737d2..cde03f4 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -402,10 +402,10 @@ for details. Run once to build the SDK harness container: ./gradlew :sdks:python:container:docker Start the Flink portable JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow - In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK + In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK -To run the pipeline in streaming mode: ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming +To run the pipeline in streaming mode: ./gradlew portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming Please see the Flink Runner page for more information on how to run portable pipelines on top of Flink. @@ -417,7 +417,7 @@ how to run portable pipelines on top of Flink. Run once to build the SDK harness container: ./gradlew :sdks:python:container:docker Start the Spark portable JobService endpoint: ./gradlew :runners:spark:job-server:runShadow - In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK + In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK Python streaming mode is not yet supported on Spark.
[beam] branch asf-site updated: Publishing website 2019/08/15 19:05:15 at commit d93f188
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cee5a9c Publishing website 2019/08/15 19:05:15 at commit d93f188
cee5a9c is described below
commit cee5a9c10db27afc34a092f36439883630d3e1b5
Author: jenkins
AuthorDate: Thu Aug 15 19:05:15 2019 +
Publishing website 2019/08/15 19:05:15 at commit d93f188
---
.../transforms/python/elementwise/keys/index.html | 78 +++-
.../python/elementwise/kvswap/index.html | 73 +++-
.../python/elementwise/withtimestamps/index.html | 207 -
3 files changed, 341 insertions(+), 17 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
b/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
index 5f1dce7..fbfc765 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/keys/index.html
@@ -437,7 +437,7 @@
- Examples
+ Example
Related transforms
@@ -460,24 +460,84 @@ limitations under the License.
-->
Keys
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-Takes a collection of key-value pairs and returns the key of each element.
-
-Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+
+Takes a collection of key-value pairs and returns the key of each
element.
+
+Example
+
+In the following example, we create a pipeline with a PCollection of key-value pairs.
+Then, we apply Keys to extract the keys
and discard the values.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ icons = (
+ pipeline
+ | 'Garden plants' beam.Create([
+ ('', 'Strawberry'),
+ ('凌', 'Carrot'),
+ ('', 'Eggplant'),
+ ('', 'Tomato'),
+ ('凜', 'Potato'),
+ ])
+ | 'Keys' beam.Keys()
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after Keys:
+
+icons = [
+'',
+'凌',
+'',
+'',
+'凜',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
Related transforms
+
KvSwap
swaps the key and value of each element.
Values
for extracting the value of each element.
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys;>
+ https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
+ Pydoc
+
+
+
+
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
b/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
index 72bd812..059486a 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/kvswap/index.html
@@ -460,25 +460,86 @@ limitations under the License.
-->
Kvswap
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-Takes a collection of key-value pairs and returns a collection of key-value
pairs
+
+
+Takes a collection of key-value pairs and returns a collection of key-value
pairs
which has each key and value swapped.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following example, we create a pipeline with a PCollection of key-value pairs.
+Then, we apply KvSwap to swap the keys
and values.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ p
[beam] branch asf-site updated: Publishing website 2019/08/15 18:57:55 at commit ab37b0f
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8fd085e Publishing website 2019/08/15 18:57:55 at commit ab37b0f
8fd085e is described below
commit 8fd085e13f3e0497011e139f175214c9f0ba7e3a
Author: jenkins
AuthorDate: Thu Aug 15 18:57:55 2019 +
Publishing website 2019/08/15 18:57:55 at commit ab37b0f
---
.../transforms/python/elementwise/map/index.html | 451 -
1 file changed, 443 insertions(+), 8 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/map/index.html
b/website/generated-content/documentation/transforms/python/elementwise/map/index.html
index e245b76..46129d7 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/map/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/map/index.html
@@ -437,7 +437,18 @@
- Examples
+ Examples
+
+ Example 1: Map
with a predefined function
+ Example 2: Map with a
function
+ Example 3: Map with
a lambda function
+ Example 4: Map with
multiple arguments
+ Example 5:
MapTuple for key-value pairs
+ Example 6:
Map with side inputs as singletons
+ Example 7:
Map with side inputs as iterators
+ Example 8:
Map with side inputs as dictionaries
+
+
Related transforms
@@ -460,28 +471,452 @@ limitations under the License.
-->
Map
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-Applies a simple 1-to-1 mapping function over each element in the
collection.
+
+
+Applies a simple 1-to-1 mapping function over each element in the
collection.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we create a pipeline with a PCollection of produce with their icon, name,
and duration.
+Then, we apply Map in multiple ways to
transform every element in the PCollection.
+
+Map accepts a function that returns
a single element for every input element in the PCollection.
+
+Example 1: Map with a
predefined function
+
+We use the function str.strip which
takes a single str element and outputs a
str.
+It strips the input element’s whitespaces, including newlines and tabs.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ ' Strawberry \n',
+ ' 凌Carrot \n',
+ ' Eggplant \n',
+ ' Tomato \n',
+ ' 凜Potato \n',
+ ])
+ | 'Strip' beam.Map(str.strip)
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after Map:
+
+plants = [
+'Strawberry',
+'凌Carrot',
+'Eggplant',
+'Tomato',
+'凜Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2: Map with a function
+
+We define a function strip_header_and_newline which strips any
'#', '
', and '\n' characters from each
element.
+
+import apache_beam as beam
+
+def strip_header_and_newline(text):
+ return text.strip('# \n')
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ '# Strawberry\n',
+ '# 凌Carrot\n',
+ '# Eggplant\n',
+ '# Tomato\n',
+ '# 凜Potato\n',
+ ])
+ | 'Strip header' beam.Map(strip_header_and_newline)
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after Map:
+
+plants = [
+'Strawberry',
+'凌Carrot',
+'Eggplant',
+'Tomato',
+'凜Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 3: Map with a lambda
function
+
+We can also use lambda functions to simplify Example 2.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam
[beam] branch asf-site updated: Publishing website 2019/08/15 17:09:06 at commit ab15bee
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 275cc51 Publishing website 2019/08/15 17:09:06 at commit ab15bee
275cc51 is described below
commit 275cc51cbef893183ee6a6b744ccc6cec60b7d6f
Author: jenkins
AuthorDate: Thu Aug 15 17:09:06 2019 +
Publishing website 2019/08/15 17:09:06 at commit ab15bee
---
.../contribute/release-guide/create-branch/index.html| 12 ++--
.../contribute/release-guide/prepare/index.html | 16 +---
.../release-guide/validate-candidate/index.html | 2 +-
.../contribute/release-guide/verify-branch/index.html| 6 +++---
4 files changed, 19 insertions(+), 17 deletions(-)
diff --git
a/website/generated-content/contribute/release-guide/create-branch/index.html
b/website/generated-content/contribute/release-guide/create-branch/index.html
index dcf64a4..9c6d7a5 100644
---
a/website/generated-content/contribute/release-guide/create-branch/index.html
+++
b/website/generated-content/contribute/release-guide/create-branch/index.html
@@ -277,12 +277,12 @@ and update version information on the original branch.
Script: https://github.com/apache/beam/blob/master/release/src/main/scripts/cut_release_branch.sh;>cut_release_branch.sh
Usage
-# Cut a release branch
-./beam/release/src/main/scripts/cut_release_branch.sh \
---release=${RELEASE_VERSION} \
---next_release=${NEXT_VERSION}
+# Cut a
release branch
+./beam/release/src/main/scripts/cut_release_branch.sh \
+--release=${RELEASE_VERSION} \
+--next_release=${NEXT_VERSION}
-# Show help page
+# Show help page
./beam/release/src/main/scripts/cut_release_branch.sh -h
@@ -361,7 +361,7 @@ so builds will be broken until a new snapshot is
available.
Script: https://github.com/apache/beam/blob/master/release/src/main/scripts/start_snapshot_build.sh;>start_snapshot_build.sh
Usage
-./beam/release/src/main/scripts/start_snapshot_build.sh
+./beam/release/src/main/scripts/start_snapshot_build.sh
diff --git
a/website/generated-content/contribute/release-guide/prepare/index.html
b/website/generated-content/contribute/release-guide/prepare/index.html
index d8d5b52..d9c2f61 100644
--- a/website/generated-content/contribute/release-guide/prepare/index.html
+++ b/website/generated-content/contribute/release-guide/prepare/index.html
@@ -318,8 +318,10 @@ Sometimes the scripts don’t handle the errors correctly
and might leave everyt
(e.g. if you fail to enter the GitHub password correctly multiple times);
- Usage
-./beam/release/src/main/scripts/preparation_before_release.sh
+
+Usage
+
+./beam/release/src/main/scripts/preparation_before_release.sh
@@ -394,11 +396,11 @@ number.
You should already have setup a local clone of apache/beam. Setting up a clone
of apache/beam-site is similar:
-$ git clone -b release-docs
https://github.com/apache/beam-site.git
-$ cd beam-site
-$ git remote add GitHub_user
g...@github.com:GitHub_user/beam-site.git
-$ git fetch --all
-$ git checkout -b my-branch origin/release-docs
+ $ git clone -b
release-docs https://github.com/apache/beam-site.git
+ $ cd beam-site
+ $ git remote add GitHub_user
g...@github.com:GitHub_user/beam-site.git
+ $ git fetch --all
+ $ git checkout -b my-branch origin/release-docs
diff --git
a/website/generated-content/contribute/release-guide/validate-candidate/index.html
b/website/generated-content/contribute/release-guide/validate-candidate/index.html
index 1c9098e..d5f40b7 100644
---
a/website/generated-content/contribute/release-guide/validate-candidate/index.html
+++
b/website/generated-content/contribute/release-guide/validate-candidate/index.html
@@ -354,7 +354,7 @@ In case of script failure, you can still run all of them
manually.
First update https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config
with correct config value (e.g. release version, rc number).
Then run
-./beam/release/src/main/scripts/run_rc_validation.sh
+./beam/release/src/main/scripts/run_rc_validation.sh
diff --git
a/website/generated-content/contribute/release-guide/verify-branch/index.html
b/website/generated-content/contribute/release-guide/verify-branch/index.html
index fac99ff..fd409d5 100644
---
a/website/generated-content/contribute/release-guide/verify-branch/index.html
+++
b/website/generated-content/contribute/release-guide/verify-branch/index.html
@@ -276,7 +276,7 @@ creation of the release candidate.
Usage
-./beam/release/src/main/scripts/verify_release_build.sh
+./beam/release/src/main/scripts/verify_release_build.sh
@@ -329,14 +329,14 @@ projects you’re interested at the mo
[beam] branch asf-site updated: Publishing website 2019/08/13 13:38:42 at commit 4b6800a
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 8430aa2 Publishing website 2019/08/13 13:38:42 at commit 4b6800a 8430aa2 is described below commit 8430aa2b26725ace32787ac88dcfc76b863cffe1 Author: jenkins AuthorDate: Tue Aug 13 13:38:42 2019 + Publishing website 2019/08/13 13:38:42 at commit 4b6800a --- website/generated-content/community/in-person/index.html | 5 + 1 file changed, 5 insertions(+) diff --git a/website/generated-content/community/in-person/index.html b/website/generated-content/community/in-person/index.html index e606438..11725cd 100644 --- a/website/generated-content/community/in-person/index.html +++ b/website/generated-content/community/in-person/index.html @@ -272,6 +272,11 @@ limitations under the License. New York City https://www.meetup.com/New-York-Apache-Beam/;>New York Apache Beam + + + + Paris + https://www.meetup.com/Paris-Apache-Beam-Meetup/;>Paris Apache Beam :
[beam] branch asf-site updated: Publishing website 2019/08/13 07:45:35 at commit 948e84e
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 3df8a65 Publishing website 2019/08/13 07:45:35 at commit 948e84e 3df8a65 is described below commit 3df8a6566b309f1eff409309a0873f1e97eda856 Author: jenkins AuthorDate: Tue Aug 13 07:45:35 2019 + Publishing website 2019/08/13 07:45:35 at commit 948e84e --- .../resources/videos-and-podcasts/index.html | 26 ++ 1 file changed, 26 insertions(+) diff --git a/website/generated-content/documentation/resources/videos-and-podcasts/index.html b/website/generated-content/documentation/resources/videos-and-podcasts/index.html index d219f1d..05d9cf3 100644 --- a/website/generated-content/documentation/resources/videos-and-podcasts/index.html +++ b/website/generated-content/documentation/resources/videos-and-podcasts/index.html @@ -455,6 +455,8 @@ Watermarks: Time and Progress in Apache Beam and Beyond Triggers in Apache Beam + Nexmark Evaluating Big Data systems with Apache Beam + Universal metrics with Apache Beam Next Steps @@ -497,6 +499,8 @@ limitations under the License. Technical Details Watermarks: Time and Progress in Apache Beam and Beyond Triggers in Apache Beam + Nexmark Evaluating Big Data systems with Apache Beam + Universal metrics with Apache Beam Next Steps @@ -583,6 +587,28 @@ limitations under the License. https://www.youtube.com/embed/E1k0B9LN46M; frameborder="0" allowfullscreen=""> +Nexmark Evaluating Big Data systems with Apache Beam + +ApacheCon, Miami, 2017 + +Presented by Etienne Chauchot and Ismaël Mejia, Apache Beam PMC members + + + Nexmark with Beam by https://www.slideshare.net/EtienneChauchot; target="_blank">Etienne Chauchot + +https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3;>Play audio podcast + +Universal metrics with Apache Beam + +ApacheCon, Montreal, 2018 + +Presented by Etienne Chauchot, Apache Beam PMC member + + + Universal metrics with Apache Beam by https://www.slideshare.net/EtienneChauchot; target="_blank">Etienne Chauchot +
[beam] branch asf-site updated: Publishing website 2019/08/13 01:56:17 at commit 0c56384
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new edc7d7f Publishing website 2019/08/13 01:56:17 at commit 0c56384
edc7d7f is described below
commit edc7d7f25c73a64dffd274d770b6e0b8f609a205
Author: jenkins
AuthorDate: Tue Aug 13 01:56:17 2019 +
Publishing website 2019/08/13 01:56:17 at commit 0c56384
---
.../python/elementwise/flatmap/index.html | 497 -
1 file changed, 490 insertions(+), 7 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
index 1e349b0..0558852 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/flatmap/index.html
@@ -437,7 +437,19 @@
- Examples
+ Examples
+
+ Example 1:
FlatMap with a predefined function
+ Example 2: FlatMap with
a function
+ Example 3:
FlatMap with a lambda function
+ Example 4: FlatMap
with a generator
+ Example 5:
FlatMapTuple for key-value pairs
+ Example 6:
FlatMap with multiple arguments
+ Example
7: FlatMap with side inputs as singletons
+ Example
8: FlatMap with side inputs as iterators
+ Example 9: FlatMap
with side inputs as dictionaries
+
+
Related transforms
@@ -460,20 +472,481 @@ limitations under the License.
-->
FlatMap
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-Applies a simple 1-to-many mapping function over each element in the
collection.
+
+
+Applies a simple 1-to-many mapping function over each element in the
collection.
The many elements are flattened into the resulting collection.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we create a pipeline with a PCollection of produce with their icon, name,
and duration.
+Then, we apply FlatMap in multiple ways
to yield zero or more elements per each input element into the resulting PCollection.
+
+FlatMap accepts a function that
returns an iterable,
+where each of the output iterable’s
elements is an element of the resulting PCollection.
+
+Example 1: FlatMap with
a predefined function
+
+We use the function str.split which
takes a single str element and outputs a
list of strs.
+This pipeline splits the input element using whitespaces, creating a list of
zero or more elements.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ 'Strawberry 凌Carrot Eggplant',
+ 'Tomato 凜Potato',
+ ])
+ | 'Split words' beam.FlatMap(str.split)
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after FlatMap:
+
+plants = [
+'Strawberry',
+'凌Carrot',
+'Eggplant',
+'Tomato',
+'凜Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2: FlatMap with a
function
+
+We define a function split_words
which splits an input str element using
the delimiter ',' and outputs a list of strs.
+
+import apache_beam as beam
+
+def split_words(text):
+ return text.split(',')
+
+with beam.Pipeline()
as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ 'Strawberry,凌Carrot,Eggplant',
+ 'Tomato,凜Potato',
+ ])
+ | 'Split words' beam.FlatMap(split_words)
+ | beam.Map(print)
+ )
+
+
+
+Output PCollection after FlatMap:
+
+plants = [
+'Strawberry',
+'凌Carrot',
+'Eggplant',
+'Tomato',
+'凜Potato',
+]
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 3: FlatMap with a
lambda function
+
[beam] branch asf-site updated: Publishing website 2019/08/13 01:51:49 at commit 1e8f8b0
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 623bee5 Publishing website 2019/08/13 01:51:49 at commit 1e8f8b0
623bee5 is described below
commit 623bee5a139824ec3c608a7116acc001ec209124
Author: jenkins
AuthorDate: Tue Aug 13 01:51:50 2019 +
Publishing website 2019/08/13 01:51:49 at commit 1e8f8b0
---
.../python/elementwise/filter/index.html | 68 +-
.../python/elementwise/values/index.html | 80 +++---
2 files changed, 136 insertions(+), 12 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index 8c58ae6..7721f27 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -490,13 +490,14 @@ on the comparison ordering of the element.
Examples
-In the following examples, we create a pipeline with a PCollection of produce their icon, name, and
duration.
+In the following examples, we create a pipeline with a PCollection of produce with their icon, name,
and duration.
Then, we apply Filter in multiple ways
to filter out produce by their duration value.
+Filter accepts a function that keeps
elements that return True, and filters
out the remaining elements.
+
Example 1: Filtering with a
function
-We define a function is_perennial
which returns True if the element’s
duration equals 'perennial', and False otherwise.
-Filter accepts this function, keeps
elements that return True, and filters
out the remaining elements.
+We define a function is_perennial
which returns True if the element’s
duration equals 'perennial', and False otherwise.
import apache_beam as beam
@@ -519,6 +520,16 @@ Then, we apply Filter in multiple ways to
+Output PCollection after Filter:
+
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
@@ -552,6 +563,16 @@ Then, we apply Filter in multiple ways to
+Output PCollection after Filter:
+
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
@@ -590,6 +611,16 @@ They are passed as additional positional arguments or
keyword arguments to the f
+Output PCollection after Filter:
+
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
@@ -631,6 +662,16 @@ We then use that value to filter out perennials.
+Output PCollection after Filter:
+
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
@@ -674,6 +715,17 @@ so it is possible to iterate over large PCollect
+Output PCollection after Filter:
+
+valid_plants = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
@@ -723,6 +775,16 @@ If the PCollection
won’t fit into memor
+Output PCollection after Filter:
+
+perennials = [
+{'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+{'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+{'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+]
+
+
+
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
diff --git
a/website/generated-con
[beam] branch asf-site updated: Publishing website 2019/08/07 22:25:46 at commit 2230cc9
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c894560 Publishing website 2019/08/07 22:25:46 at commit 2230cc9 c894560 is described below commit c8945603635481b7c4ae60c8aa03e8f5f50ef91b Author: jenkins AuthorDate: Wed Aug 7 22:25:46 2019 + Publishing website 2019/08/07 22:25:46 at commit 2230cc9 --- website/generated-content/roadmap/python-sdk/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/roadmap/python-sdk/index.html b/website/generated-content/roadmap/python-sdk/index.html index c8bdd13..8b8fc20 100644 --- a/website/generated-content/roadmap/python-sdk/index.html +++ b/website/generated-content/roadmap/python-sdk/index.html @@ -249,7 +249,7 @@ limitations under the License. Python 3 Support -Apache Beam first offered Python 3 support with 2.11.0 release, however we continue to polish some https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050;>rough edges and strengthen Beam’s Python 3 offering: +Apache Beam first offered Python 3.5 support with the 2.11.0 SDK release and added Python 3.6, Python 3.7 support with the 2.14.0 version. However, we continue to polish some https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050;>rough edges and strengthen Beam’s Python 3 offering: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245view=detail;>Kanban Board
[beam] branch asf-site updated: Publishing website 2019/08/07 19:58:41 at commit 28a4057
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 40b0a5c Publishing website 2019/08/07 19:58:41 at commit 28a4057 40b0a5c is described below commit 40b0a5c2425376779546eb33dfe26f84184c67a2 Author: jenkins AuthorDate: Wed Aug 7 19:58:41 2019 + Publishing website 2019/08/07 19:58:41 at commit 28a4057 --- .../documentation/sdks/python-pipeline-dependencies/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html index 43c2f98..d9f3041 100644 --- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html +++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html @@ -292,7 +292,7 @@ limitations under the License. When you run your pipeline locally, the packages that your pipeline depends on are available because they are installed on your local machine. However, when you want to run your pipeline remotely, you must make sure these dependencies are available on the remote machines. This tutorial shows you how to make your dependencies available to the remote workers. Each section below refers to a different source that your package may have been installed from. -Note: Remote workers used for pipeline execution typically have a standard Python 2.7 distribution installation. If your code relies only on standard Python packages, then you probably don’t need to do anything on this page. +Note: Remote workers used for pipeline execution typically have a standard Python distribution installation in a Debian-based container image. If your code relies only on standard Python packages, then you probably don’t need to do anything on this page. PyPI Dependencies
[beam] branch asf-site updated: Publishing website 2019/08/06 16:11:57 at commit dda2061
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 1bbdce4 Publishing website 2019/08/06 16:11:57 at commit dda2061 1bbdce4 is described below commit 1bbdce45a144b755a779230f9d5b648fef434a75 Author: jenkins AuthorDate: Tue Aug 6 16:11:58 2019 + Publishing website 2019/08/06 16:11:57 at commit dda2061 --- website/generated-content/get-started/quickstart-py/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html index 0f2f3f7..61014a9 100644 --- a/website/generated-content/get-started/quickstart-py/index.html +++ b/website/generated-content/get-started/quickstart-py/index.html @@ -272,7 +272,7 @@ limitations under the License. Check your Python version -The Beam SDK for Python requires Python version 2.7.x. Check that you have version 2.7.x by running: +The Beam SDK requires Python 2 users to use Python 2.7 and Python 3 users to use Python 3.5 or higher. Check your version by running: python --version
[beam] branch asf-site updated: Publishing website 2019/08/06 13:38:44 at commit a5c8ae0
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 4f361d8 Publishing website 2019/08/06 13:38:44 at commit a5c8ae0 4f361d8 is described below commit 4f361d819897edbd182d17ab0491ebef61fdf95a Author: jenkins AuthorDate: Tue Aug 6 13:38:44 2019 + Publishing website 2019/08/06 13:38:44 at commit a5c8ae0 --- .../documentation/runners/jet/index.html | 40 ++ 1 file changed, 3 insertions(+), 37 deletions(-) diff --git a/website/generated-content/documentation/runners/jet/index.html b/website/generated-content/documentation/runners/jet/index.html index 4781f5a..6570b44 100644 --- a/website/generated-content/documentation/runners/jet/index.html +++ b/website/generated-content/documentation/runners/jet/index.html @@ -201,8 +201,7 @@ Overview Running WordCount with the Hazelcast Jet Runner - Generating the Beam examples project from SNAPSHOT versions of Beam - Generating the Beam examples project from RELEASED versions of Beam + Generating the Beam examples project Running WordCount on a Local Jet Cluster Running WordCount on a Remote Jet Cluster @@ -254,41 +253,8 @@ supported capabilities of the Jet Runner. Running WordCount with the Hazelcast Jet Runner -Generating the Beam examples project from SNAPSHOT versions of Beam -Make sure that your maven config (~/.m2/settings.xml) is set up to have access to the Apache Snapshot Repository. It -should contain this: - repositories - repository -idapache.snapshots/id -nameApache Development Snapshot Repository/name - urlhttps://repository.apache.org/content/repositories/snapshots//url; -releases - enabledtrue/enabled -/releases -snapshots - enabledtrue/enabled -/snapshots - /repository -/repositories - - - -Generate the Examples Maven Project just like when the archetype is local: -$ mvn archetype:generate \ --DarchetypeGroupId=org.apache.beam \ --DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ --DarchetypeVersion=2.14.0-SNAPSHOT \ --DgroupId=org.example \ --DartifactId=word-count-beam \ --Dversion="0.1" \ --Dpackage=org.apache.beam.examples \ --DinteractiveMode=false - - - -Generating the Beam examples project from RELEASED versions of Beam - -Caution: The released Beam versions don’t contain the Jet Runner yet. The SNAPSHOT version has to be used until Jet is released. +Generating the Beam examples project +Just follow the instruction from the Java Quickstart page Running WordCount on a Local Jet Cluster Issue following command in the Beam examples project to start new Jet cluster and run the WordCount example on it.
[beam] branch asf-site updated: Publishing website 2019/08/05 16:09:25 at commit cca9e6d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new cc98825 Publishing website 2019/08/05 16:09:25 at commit cca9e6d cc98825 is described below commit cc98825c53203ee41a1a3cf1c7cfe127d89f289a Author: jenkins AuthorDate: Mon Aug 5 16:09:25 2019 + Publishing website 2019/08/05 16:09:25 at commit cca9e6d --- .../contribute/release-guide/index.html | 17 + 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content/contribute/release-guide/index.html index 601a343..b9889ac 100644 --- a/website/generated-content/contribute/release-guide/index.html +++ b/website/generated-content/contribute/release-guide/index.html @@ -786,9 +786,7 @@ so builds will be broken until a new snapshot is available. Usage -``` -./beam/release/src/main/scripts/verify_release_build.sh -``` +./beam/release/src/main/scripts/verify_release_build.sh @@ -1325,12 +1323,15 @@ Thanks everyone! Script: https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh;>run_rc_validation.sh - -Usage - -./beam/release/src/main/scripts/run_rc_validation.sh + Usage + + First update https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config;>script.config with correct config value (e.g. release version, rc number). + Then run + ./beam/release/src/main/scripts/run_rc_validation.sh - + + + Tasks included
[beam] branch asf-site updated: Publishing website 2019/08/04 19:11:09 at commit 2c1932a
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 2997845 Publishing website 2019/08/04 19:11:09 at commit 2c1932a 2997845 is described below commit 29978459ddf184fabc8f842423a037673703adc5 Author: jenkins AuthorDate: Sun Aug 4 19:11:09 2019 + Publishing website 2019/08/04 19:11:09 at commit 2c1932a --- website/generated-content/contribute/design-documents/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index 5662503..e7a2224 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -348,6 +348,7 @@ limitations under the License. Unbounded limit [https://docs.google.com/document/d/13zeTewHH9nfwhSlcE4x77WQwr1U2Z4sTiNRjOXUj2aw;>doc] Portable Beam Schemas [https://s.apache.org/beam-schemas;>doc] Cost Based Optimizer [https://docs.google.com/document/d/1DM_bcfFbIoc_vEoqQxhC7AvHBUDVCAwToC8TYGukkII;>doc1, https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY/edit#heading=h.6rlkpwwx7gvf;>doc2] + ZetaSQL as a dialect in BeamSQL [https://docs.google.com/document/d/14Yi4oEMzqS3n9-LfSNi6Q6kQpEP3gWTHzX0HxqUksdc/edit?usp=sharing;>doc] Portability
[beam] branch asf-site updated: Publishing website 2019/08/02 17:58:12 at commit c6c3bce
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 4d5e2ed Publishing website 2019/08/02 17:58:12 at commit c6c3bce 4d5e2ed is described below commit 4d5e2ed9040590b56ef7692bfac7f4524c6c2f00 Author: jenkins AuthorDate: Fri Aug 2 17:58:12 2019 + Publishing website 2019/08/02 17:58:12 at commit c6c3bce --- .../contribute/runner-guide/index.html | 28 .../documentation/io/built-in/index.html | 4 +-- .../patterns/file-processing-patterns/index.html | 2 +- .../documentation/runners/jet/index.html | 2 +- .../sdks/python-dependencies/index.html| 39 ++ .../transforms/python/other/reshuffle/index.html | 2 +- .../transforms/python/overview/index.html | 2 +- 7 files changed, 59 insertions(+), 20 deletions(-) diff --git a/website/generated-content/contribute/runner-guide/index.html b/website/generated-content/contribute/runner-guide/index.html index db0e1d3..5ef955c 100644 --- a/website/generated-content/contribute/runner-guide/index.html +++ b/website/generated-content/contribute/runner-guide/index.html @@ -574,7 +574,7 @@ match across all SDKs. The run(Pipeline) method should be asynchronous and results in a PipelineResult which generally will be a job descriptor for your data -processing engine, provides methods for checking its status, canceling it, and +processing engine, providing methods for checking its status, canceling it, and waiting for it to terminate. Implementing the Beam Primitives @@ -588,7 +588,7 @@ provided. The primitives are designed for the benefit of pipeline authors, not runner authors. Each represents a different conceptual mode of operation (external IO, element-wise, grouping, windowing, union) rather than a specific implementation -decision. The same primitive may require very different implementation based +decision. The same primitive may require a very different implementation based on how the user instantiates it. For example, a ParDo that uses state or timers may require key partitioning, a GroupByKey with speculative triggering may require a more costly or complex implementation, and Read is completely @@ -657,7 +657,7 @@ initialization is almost always equivalent and more efficient, but this hook remains for simplicity for users) ProcessElement / OnTimer - called for each element and timer activation FinishBundle - essentially “flush”; required to be called before -considering elements actually processed +considering elements as actually processed Teardown - release resources that were used across bundles; calling this can be best effort due to failures @@ -712,7 +712,7 @@ via the Fn API may manifest as another implementation A side input is a global view of a window of a PCollection. This distinguishes it from the main input, which is processed one element at a time. The SDK/user prepares a PCollection adequately, the runner materializes it, and then the -runner feeds it to the DoFn. See the +runner feeds it to the DoFn. What you will need to implement is to inspect the materialization requested for the side input, and prepare it appropriately, and corresponding interactions @@ -758,7 +758,7 @@ function. See Main design document: https://s.apache.org/beam-state;>https://s.apache.org/beam-state -When ParDo includes state and timers, its execution on your runner is usually +When a ParDo includes state and timers, its execution on your runner is usually very different. See the full details beyond those covered here. State and timers are partitioned per key and window. You may need or want to @@ -778,7 +778,7 @@ this to implement user-facing state. Main design document: https://s.apache.org/splittable-do-fn;>https://s.apache.org/splittable-do-fn Splittable DoFn is a generalization and combination of ParDo and Read. It -is per-element processing where each element the capabilities of being “split” +is per-element processing where each element has the capability of being “split” in the same ways as a BoundedSource or UnboundedSource. This enables better performance for use cases such as a PCollection of names of large files where you want to read each of them. Previously they would have to be static data in @@ -821,7 +821,7 @@ grouping. Implementing via GroupByKeyOnly + GroupAlsoByWindow The Java codebase includes support code for a particularly common way of -implement the full GroupByKey operation: first group the keys, and then group +implementing the full GroupByKey operation: first group the keys, and then group by window. For merging windows, this is essentially required, since merging is per key. @@ -868,7 +868,7 @@ inputs, or just ignore inputs and choose the end of the
[beam] branch asf-site updated: Publishing website 2019/08/01 23:24:59 at commit b14bbd4
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a88604f Publishing website 2019/08/01 23:24:59 at commit b14bbd4 a88604f is described below commit a88604f2dd9a00558a7304f548822eb8eac7f7df Author: jenkins AuthorDate: Thu Aug 1 23:24:59 2019 + Publishing website 2019/08/01 23:24:59 at commit b14bbd4 --- website/generated-content/blog/2019/07/31/beam-2.14.0.html | 2 +- website/generated-content/blog/index.html | 2 +- website/generated-content/feed.xml | 2 +- website/generated-content/get-started/downloads/index.html | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/website/generated-content/blog/2019/07/31/beam-2.14.0.html b/website/generated-content/blog/2019/07/31/beam-2.14.0.html index 758bab2..e25780e 100644 --- a/website/generated-content/blog/2019/07/31/beam-2.14.0.html +++ b/website/generated-content/blog/2019/07/31/beam-2.14.0.html @@ -189,7 +189,7 @@ limitations under the License. --> We are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality. -See the download page for this release. +See the download page for this release. For more information on changes in 2.14.0, check out the https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345431;>detailed release notes. diff --git a/website/generated-content/blog/index.html b/website/generated-content/blog/index.html index cf831e7..e2f259b 100644 --- a/website/generated-content/blog/index.html +++ b/website/generated-content/blog/index.html @@ -189,7 +189,7 @@ limitations under the License. --> We are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality. -See the download page for this release. +See the download page for this release. diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index bb27750..77793c5 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -36,7 +36,7 @@ limitations under the License. -- pWe are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality. -See the a href=/get-started/downloads/#2140-2019-07-25download page/a for this release.!--more-- +See the a href=/get-started/downloads/#2140-2019-08-01download page/a for this release.!--more-- For more information on changes in 2.14.0, check out the a href=https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527amp;version=12345431detailed release notes/a./p diff --git a/website/generated-content/get-started/downloads/index.html b/website/generated-content/get-started/downloads/index.html index d5c24bf..831393c 100644 --- a/website/generated-content/get-started/downloads/index.html +++ b/website/generated-content/get-started/downloads/index.html @@ -216,7 +216,7 @@ API stability Releases - 2.14.0 (2019-07-25) + 2.14.0 (2019-08-01) 2.13.0 (2019-05-21) 2.12.0 (2019-04-25) @@ -329,7 +329,7 @@ versions denoted 0.x.y. Releases -2.14.0 (2019-07-25) +2.14.0 (2019-08-01) Official http://www.apache.org/dyn/closer.cgi/beam/2.14.0/apache-beam-2.14.0-source-release.zip;>source code download. https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.sha512;>SHA-512. https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.asc;>signature.
[beam] branch asf-site updated: Publishing website 2019/08/01 22:34:32 at commit a866d6c
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 866e68b Publishing website 2019/08/01 22:34:32 at commit a866d6c 866e68b is described below commit 866e68b82bf689ec41cbc34d978bba43023cc9c5 Author: jenkins AuthorDate: Thu Aug 1 22:34:32 2019 + Publishing website 2019/08/01 22:34:32 at commit a866d6c --- .../2019/07/31/beam-2.14.0.html} | 316 - website/generated-content/blog/index.html | 32 +++ website/generated-content/feed.xml | 236 --- website/generated-content/index.html | 10 +- 4 files changed, 264 insertions(+), 330 deletions(-) diff --git a/website/generated-content/index.html b/website/generated-content/blog/2019/07/31/beam-2.14.0.html similarity index 56% copy from website/generated-content/index.html copy to website/generated-content/blog/2019/07/31/beam-2.14.0.html index 01b7f2d..758bab2 100644 --- a/website/generated-content/index.html +++ b/website/generated-content/blog/2019/07/31/beam-2.14.0.html @@ -28,9 +28,8 @@ - Apache Beam - + Apache Beam 2.14.0 + https://fonts.googleapis.com/css?family=Roboto:100,300,400; rel="stylesheet"> https://code.jquery.com/jquery-2.2.4.min.js"</a>;> @@ -39,7 +38,7 @@ - https://beam.apache.org/; data-proofer-ignore> + https://beam.apache.org/blog/2019/07/31/beam-2.14.0.html; data-proofer-ignore> https://beam.apache.org/feed.xml;> https://use.fontawesome.com/releases/v5.4.1/css/all.css; integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous"> @@ -53,7 +52,7 @@ - + - https://github.com/apache/beam/edit/master/website/src/index.md; data-proofer-ignore> + https://github.com/apache/beam/edit/master/website/src/_posts/2019-07-31-beam-2.14.0.md; data-proofer-ignore> @@ -151,6 +150,31 @@ + + + +http://schema.org/BlogPosting;> + + +Apache Beam 2.14.0 +Jul 31, 2019 • + Anton Kedin Ahmet Altay + + + + + + - - - - - - -Apache Beam: An advanced unified programming model - - -Implement batch and streaming data processing jobs that run on any execution engine. - - -Learn more -Try Beam -Download Beam SDK 2.14.0 - - -Java Quickstart -Python Quickstart -Go Quickstart - - - - - - -The latest from the blog - - - - - Looping timers in Apache Beam - Jun 11, 2019 - - - - Apache Beam 2.13.0 - Jun 7, 2019 - - - - Adding new Data Sources to Beam SQL CLI - Jun 4, 2019 - - - - - - - - - - -All about Apache Beam - - - - - -Unified - - -Use a single programming model for both batch and streaming use cases. - - - - - -Portable - - -Execute pipelines on multiple execution environments. - - - - - -Extensible - - -Write and share new SDKs, IO connectors, and transformation libraries. - - - - - +We are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality. +See the download page for this release. +For more information on changes in 2.14.0, check out the +https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12345431;>detailed release notes. - - - - - +Highlights - - -Works with - - - - - http://apex.apache.org;> - - - - http://flink.apache.org;> - - - - http://spark.apache.org/;> - - - - https://cloud.google.com/dataflow/;> - - - - http://gearpump.apache.org/;> - - - - http://samza.apache.org/;> - - - - + + Python 3 support is extended to Python 3.6 and 3.7; in addition to various other Python 3 https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16890504;>improvements. + Spark portable runner (batch) now https://lists.apache.org/thread.html/c43678fc24c9a1dc9f48c51c51950aedcb9bc0fd3b633d
[beam] branch asf-site updated: Publishing website 2019/08/01 20:34:47 at commit 13c4bb6
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new db213ed Publishing website 2019/08/01 20:34:47 at commit 13c4bb6 db213ed is described below commit db213ed9c3addcefef1a0ac5a72b9a2bef571be3 Author: jenkins AuthorDate: Thu Aug 1 20:34:47 2019 + Publishing website 2019/08/01 20:34:47 at commit 13c4bb6 --- website/generated-content/.htaccess| 2 +- .../blog/2016/10/20/test-stream.html | 2 +- .../blog/2017/08/16/splittable-do-fn.html | 10 +- .../08/20/review-input-streaming-connectors.html | 22 +++--- .../contribute/ptransform-style-guide/index.html | 4 ++-- .../sql/calcite/aggregate-functions/index.html | 4 ++-- .../dsls/sql/calcite/data-types/index.html | 4 ++-- .../dsls/sql/calcite/lexical/index.html| 4 ++-- .../dsls/sql/calcite/overview/index.html | 4 ++-- .../dsls/sql/calcite/query-syntax/index.html | 4 ++-- .../dsls/sql/calcite/scalar-functions/index.html | 4 ++-- .../extensions/create-external-table/index.html| 4 ++-- .../dsls/sql/extensions/joins/index.html | 4 ++-- .../dsls/sql/extensions/set/index.html | 4 ++-- .../extensions/user-defined-functions/index.html | 4 ++-- .../extensions/windowing-and-triggering/index.html | 4 ++-- .../documentation/dsls/sql/overview/index.html | 8 .../documentation/dsls/sql/shell/index.html| 4 ++-- .../documentation/dsls/sql/walkthrough/index.html | 10 +- .../io/built-in/google-bigquery/index.html | 14 +++--- .../io/developing-io-python/index.html | 6 +++--- .../pipelines/test-your-pipeline/index.html| 4 ++-- .../documentation/programming-guide/index.html | 16 .../documentation/runners/dataflow/index.html | 6 +++--- .../documentation/runners/direct/index.html| 14 +++--- .../documentation/runners/flink/index.html | 4 ++-- .../documentation/runners/gearpump/index.html | 4 ++-- .../documentation/runners/jstorm/index.html| 2 +- .../documentation/runners/mapreduce/index.html | 2 +- .../documentation/runners/samza/index.html | 2 +- .../documentation/runners/spark/index.html | 2 +- .../sdks/feature-comparison/index.html | 4 ++-- .../documentation/sdks/go/index.html | 4 ++-- .../sdks/java-dependencies/index.html | 4 ++-- .../documentation/sdks/java-extensions/index.html | 4 ++-- .../documentation/sdks/java-thirdparty/index.html | 4 ++-- .../documentation/sdks/java/euphoria/index.html| 6 +++--- .../documentation/sdks/java/index.html | 4 ++-- .../sdks/java/testing/nexmark/index.html | 8 .../sdks/python-dependencies/index.html| 4 ++-- .../sdks/python-pipeline-dependencies/index.html | 4 ++-- .../documentation/sdks/python-streaming/index.html | 4 ++-- .../sdks/python-type-safety/index.html | 4 ++-- .../documentation/sdks/python/index.html | 4 ++-- .../get-started/downloads/index.html | 20 ++-- .../get-started/quickstart-java/index.html | 4 ++-- .../get-started/wordcount-example/index.html | 4 ++-- website/generated-content/index.html | 4 ++-- 48 files changed, 140 insertions(+), 132 deletions(-) diff --git a/website/generated-content/.htaccess b/website/generated-content/.htaccess index ce639dc..48b5559 100644 --- a/website/generated-content/.htaccess +++ b/website/generated-content/.htaccess @@ -21,4 +21,4 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301] # The following redirect maintains the previously supported URLs. RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2; # Keep this updated to point to the current release. -RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.13.0$2; +RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.14.0$2; diff --git a/website/generated-content/blog/2016/10/20/test-stream.html b/website/generated-content/blog/2016/10/20/test-stream.html index 072b602..bc8bd6f 100644 --- a/website/generated-content/blog/2016/10/20/test-stream.html +++ b/website/generated-content/blog/2016/10/20/test-stream.html @@ -240,7 +240,7 @@ be controlled within a test. Writing Deterministic Tests to Emulate Nondeterminism The Beam testing infrastructure provides the -https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/testing/PAssert.html;>PAssert +https://beam.apache.org/releases/javadoc/2.14.0/org/apache/beam/s
[beam] branch asf-site updated: Publishing website 2019/07/31 22:10:43 at commit 7472f81
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6dca566 Publishing website 2019/07/31 22:10:43 at commit 7472f81
6dca566 is described below
commit 6dca5662bba8610b372df461a5aa2145b8ec00d7
Author: jenkins
AuthorDate: Wed Jul 31 22:10:44 2019 +
Publishing website 2019/07/31 22:10:43 at commit 7472f81
---
.../python/elementwise/filter/index.html | 284 -
1 file changed, 277 insertions(+), 7 deletions(-)
diff --git
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
index 9b8871c..8c58ae6 100644
---
a/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
+++
b/website/generated-content/documentation/transforms/python/elementwise/filter/index.html
@@ -437,7 +437,16 @@
- Examples
+ Examples
+
+ Example 1: Filtering
with a function
+ Example 2:
Filtering with a lambda function
+ Example 3:
Filtering with multiple arguments
+ Example 4: Filtering
with side inputs as singletons
+ Example
5: Filtering with side inputs as iterators
+ Example 6:
Filtering with side inputs as dictionaries
+
+
Related transforms
@@ -460,21 +469,272 @@ limitations under the License.
-->
Filter
-
-https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>
+
+
+localStorage.setItem('language', 'language-py')
+
+
+
+
+https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter;>
https://beam.apache.org/images/logos/sdks/python.png;
width="20px" height="20px" alt="Pydoc" />
- Pydoc
+ Pydoc
+
-
-Given a predicate, filter out all elements that don’t satisfy that predicate.
+
+
+Given a predicate, filter out all elements that don’t satisfy that
predicate.
May also be used to filter based on an inequality with a given value based
on the comparison ordering of the element.
Examples
-See https://issues.apache.org/jira/browse/BEAM-7389;>BEAM-7389
for updates.
+
+In the following examples, we create a pipeline with a PCollection of produce their icon, name, and
duration.
+Then, we apply Filter in multiple ways
to filter out produce by their duration value.
+
+Example 1: Filtering with a
function
+
+We define a function is_perennial
which returns True if the element’s
duration equals 'perennial', and False otherwise.
+Filter accepts this function, keeps
elements that return True, and filters
out the remaining elements.
+
+import apache_beam as beam
+
+def is_perennial(plant):
+ return plant['duration']
== 'perennial'
+
+with beam.Pipeline()
as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ {'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials'
beam.Filter(is_perennial)
+ | beam.Map(print)
+ )
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 2: Filtering with
a lambda function
+
+We can also use lambda functions to simplify Example 1.
+
+import apache_beam as beam
+
+with beam.Pipeline()
as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' beam.Create([
+ {'icon': '', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '凌', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '凜', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials'
beam.Filter(
+ lambda plant: plant['duration'] ==
'perennial')
+ | beam.Map(print)
+ )
+
+
+
+
+
+https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py;>
+ https://www.tensorflow.org/images/GitHub-Mark-32px.png;
width="20px" height="20px" alt="View on GitHub" />
+ View on GitHub
+
+
+
+
+
+Example 3: Filtering
[beam] branch asf-site updated: Publishing website 2019/07/31 21:39:00 at commit e4e4590
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 2ee121a Publishing website 2019/07/31 21:39:00 at commit e4e4590 2ee121a is described below commit 2ee121a9b1abfe010616a12750e0dbd0fe883968 Author: jenkins AuthorDate: Wed Jul 31 21:39:00 2019 + Publishing website 2019/07/31 21:39:00 at commit e4e4590 --- website/generated-content/contribute/design-documents/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index 15b08fd..5662503 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -347,7 +347,7 @@ limitations under the License. Beam SQL Pipeline Options [https://docs.google.com/document/d/1UTsSBuruJRfGnVOS9eXbQI6NauCD4WnSAPgA_Y0zjdk;>doc] Unbounded limit [https://docs.google.com/document/d/13zeTewHH9nfwhSlcE4x77WQwr1U2Z4sTiNRjOXUj2aw;>doc] Portable Beam Schemas [https://s.apache.org/beam-schemas;>doc] - Cost Estimation [https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY/edit#heading=h.6rlkpwwx7gvf;>doc] + Cost Based Optimizer [https://docs.google.com/document/d/1DM_bcfFbIoc_vEoqQxhC7AvHBUDVCAwToC8TYGukkII;>doc1, https://docs.google.com/document/d/1vi1PBBu5IqSy-qZl1Gk-49CcANOpbNs1UAud6LnOaiY/edit#heading=h.6rlkpwwx7gvf;>doc2] Portability
[beam] branch asf-site updated: Publishing website 2019/07/31 07:44:09 at commit 43a6430
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c8f11c2 Publishing website 2019/07/31 07:44:09 at commit 43a6430 c8f11c2 is described below commit c8f11c2873e97d208ea09f966cb5b72e36c00924 Author: jenkins AuthorDate: Wed Jul 31 07:44:10 2019 + Publishing website 2019/07/31 07:44:09 at commit 43a6430 --- website/generated-content/get-started/wordcount-example/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html index 73a8898..4c4f318 100644 --- a/website/generated-content/get-started/wordcount-example/index.html +++ b/website/generated-content/get-started/wordcount-example/index.html @@ -545,7 +545,7 @@ transform applies a function that produces exactly one output element. -| beam.Map(lambda word_count: '%s: %s' % (word_count< [...] +| beam.MapTuple(lambda word, count: '%s: %s' % [...]
[beam] branch asf-site updated: Publishing website 2019/07/30 11:16:41 at commit ccaf3fe
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 5b4c00a Publishing website 2019/07/30 11:16:41 at commit ccaf3fe 5b4c00a is described below commit 5b4c00ac4063b8c386db24c36cee5ee3983ed0e2 Author: jenkins AuthorDate: Tue Jul 30 11:16:41 2019 + Publishing website 2019/07/30 11:16:41 at commit ccaf3fe --- website/generated-content/documentation/programming-guide/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index bf83faf..f539693 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -3257,7 +3257,7 @@ data. watermark that estimates the lag time. In practice, your PCollection’s data source determines the watermark, and watermarks can be more precise or complex. -Beam’s default windowing configuration tries to determines when all data has +Beam’s default windowing configuration tries to determine when all data has arrived (based on the type of data source) and then advances the watermark past the end of the window. This default configuration does not allow late data. Triggers allow you to modify and refine the windowing strategy for
[beam] branch asf-site updated: Publishing website 2019/07/26 12:58:52 at commit a999e85
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 85cb9f7 Publishing website 2019/07/26 12:58:52 at commit a999e85
85cb9f7 is described below
commit 85cb9f7248702e4ea41747b3ee2107c27298ae4a
Author: jenkins
AuthorDate: Fri Jul 26 12:58:52 2019 +
Publishing website 2019/07/26 12:58:52 at commit a999e85
---
.../documentation/sdks/java/testing/nexmark/index.html | 3 ---
1 file changed, 3 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
b/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
index 96eb39d..b9fd92a 100644
---
a/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
+++
b/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
@@ -1008,17 +1008,14 @@ PUBSUB_TOPCI=existing pubsub topic
--zone=${ZONE}
--workerMachineType=n1-highmem-8
--stagingLocation=${STAGING_LOCATION}
---streaming=true
--sourceType=PUBSUB
--pubSubMode=PUBLISH_ONLY
--pubsubTopic=${PUBSUB_TOPIC}
--resourceNameMode=VERBATIM
--manageResources=false
---monitorJobs=false
--numEventGenerators=64
--numWorkers=16
--maxNumWorkers=16
---suite=SMOKE
--firstEventRate=10
--nextEventRate=10
--ratePeriodSec=3600
[beam] branch asf-site updated: Publishing website 2019/07/24 09:31:00 at commit 4935d0c
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d2d4607 Publishing website 2019/07/24 09:31:00 at commit 4935d0c
d2d4607 is described below
commit d2d4607eade3d63b38a16453e15103d37390adcc
Author: jenkins
AuthorDate: Wed Jul 24 09:31:00 2019 +
Publishing website 2019/07/24 09:31:00 at commit 4935d0c
---
.../documentation/sdks/python-type-safety/index.html| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/python-type-safety/index.html
b/website/generated-content/documentation/sdks/python-type-safety/index.html
index 42ad4d9..1e34dd3 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -368,10 +368,10 @@ evens = numbers | beam.ParDo(FilterEvensDoFn())
You can use type hint annotations to define generic types. The following
code specifies an input type hint that asserts the generic type T, and an output type hint that asserts the
type Tuple[int, T].
-T =
beam.typehints.TypeVariable('T')
+T =
typing.TypeVar('T')
@beam.typehints.with_input_types(T)
-@beam.typehints.with_output_types(beam.typehints.Tuple[int, T])
+@beam.typehints.with_output_types(typing.Tuple[int, T])
class MyTransform(beam.PTransform):
def expand(self, pcoll):
return pcoll | beam.Map(lambda x: (len(x), x))
@@ -484,7 +484,7 @@ totals = (
lines
| beam.Map(parse_player_and_score)
| beam.CombinePerKey(sum).with_input_types(
-beam.typehints.Tuple[Player, int]))
+typing.Tuple[Player, int]))
[beam] branch asf-site updated: Publishing website 2019/07/23 01:04:21 at commit 8bd3b50
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f0addbc Publishing website 2019/07/23 01:04:21 at commit 8bd3b50
f0addbc is described below
commit f0addbc3c42babb28fa8f15fc4622fa03fd04754
Author: jenkins
AuthorDate: Tue Jul 23 01:04:22 2019 +
Publishing website 2019/07/23 01:04:21 at commit 8bd3b50
---
.../documentation/sdks/python-type-safety/index.html| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/python-type-safety/index.html
b/website/generated-content/documentation/sdks/python-type-safety/index.html
index 1e34dd3..42ad4d9 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -368,10 +368,10 @@ evens = numbers | beam.ParDo(FilterEvensDoFn())
You can use type hint annotations to define generic types. The following
code specifies an input type hint that asserts the generic type T, and an output type hint that asserts the
type Tuple[int, T].
-T =
typing.TypeVar('T')
+T =
beam.typehints.TypeVariable('T')
@beam.typehints.with_input_types(T)
-@beam.typehints.with_output_types(typing.Tuple[int, T])
+@beam.typehints.with_output_types(beam.typehints.Tuple[int, T])
class MyTransform(beam.PTransform):
def expand(self, pcoll):
return pcoll | beam.Map(lambda x: (len(x), x))
@@ -484,7 +484,7 @@ totals = (
lines
| beam.Map(parse_player_and_score)
| beam.CombinePerKey(sum).with_input_types(
-typing.Tuple[Player, int]))
+beam.typehints.Tuple[Player, int]))
[beam] branch asf-site updated: Publishing website 2019/07/22 22:56:11 at commit 6518abf
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 7d11229 Publishing website 2019/07/22 22:56:11 at commit 6518abf 7d11229 is described below commit 7d11229ade52e89a21c3984aa5bcbe0e734ba998 Author: jenkins AuthorDate: Mon Jul 22 22:56:12 2019 + Publishing website 2019/07/22 22:56:11 at commit 6518abf --- website/generated-content/roadmap/python-sdk/index.html | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/website/generated-content/roadmap/python-sdk/index.html b/website/generated-content/roadmap/python-sdk/index.html index d1108a2..c8bdd13 100644 --- a/website/generated-content/roadmap/python-sdk/index.html +++ b/website/generated-content/roadmap/python-sdk/index.html @@ -249,18 +249,20 @@ limitations under the License. Python 3 Support -Apache Beam 2.11.0 is the first release that offers partial support for Python 3. As of 2.11.0, only Python 3.5 on Direct and Dataflow runners has been sufficiently tested, and Python 3 support remains an active work in progress. Current goal is to extend Beam codebase compatibility with Python 3.6, 3.7, address https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16789854page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16789854 [...] +Apache Beam first offered Python 3 support with 2.11.0 release, however we continue to polish some https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050;>rough edges and strengthen Beam’s Python 3 offering: - https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE;>Proposal https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245view=detail;>Kanban Board https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA;>Python 3 Conversion Quick Start Guide https://issues.apache.org/jira/browse/BEAM-1251;>Tracking Issue + https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE;>Original Proposal -Contributions are welcome! If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself. -When submitting a new PR, please tag https://github.com/aaltay;>@aaltay, https://github.com/fredo838;>@fredo838, https://github.com/Juta;>@Juta, and https://github.com/tvalentyn;>@tvalentyn. +Contributions and feedback are welcome! +If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself. When submitting a new PR, please tag https://github.com/aaltay;>@aaltay, and https://github.com/tvalentyn;>@tvalentyn. + +To report a Python3-related issue, it is best to create a subtask to https://issues.apache.org/jira/browse/BEAM-1251;>BEAM-1251 , and cc: [~altay] and [~tvalentyn] in JIRA comment. You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate.
[beam] branch asf-site updated: Publishing website 2019/07/22 10:46:13 at commit c188582
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 5d1c0d9 Publishing website 2019/07/22 10:46:13 at commit c188582
5d1c0d9 is described below
commit 5d1c0d9c5d090b134fe84b1cfb49904166a62a62
Author: jenkins
AuthorDate: Mon Jul 22 10:46:14 2019 +
Publishing website 2019/07/22 10:46:13 at commit c188582
---
.../documentation/sdks/python-type-safety/index.html| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/website/generated-content/documentation/sdks/python-type-safety/index.html
b/website/generated-content/documentation/sdks/python-type-safety/index.html
index 42ad4d9..1e34dd3 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -368,10 +368,10 @@ evens = numbers | beam.ParDo(FilterEvensDoFn())
You can use type hint annotations to define generic types. The following
code specifies an input type hint that asserts the generic type T, and an output type hint that asserts the
type Tuple[int, T].
-T =
beam.typehints.TypeVariable('T')
+T =
typing.TypeVar('T')
@beam.typehints.with_input_types(T)
-@beam.typehints.with_output_types(beam.typehints.Tuple[int, T])
+@beam.typehints.with_output_types(typing.Tuple[int, T])
class MyTransform(beam.PTransform):
def expand(self, pcoll):
return pcoll | beam.Map(lambda x: (len(x), x))
@@ -484,7 +484,7 @@ totals = (
lines
| beam.Map(parse_player_and_score)
| beam.CombinePerKey(sum).with_input_types(
-beam.typehints.Tuple[Player, int]))
+typing.Tuple[Player, int]))
[beam] branch asf-site updated: Publishing website 2019/07/20 08:37:13 at commit 23a879f
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 9b41457 Publishing website 2019/07/20 08:37:13 at commit 23a879f 9b41457 is described below commit 9b414575418a22c9fdfdc22aa651a6f57ba1b402 Author: jenkins AuthorDate: Sat Jul 20 08:37:14 2019 + Publishing website 2019/07/20 08:37:13 at commit 23a879f --- website/generated-content/documentation/runners/flink/index.html | 5 - 1 file changed, 5 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 7ab9e43..6ec3bf2 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -319,11 +319,6 @@ period. Artifact Id - =2.14.0 - 1.9.x - beam-runners-flink-1.9 - - =2.13.0 1.8.x beam-runners-flink-1.8
[beam] branch asf-site updated: Publishing website 2019/07/19 14:13:55 at commit 9589835
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new ab4df17 Publishing website 2019/07/19 14:13:55 at commit 9589835 ab4df17 is described below commit ab4df170e26d2c45a33f8a7a8cb5290110c4147b Author: jenkins AuthorDate: Fri Jul 19 14:13:56 2019 + Publishing website 2019/07/19 14:13:55 at commit 9589835 --- .../documentation/runners/flink/index.html | 23 +++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index e6fae5c..7ab9e43 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -319,6 +319,11 @@ period. Artifact Id + =2.14.0 + 1.9.x + beam-runners-flink-1.9 + + =2.13.0 1.8.x beam-runners-flink-1.8 @@ -484,9 +489,8 @@ provided with the Flink JobManager address. from apache_beam.options.pipeline_options import PipelineOptions options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"]) -p = beam.Pipeline(options) -.. -p.run() +with beam.Pipeline(options) as p: +... @@ -503,6 +507,19 @@ To run on a separate https://ci.apache.org/projects/flink/flink-docs-re 3. Submit the pipeline as above. +As of Beam 2.15.0, steps 2 and 3 can be automated in Python by using the FlinkRunner, +plus the optional flink_version and flink_master_url options if required, i.e. + + +import apache_beam as beam +from apache_beam.options.pipeline_options import PipelineOptions + +options = PipelineOptions(["--runner=FlinkRunner", "--flink_version=1.8", "--flink_master_url=localhost:8081"]) +with beam.Pipeline(options) as p: +... + + + Additional information and caveats Monitoring your job
[beam] branch asf-site updated: Publishing website 2019/07/18 20:11:21 at commit d626584
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 5dd478a Publishing website 2019/07/18 20:11:21 at commit d626584 5dd478a is described below commit 5dd478a831e4ea5275d8270cf6ea92fb19dc4116 Author: jenkins AuthorDate: Thu Jul 18 20:11:21 2019 + Publishing website 2019/07/18 20:11:21 at commit d626584 --- website/generated-content/contribute/design-documents/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/contribute/design-documents/index.html b/website/generated-content/contribute/design-documents/index.html index 40d117e..15b08fd 100644 --- a/website/generated-content/contribute/design-documents/index.html +++ b/website/generated-content/contribute/design-documents/index.html @@ -299,6 +299,7 @@ limitations under the License. [PROPOSAL] @onwindowexpiration [https://lists.apache.org/thread.html/1dab7f17c97378e665928b6cbd887dc7be93390ab26c593ee49a@%3Cdev.beam.apache.org%3E;>mail] AutoValue Coding and Row Support [https://docs.google.com/document/d/1ucoik4WzUDfilqIz3I1AuMHc1J8DE6iv7gaUCDI42BI;>doc] HyperLogLog++ Integration with Apache Beam [https://docs.google.com/document/d/1U5aXdC9lDSOqT6FPHRulp-EutYiQ9KeHpgu-19CIfEI;>doc] + Retractions [https://docs.google.com/document/d/14WRfxwk_iLUHGPty3C6ZenddPsp_d6jhmx0vuafXqmE/edit?usp=sharing;>doc] IO / Filesystem
[beam] branch asf-site updated: Publishing website 2019/07/17 00:33:55 at commit 4a7ba2d
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new e90c3ba Publishing website 2019/07/17 00:33:55 at commit 4a7ba2d e90c3ba is described below commit e90c3ba51d553e065140ea711bfd11fe628df371 Author: jenkins AuthorDate: Wed Jul 17 00:33:55 2019 + Publishing website 2019/07/17 00:33:55 at commit 4a7ba2d --- .../get-started/quickstart-py/index.html | 9 -- .../get-started/wordcount-example/index.html | 9 -- .../roadmap/portability/index.html | 37 ++ 3 files changed, 35 insertions(+), 20 deletions(-) diff --git a/website/generated-content/get-started/quickstart-py/index.html b/website/generated-content/get-started/quickstart-py/index.html index 9979577..0f2f3f7 100644 --- a/website/generated-content/get-started/quickstart-py/index.html +++ b/website/generated-content/get-started/quickstart-py/index.html @@ -414,15 +414,18 @@ environment’s directories. -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-flink for more information. -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/documentation/runners/flink/ for more information. -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Spark requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-spark for more information. diff --git a/website/generated-content/get-started/wordcount-example/index.html b/website/generated-content/get-started/wordcount-example/index.html index ed2cb21..73a8898 100644 --- a/website/generated-content/get-started/wordcount-example/index.html +++ b/website/generated-content/get-started/wordcount-example/index.html @@ -675,15 +675,18 @@ You can monitor the running job by visiting the Flink dashboard at http://fl -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-flink for more information. -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Flink requires a full download of the Beam source code. +See https://beam.apache.org/documentation/runners/flink/ for more information. -This runner is not yet available for the Python SDK. +Currently, running wordcount.py on Spark requires a full download of the Beam source code. +See https://beam.apache.org/roadmap/portability/#python-on-spark for more information. diff --git a/website/generated-content/roadmap/portability/index.html b/website/generated-content/roadmap/portability/index.html index 7e0fb1b..01737d2 100644 --- a/website/generated-content/roadmap/portability/index.html +++ b/website/generated-content/roadmap/portability/index.html @@ -229,7 +229,8 @@ Issues Status - Running Python wordcount on Flink or Spark + Running Python wordcount on Flink + Running Python wordcount on Spark @@ -389,33 +390,41 @@ their respective components. MVP, and FeatureCompletness nearly done (missing SDF, timers) for SDKs, Python ULR, and shared java runners library. -Flink is the first runner to fully leverage this, with focus moving to -Performance. +Currently, the Flink and Spark runners support portable pipeline execution. See the https://s.apache.org/apache-beam-portability-support-table;>Portability support table for details. -Running Python wordcount on Flink or Spark +Running Python wordcount on Flink -Currently, the Flink and Spark runners support portable pipeline execution. -To run a basic Python wordcount (in batch mode) with embedded Flink or Spark: +To run a basic Python wordcount (in batch mode) with embedded Flink: Run once to build the SDK harness container: ./gradlew :sdks:python:container:docker - Choose one: - - Start the Flink portable JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow - Or start the Spark portable JobService endpoint: ./gradlew :runners:spark:job-server:runShadow - - - Submit the wordcount pipeline to above endpoint: ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK + Start the Flink portable JobService endpoint: ./gradlew :runners:flink:1.5:job-server:runShadow + In a new terminal, submit the wordcount pipeline to above endpoint: ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentT
[beam] branch asf-site updated: Publishing website 2019/07/16 04:17:12 at commit 708cea5
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6bcfedf Publishing website 2019/07/16 04:17:12 at commit 708cea5
6bcfedf is described below
commit 6bcfedf48d233dbbfce0cdebed15dea52990b244
Author: jenkins
AuthorDate: Tue Jul 16 04:17:12 2019 +
Publishing website 2019/07/16 04:17:12 at commit 708cea5
---
.../blog/2019/06/11/looping-timers.html| 67 --
website/generated-content/feed.xml | 67 --
2 files changed, 74 insertions(+), 60 deletions(-)
diff --git a/website/generated-content/blog/2019/06/11/looping-timers.html
b/website/generated-content/blog/2019/06/11/looping-timers.html
index ee14c18..67202a8 100644
--- a/website/generated-content/blog/2019/06/11/looping-timers.html
+++ b/website/generated-content/blog/2019/06/11/looping-timers.html
@@ -337,45 +337,52 @@ side-input into the FanOut DoFn.
So how do timers help? Well let’s have a look at a new transform:
+Edit: Looping Timer State changed from Boolean to Long to allow for min
value check.
+
public static class LoopingStatefulTimer extends DoFnKVString,
Integer, KVInstant stopTimerTime;
+Instant stopTimerTime;
- LoopingStatefulTimer(Instant stopTime){
-this.stopTimerTime = stopTime;
- }
+LoopingStatefulTimer(Instant stopTime){
+ this.stopTimerTime = stopTime;
+}
- @StateId("timerRunning")
-private final StateSpecValueStateBoolean timerRunning =
- StateSpecs.value(BooleanCoder.of());
+@StateId("loopingTimerTime")
+private final StateSpecValueStateLong loopingTimerTime =
+StateSpecs.value(BigEndianLongCoder.of());
- @StateId("key")
+@StateId("key")
private final StateSpecValueStateString key =
- StateSpecs.value(StringUtf8Coder.of());
+StateSpecs.value(StringUtf8Coder.of());
- @TimerId("loopingTimer")
+@TimerId("loopingTimer")
private final TimerSpec loopingTimer =
- TimerSpecs.timer(TimeDomain.EVENT_TIME);
+TimerSpecs.timer(TimeDomain.EVENT_TIME);
- @ProcessElement public void process(ProcessContext c, @StateId("key") ValueStateString key@StateId("timerRunning") ValueStateBoolean timerRunning,
+@ProcessElement public
void process(ProcessContext c, @StateId("key") ValueStateString key< [...]
+@StateId("loopingTimerTime") ValueStateLong loopingTimerTime,
@TimerId("loopingTimer") Timer loopingTimer) {
-// If the timer has been set already, do not reset
-if (timerRunning.read() == null) {
- loopingTimer.set(c.timestamp().plus(Duration.standardMinutes(1)));
- timerRunning.write(true);
-}
+ // If the timer has been set already, or if the value
is smaller than
+ // the current element + window duration, do not
set
+ Long currentTimerValue
= loopingTimerTime.read();
+ Instant nextTimerTimeBasedOnCurrentElement =
c.timestamp().plus(Duration.standardMinutes(1));
-// We need this value so that we can output a value for
the correct key in OnTimer
-if (key.read() == null) {
- key.write(c.element().getKey());
-}
+ if (currentTimerValue == null || currentTimerValue
+ nextTimerTimeBasedOnCurrentElement.getMillis())
{
+loopingTimer.set(nextTimerTimeBasedOnCurrentElement);
+loopingTimerTime.write(nextTimerTimeBasedOnCurrentElement.getMillis());
+ }
-c.output(c.element());
- }
+ // We need this value so that we can output a value for
the correct key in OnTimer
+ if (key.read() == null) {
+key.write(c.element().getKey());
+ }
+
+ c.output(c.element());
+}
- @OnTimer("loopingTimer")
+@OnTimer("loopingTimer")
public void onTimer(
OnTimerContext c,
@StateId("key") ValueStateString key,
@@ -387,14 +394,14 @@ side-input into the FanOut DoFn.
// If we do not put in a “time to live” value, then the
timer would loop forever
Instant nextTimer = c.timestamp().plus(Duration.standardMinutes(1));
if (nextTimer.isBefore(stopTimerTime)) {
- loopingTimer.set(nextTimer);
+loopingTimer.set(nextTimer);
} else {
LOG.info(
- "Timer not being set as exceeded Stop Timer value {}
",
- stopTimerTime);
+"Timer not being set as exceeded Stop Timer value
{} ",
+stopTimerTime);
}
- }
-}
+}
+ }
diff --git a/website/generated-content/feed.xml
b/website/generated-content/feed.xml
index 704943b..001cb30 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -184,45 +184,52 @@ side-input i
[beam] branch asf-site updated: Publishing website 2019/07/15 12:19:17 at commit 3f53bec
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 534810b Publishing website 2019/07/15 12:19:17 at commit 3f53bec
534810b is described below
commit 534810b303580e64d1f12522543259fbb044c6ee
Author: jenkins
AuthorDate: Mon Jul 15 12:19:18 2019 +
Publishing website 2019/07/15 12:19:17 at commit 3f53bec
---
.../generated-content/contribute/ptransform-style-guide/index.html| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/website/generated-content/contribute/ptransform-style-guide/index.html
b/website/generated-content/contribute/ptransform-style-guide/index.html
index ab62c7d..b4f508c 100644
--- a/website/generated-content/contribute/ptransform-style-guide/index.html
+++ b/website/generated-content/contribute/ptransform-style-guide/index.html
@@ -759,7 +759,7 @@ Strive to make such incompatible behavior changes cause a
compile error (e.g. it
The container class must have a private constructor, so it can’t be
instantiated directly.
-Document common stuff at BlahIO
level, and each factory method individually.
+Document common stuff at FooIO
level, and each factory method individually.
/** Transforms for clustering data.
*/
public class Cluster {
@@ -775,7 +775,7 @@ Strive to make such incompatible behavior changes cause a
compile error (e.g. it
public static class Hierarchically extends PTransform... { ... }
}
-public lass FooIO {
+public class FooIO {
// Force use of static factory methods.
private FooIO() {}
[beam] branch asf-site updated: Publishing website 2019/07/11 14:26:59 at commit 1c9799d
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bad5fe4 Publishing website 2019/07/11 14:26:59 at commit 1c9799d
bad5fe4 is described below
commit bad5fe433bf09c75d7609990a3446545c1dbe76f
Author: jenkins
AuthorDate: Thu Jul 11 14:26:59 2019 +
Publishing website 2019/07/11 14:26:59 at commit 1c9799d
---
.../blog/2018/01/09/beam-a-look-back.html | 2 +-
.../documentation/dsls/sql/shell/index.html| 2 +-
.../documentation/io/built-in/hadoop/index.html| 16 +-
.../patterns/file-processing-patterns/index.html | 9 +++---
.../documentation/patterns/overview/index.html | 2 +-
.../patterns/side-input-patterns/index.html| 36 +-
.../documentation/programming-guide/index.html | 6
7 files changed, 31 insertions(+), 42 deletions(-)
diff --git a/website/generated-content/blog/2018/01/09/beam-a-look-back.html
b/website/generated-content/blog/2018/01/09/beam-a-look-back.html
index b4200b9..650987a 100644
--- a/website/generated-content/blog/2018/01/09/beam-a-look-back.html
+++ b/website/generated-content/blog/2018/01/09/beam-a-look-back.html
@@ -229,7 +229,7 @@ new and updated runners were developed:
In addition to runners, Beam added new IO connectors, some notable ones
being
the Cassandra, MQTT, AMQP, HBase/HCatalog, JDBC, Solr, Tika, Redis, and
-ElasticSearch connectors. Beam’s IO connectors make it possible to read from or
+Elasticsearch connectors. Beam’s IO connectors make it possible to read from or
write to data sources/sinks even when they are not natively supported by the
underlying execution engine. Beam also provides fully pluggable filesystem
support, allowing us to support and extend our coverage to HDFS, S3, Azure
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html
b/website/generated-content/documentation/dsls/sql/shell/index.html
index 9d04b38..a1e5fd4 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -303,7 +303,7 @@ limitations under the License.
./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
-./sdks/java/extensions/sql/shell/build/install/beam-sdks-java-extensions-sql-shell/bin/beam-sdks-java-extensions-sql-shell
+./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
diff --git
a/website/generated-content/documentation/io/built-in/hadoop/index.html
b/website/generated-content/documentation/io/built-in/hadoop/index.html
index aaf776e..44dabf9 100644
--- a/website/generated-content/documentation/io/built-in/hadoop/index.html
+++ b/website/generated-content/documentation/io/built-in/hadoop/index.html
@@ -572,13 +572,13 @@ limitations under the License.
To read data from Elasticsearch, use EsInputFormat, which needs following
properties to be set:
-Configuration elasticSearchConf = new Configuration();
-elasticSearchConf.set("es.nodes", ElasticsearchHostIp);
-elasticSearchConf.set("es.port", "9200");
-elasticSearchConf.set("es.resource", "ElasticIndexName/ElasticTypeName");
-elasticSearchConf.setClass("key.class", org.apache.hadoop.io.Text Text.classelasticSearchConf.setClass("value.class", org.elasticsearch.hadoop.mr.LinkedMapWritable LinkedMapWritableelasticSearchConf.setClass("mapreduce.job.inputformat.class",
org.elasticsearch.hadoop.mr.EsInputFormat EsInputFormatConfiguration elasticsearchConf = new Configuration();
+elasticsearchConf.set("es.nodes", ElasticsearchHostIp);
+elasticsearchConf.set("es.port", "9200");
+elasticsearchConf.set("es.resource", "ElasticIndexName/ElasticTypeName");
+elasticsearchConf.setClass("key.class", org.apache.hadoop.io.Text Text.classelasticsearchConf.setClass("value.class", org.elasticsearch.hadoop.mr.LinkedMapWritable LinkedMapWritableelasticsearchConf.setClass("mapreduce.job.inputformat.class",
org.elasticsearch.hadoop.mr.EsInputFormat EsInputFormat
@@ -589,7 +589,7 @@ limitations under the License.
Call Read transform as follows:
PCollectionKVText, LinkedMapWritable elasticData = p.apply("rea [...]
- HadoopFormatIO.Text, LinkedMapWritableread().withConfiguration(elasticSearchConf));
+ HadoopFormatIO.Text, LinkedMapWritableread().withConfiguration(elasticsearchConf));
diff --git
a/website/generated-content/documentation/patterns/file-processing-patterns/index.html
b/website/generated-content/documentation/patterns/file-processing-patterns/index.html
index 3734a68..cdc7e31 100644
---
a/website/generated-content/documentation/patterns/
[beam] branch asf-site updated: Publishing website 2019/07/04 11:24:45 at commit 73735cc
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 5be1df9 Publishing website 2019/07/04 11:24:45 at commit 73735cc 5be1df9 is described below commit 5be1df9d5cec8429de286e464e5c1be983f377f1 Author: jenkins AuthorDate: Thu Jul 4 11:24:45 2019 + Publishing website 2019/07/04 11:24:45 at commit 73735cc --- .../get-started/beam-overview/index.html | 1 + website/generated-content/images/logos/runners/jet.png | Bin 0 -> 9128 bytes 2 files changed, 1 insertion(+) diff --git a/website/generated-content/get-started/beam-overview/index.html b/website/generated-content/get-started/beam-overview/index.html index 3d38424..b249dde 100644 --- a/website/generated-content/get-started/beam-overview/index.html +++ b/website/generated-content/get-started/beam-overview/index.html @@ -265,6 +265,7 @@ limitations under the License. Apache Samza Apache Spark Google Cloud Dataflow + Hazelcast Jet Note: You can always execute your pipeline locally for testing and debugging purposes. diff --git a/website/generated-content/images/logos/runners/jet.png b/website/generated-content/images/logos/runners/jet.png new file mode 100644 index 000..1eb2738 Binary files /dev/null and b/website/generated-content/images/logos/runners/jet.png differ
