[beam] branch asf-site updated: Publishing website 2022/02/08 06:01:46 at commit 56e5c2c
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ec6a3cb Publishing website 2022/02/08 06:01:46 at commit 56e5c2c
ec6a3cb is described below
commit ec6a3cbf12e1ba1f49cc4c259df9432e1488ee57
Author: jenkins
AuthorDate: Tue Feb 8 06:01:47 2022 +
Publishing website 2022/02/08 06:01:46 at commit 56e5c2c
---
website/generated-content/blog/beam-2.36.0/index.html | 2 +-
website/generated-content/blog/index.xml | 2 +-
website/generated-content/categories/blog/index.xml | 2 +-
website/generated-content/feed.xml| 2 +-
website/generated-content/sitemap.xml | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/website/generated-content/blog/beam-2.36.0/index.html
b/website/generated-content/blog/beam-2.36.0/index.html
index 6880d90..91467a0 100644
--- a/website/generated-content/blog/beam-2.36.0/index.html
+++ b/website/generated-content/blog/beam-2.36.0/index.html
@@ -20,7 +20,7 @@ function endSearch(){var
search=document.querySelector(".searchBar");search.clas
function blockScroll(){$("body").toggleClass("fixedPosition");}
function openMenu(){addPlaceholder();blockScroll();}http://schema.org/BlogPosting>blog2022/02/07Apache Beam 2.36.0Emily YeWe are happy to present [...]
This release includes both improvements and new functionality.
-See the download page for
this release.For more information on changes in 2.36.0, check out the https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350406";>detailed
release
+See the download page for
this release.For more information on changes in 2.36.0, check out the https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12350407";>detailed
release
notes.I/OsSupport for stopReadTime on KafkaIO
SDF (Java).(https://issues.apache.org/jira/browse/BEAM-13171>BEAM-13171).New Features / ImprovementsAdded
support for cloudpickle as a pickling library for Python SDK (https://issues.apache.org/jira/browse/BEAM-8123>BEAM-8123). To use
cloudpickle, set pipeline option: –pickler_lib=cloudpickleAdded
option to specify t [...]
(https://issues.apache.org/jira/browse/BEAM-13421%5D>BEAM-13421).Known IssuesUsers may encounter an unexpected
java.lang.ArithmeticException when outputting a timestamp
for an element further than allowedSkew from an allowed DoFN skew set to a
value more than
diff --git a/website/generated-content/blog/index.xml
b/website/generated-content/blog/index.xml
index 92defaa..cce94fe 100644
--- a/website/generated-content/blog/index.xml
+++ b/website/generated-content/blog/index.xml
@@ -13,7 +13,7 @@ limitations under the License.
We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download
page for this release.
-For more information on changes in 2.36.0, check out the detailed
release
+For more information on changes in 2.36.0, check out the detailed
release
notes.
I/Os
diff --git a/website/generated-content/categories/blog/index.xml
b/website/generated-content/categories/blog/index.xml
index 007ac83..42848ef 100644
--- a/website/generated-content/categories/blog/index.xml
+++ b/website/generated-content/categories/blog/index.xml
@@ -13,7 +13,7 @@ limitations under the License.
We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download
page for this release.
-For more information on changes in 2.36.0, check out the detailed
release
+For more information on changes in 2.36.0, check out the detailed
release
notes.
I/Os
diff --git a/website/generated-content/feed.xml
b/website/generated-content/feed.xml
index 1a076e5..34658e6 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -13,7 +13,7 @@ limitations under the License.
We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download
page for this release.
-For more information on changes
[beam] branch nightly-refs/heads/master updated (30a667d -> 7bb7fab)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 30a667d Merge pull request #16727: [BEAM-11971] remove unsafe
Concurrent data structure
add 7bafabc Merge pull request #16726 from [BEAM-12164]: Parses change
streams fields as json / strings
add 6b09d10 [BEAM-13147] Avoid nullness issue during init of AwsModule
(AWS Sdk v2)
add 7e2b637 Merge pull request #16752: [BEAM-13147] Avoid nullness issue
during init of AwsModule (AWS Sdk v2)
add 98e5fc5 [BEAM-11971] Revert "Fix timer consistency in direct runner"
(#16748)
add 74d94ea [BEAM-13193] Aggregates fn api outbound data/timers of
different endpoints (#16439)
add 64581e0 [BEAM-13767] Migrate a bundle of grade tasks to use
configuration avoidance API. (#16648)
add d3b680e Merge pull request #16653 from [BEAM-12164]: Add integration
tests for spanner change streams
add d27d709 Merge pull request #16728 from [BEAM-13823] Update docs for
SnowflakeIO
add 3f75777 [BEAM-12572] Change jobs to run as cron jobs
add 93ef2f5 Merge pull request #16746 from benWize/BEAM-12572-cron-jobs
add c182e4f Merge pull request #16660 from [BEAM-13771][Playground] Send
multifile value to the frontend
add 7d76d58 Merge pull request #16646 from [BEAM-13643][Playground] Setup
running backend to verify SCIO SDK examples
add eef048b [BEAM-13015] Add state caching benchmark and move benchmarks
to their own module. (#16542)
add 87f10bb [BEAM-12976] Log projection pushdown optimizations.
add 6595861 Merge pull request #16724 from ibzib/log-opt
add 5e8c258 [BEAM-13419] Check for initialization in dataflow runner
(#16765)
add de7e9cd [BEAM-13820] Changed color of delete icon in pipeline options
dropdown, removed unused imports
add 63d9577 Merge pull request #16758 from [BEAM-13820] [Playground]
Outline delete icon in Pipeline options
add cf8af05 Merge pull request #16701 from [BEAM-13786] [Playground]
[Bugfix] Update CI/CD to verify only single-file examples
add bb15d18 Merge pull request #16754 from [BEAM-13838][Playground] Add
logs in case of empty graph for CD step
add 1ef5056 [BEAM-13293] consistent naming for expansion service address
and flag update (#16764)
add 1693483 Merge pull request #16700 from [BEAM-13790][Playground]
Change logic of receiving examples from the bucket on backend side
add 67d4b10 [BEAM-13830] update dependency for debeziumio expansion
service (#16743)
add ed41a82 [BEAM-13761] consistent namings for expansion address in
Debezium IO (#16766)
add 6bc5b66 [BEAM-13806] Shutting down SchemaIO expansion services from
Go VR script. (#16770)
add ee9bdbc [release-2.36.0] Update website/changelog for release 2.36.0
(#16627)
add 730b941 [BEAM-13848] Update numpy intersphinx link (#16767)
add 7bb7fab [release-23.6.0] Fix JIRA link for 2.36 blog (#16771)
No new revisions were added by this update.
Summary of changes:
.github/workflows/build_playground_backend.yml | 9 +
.github/workflows/build_playground_frontend.yml| 5 +-
.../job_PostCommit_Python_Examples_Dataflow.groovy | 37 +-
.../job_PostCommit_Python_Examples_Direct.groovy | 39 +-
.../job_PostCommit_Python_Examples_Flink.groovy| 39 +-
.../job_PostCommit_Python_Examples_Spark.groovy| 46 ++-
CHANGES.md | 12 +-
build.gradle.kts | 66 ++--
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 183 ++
playground/api/v1/api.proto| 3 +-
playground/backend/containers/scio/Dockerfile | 2 +-
playground/backend/containers/scio/build.gradle| 2 +-
playground/backend/internal/api/v1/api.pb.go | 398 +++--
.../internal/cloud_bucket/precompiled_objects.go | 33 +-
.../cloud_bucket/precompiled_objects_test.go | 4 +-
.../internal/utils/precompiled_objects_utils.go| 1 +
.../utils/precompiled_objects_utils_test.go| 2 +-
playground/frontend/lib/api/v1/api.pb.dart | 24 +-
playground/frontend/lib/api/v1/api.pbjson.dart | 5 +-
.../pipeline_options_dropdown_body.dart| 2 -
.../pipeline_options_form.dart | 6 +-
playground/infrastructure/api/v1/api_pb2.py| 87 ++---
playground/infrastructure/cd_helper.py | 16 +-
playground/infrastructure/ci_cd.py | 7 +-
playground/infrastructure/ci_helper.py | 8 +-
playground/infrastructure/grpc_client.py | 9 +-
playground/infrastructure/helper.py| 23 +-
.../terraform/applications/backend-router/main.tf | 15 +-
.../{backend-router => backend-scio}/main.tf | 37 +-
.../main.tf => backend-scio/variables.tf} | 73
[beam] branch master updated (7bb7fab -> 56e5c2c)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7bb7fab [release-23.6.0] Fix JIRA link for 2.36 blog (#16771) add 56e5c2c [BEAM-13647] Use role for Go worker binary. (#16729) No new revisions were added by this update. Summary of changes: .../pipeline/src/main/proto/beam_runner_api.proto | 6 + sdks/go/container/boot.go | 16 +- sdks/go/pkg/beam/artifact/materialize.go | 5 +- sdks/go/pkg/beam/core/runtime/graphx/translate.go | 48 -- .../pkg/beam/core/runtime/graphx/translate_test.go | 165 + .../beam/runners/dataflow/dataflowlib/execute.go | 16 +- .../pkg/beam/runners/dataflow/dataflowlib/stage.go | 18 ++- .../beam/runners/universal/runnerlib/execute.go| 64 ++-- .../pkg/beam/runners/universal/runnerlib/stage.go | 4 +- 9 files changed, 303 insertions(+), 39 deletions(-)
[beam] branch master updated (730b941 -> 7bb7fab)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 730b941 [BEAM-13848] Update numpy intersphinx link (#16767) add 7bb7fab [release-23.6.0] Fix JIRA link for 2.36 blog (#16771) No new revisions were added by this update. Summary of changes: website/www/site/content/en/blog/beam-2.36.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch v2.36.0 created (now 730b941)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch v2.36.0 in repository https://gitbox.apache.org/repos/asf/beam.git. at 730b941 [BEAM-13848] Update numpy intersphinx link (#16767) No new revisions were added by this update.
[beam] branch v2.35.0 created (now 81f3a16)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch v2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git. at 81f3a16 Set version for 2.36.0 RC3 No new revisions were added by this update.
[beam] branch master updated (ee9bdbc -> 730b941)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ee9bdbc [release-2.36.0] Update website/changelog for release 2.36.0 (#16627) add 730b941 [BEAM-13848] Update numpy intersphinx link (#16767) No new revisions were added by this update. Summary of changes: sdks/python/scripts/generate_pydoc.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
svn commit: r52443 - in /release/beam/2.36.0: ./ python/
Author: lcwik Date: Tue Feb 8 00:34:00 2022 New Revision: 52443 Log: Finalize Apache Beam 2.36.0 release. Added: release/beam/2.36.0/ release/beam/2.36.0/apache-beam-2.36.0-source-release.zip (with props) release/beam/2.36.0/apache-beam-2.36.0-source-release.zip.asc release/beam/2.36.0/apache-beam-2.36.0-source-release.zip.sha512 release/beam/2.36.0/python/ release/beam/2.36.0/python/apache-beam-2.36.0.zip (with props) release/beam/2.36.0/python/apache-beam-2.36.0.zip.asc release/beam/2.36.0/python/apache-beam-2.36.0.zip.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-macosx_10_9_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-macosx_10_9_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-macosx_10_9_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_i686.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_i686.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_i686.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_i686.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_i686.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_i686.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2010_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2014_aarch64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2014_aarch64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-manylinux2014_aarch64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win32.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win32.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win32.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win_amd64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win_amd64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp36-cp36m-win_amd64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-macosx_10_9_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-macosx_10_9_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-macosx_10_9_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_i686.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_i686.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_i686.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_i686.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_i686.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_i686.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_x86_64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_x86_64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2010_x86_64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2014_aarch64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2014_aarch64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-manylinux2014_aarch64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win32.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win32.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win32.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win_amd64.whl (with props) release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win_amd64.whl.asc release/beam/2.36.0/python/apache_beam-2.36.0-cp37-cp37m-win_amd64.whl.sha512 release/beam/2.36.0/python/apache_beam-2.36.0
svn commit: r52442 - /dev/beam/2.36.0/
Author: lcwik Date: Tue Feb 8 00:33:12 2022 New Revision: 52442 Log: Finalize Apache Beam 2.36.0 release Removed: dev/beam/2.36.0/
[beam] annotated tag v2.36.0 updated (81f3a16 -> 95a76a3)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to annotated tag v2.36.0 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.36.0 was modified! *** from 81f3a16 (commit) to 95a76a3 (tag) tagging cb30abe5fc116ab342c62466bd9c070b7ba0c30f (tag) length 143 bytes by Emily Ye on Tue Feb 8 00:00:18 2022 + - Log - Release 2.36.0 tag -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEcw1ejUyoEMHwHTBLYQs4qhvhFlYFAmIBspwACgkQYQs4qhvh FlZP/hAAgTuaD7XsNodneT7mKoHORERX1EJersJ+0f652iRxVNNy7/nLw0PUGsIY /0KdQwz+vx7Ep1Y0mNb1QWVVNZMGprntKZOpTVnZXVrLIkdigILAWEXTrQZ2QmX7 xkWbXoMUzR5xQHp8R4IPpiq7hA9HhYQrBYSMfQ2bZTKEd5UF7CbTFWSe/ipLAFjf ObdT7g1A0Ee8UjK+dQTQIk+dOx23h50i/IERUHlJDfeFbfiwGDYr2xoqqCY3AYLY xsDrs0E1Sy3ZUTEbjHWrYf8Ikei+6pdrneefFG6bwnudl96iIhEOvwFy5zuqylAb 1siAvygrvdYvu0+IcnKKyCJmgNaa+YKui2HSLehRsf8DHw/jM5MMBugj6xXQXreE 6jipSqUlgH3TDx/zQ1r45m0hjdAvsQu3+edLtmNvhX965shQ9aYlMPLM29xDcPX3 uKsGXbvGZ7+UZwkbyE5H3WI6gBtVJ225gNin4HZWZWWXuoAdBaxFxL0r7eD6T+8I tQw54lbXfVMQlaXrhoHiZgs5AhEzD3bLbJ9UV/ZkBdVJhjpUjlfbE/n0vRUjSEe1 yil+CPbSg1F+Lm9h3j9tcVCQAQkIjtnDTF7oNTOntaE/PQL2U+WKEnaArjwCySJX Ef1bWRuV+UXRo+ORgTAlMVnN7bYaYlScuOWGi6ucK0g/SijnZvE= =9Luk -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] annotated tag sdks/v2.36.0 updated (81f3a16 -> 5706351)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to annotated tag sdks/v2.36.0 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag sdks/v2.36.0 was modified! *** from 81f3a16 (commit) to 5706351 (tag) tagging cb30abe5fc116ab342c62466bd9c070b7ba0c30f (tag) length 143 bytes by Emily Ye on Mon Feb 7 23:59:41 2022 + - Log - Go SDK tag for v2.36.0 -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEcw1ejUyoEMHwHTBLYQs4qhvhFlYFAmIBsnoACgkQYQs4qhvh FlaVZRAAlqczjpYHhghTrd81RL8bX4F/8Iw7j5YvbY6bBdpYKdYfKGLIvBsdbj+G Twrd1Ls0JvSa/qrjXzZ+vl0/hrWzyc4xTO4c9QZSEesw6QmNuFE3hk4/Y2GvDnLK xCjdLpUG2I+c7+QBIzTeX/Hbz581kGcZekXWPwakjw/0IbWYR8RoBhGPHpTwHEea OQ3s4Z3loZFs5xHrl2jRP+Col0STXI+RlWrmOFKNNU88UOWaMSLTN8r+Wywht3wq bup/hgqdKtSb78xDxpZ/LqhSKfTm1YI9FC78ONkLwN77c4ufgDEJV/tS0Jj3NftR 80aAT2VtWxiHVjh/3ZoJs2E4rUlAvCcCziX+BSg337YVjSxggu/Qlion8OHaNClW jycv/FacbUoTVRyhjxlQGkW4WeK7kZaOZZ1aJiYTQcBlqbjuQQ2JsihxwGpC/5Ni dQhtOOoCAc9/dIRTEyD6wq269svIrxeGJPMjrJeR53iC023PjPhVAWXhXen+fI1H kUOrCpN8Iq9gyiE+CArby42vSHErI85Sty6Sv9LnWB023uOSUtIxHlF8kNQBUP7d R4xLRxOVrJGNZQ7GQqdjSHIrAWHfpEPpM8CCUeEJm12oMTxNtQxDpALvl9Xl7v64 ibyuujQN9BqhxYn2rsWhTv66wI9k3/7ijk7QbyS/w8Z9Zota100= =0O+9 -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] branch master updated (6bc5b66 -> ee9bdbc)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6bc5b66 [BEAM-13806] Shutting down SchemaIO expansion services from Go VR script. (#16770) add ee9bdbc [release-2.36.0] Update website/changelog for release 2.36.0 (#16627) No new revisions were added by this update. Summary of changes: CHANGES.md | 12 +- website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.36.0.md| 153 + .../www/site/content/en/get-started/downloads.md | 8 ++ website/www/site/data/authors.yml | 4 + website/www/site/static/.htaccess | 2 +- 6 files changed, 173 insertions(+), 8 deletions(-) create mode 100644 website/www/site/content/en/blog/beam-2.36.0.md
[beam] branch master updated (ed41a82 -> 6bc5b66)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ed41a82 [BEAM-13761] consistent namings for expansion address in Debezium IO (#16766) add 6bc5b66 [BEAM-13806] Shutting down SchemaIO expansion services from Go VR script. (#16770) No new revisions were added by this update. Summary of changes: sdks/go/test/run_validatesrunner_tests.sh | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (67d4b10 -> ed41a82)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 67d4b10 [BEAM-13830] update dependency for debeziumio expansion service (#16743) add ed41a82 [BEAM-13761] consistent namings for expansion address in Debezium IO (#16766) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/io/xlang/debeziumio/debezium.go | 20 ++-- .../test/integration/io/xlang/debezium/debezium.go | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-)
[beam] branch master updated (1693483 -> 67d4b10)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1693483 Merge pull request #16700 from [BEAM-13790][Playground] Change logic of receiving examples from the bucket on backend side add 67d4b10 [BEAM-13830] update dependency for debeziumio expansion service (#16743) No new revisions were added by this update. Summary of changes: sdks/java/io/expansion-service/build.gradle | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (1ef5056 -> 1693483)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1ef5056 [BEAM-13293] consistent naming for expansion service address and flag update (#16764) add 1693483 Merge pull request #16700 from [BEAM-13790][Playground] Change logic of receiving examples from the bucket on backend side No new revisions were added by this update. Summary of changes: .../internal/cloud_bucket/precompiled_objects.go | 32 ++ .../cloud_bucket/precompiled_objects_test.go | 4 +-- playground/infrastructure/helper.py| 6 +++- 3 files changed, 33 insertions(+), 9 deletions(-)
[beam] branch master updated (bb15d18 -> 1ef5056)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bb15d18 Merge pull request #16754 from [BEAM-13838][Playground] Add logs in case of empty graph for CD step add 1ef5056 [BEAM-13293] consistent naming for expansion service address and flag update (#16764) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/io/xlang/jdbcio/jdbc.go | 36 +++--- sdks/go/test/integration/integration.go| 11 +-- sdks/go/test/integration/io/xlang/jdbc/jdbc.go | 4 +-- .../go/test/integration/io/xlang/jdbc/jdbc_test.go | 8 ++--- 4 files changed, 25 insertions(+), 34 deletions(-)
[beam] branch master updated: Merge pull request #16754 from [BEAM-13838][Playground] Add logs in case of empty graph for CD step
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bb15d18 Merge pull request #16754 from [BEAM-13838][Playground] Add
logs in case of empty graph for CD step
bb15d18 is described below
commit bb15d1896a2972bb6b95eaa28408345ac8c27b50
Author: Aydar Zainutdinov
AuthorDate: Tue Feb 8 00:27:11 2022 +0300
Merge pull request #16754 from [BEAM-13838][Playground] Add logs in case of
empty graph for CD step
* [BEAM-13838][Playground]
Add logs for CD step
* [BEAM-13838][Playground]
Fix typos
---
playground/infrastructure/cd_helper.py | 5 -
playground/infrastructure/grpc_client.py | 9 ++---
playground/infrastructure/helper.py | 17 +
3 files changed, 19 insertions(+), 12 deletions(-)
diff --git a/playground/infrastructure/cd_helper.py
b/playground/infrastructure/cd_helper.py
index e2e8599..5797b1f 100644
--- a/playground/infrastructure/cd_helper.py
+++ b/playground/infrastructure/cd_helper.py
@@ -79,7 +79,10 @@ class CDHelper:
if len(examples) > 0 and (examples[0].sdk is SDK_PYTHON or
examples[0].sdk is SDK_JAVA):
- tasks = [client.get_graph(example.pipeline_id) for example in examples]
+ tasks = [
+ client.get_graph(example.pipeline_id, example.filepath)
+ for example in examples
+ ]
graphs = await asyncio.gather(*tasks)
for graph, example in zip(graphs, examples):
diff --git a/playground/infrastructure/grpc_client.py
b/playground/infrastructure/grpc_client.py
index ddd9152..40543d5 100644
--- a/playground/infrastructure/grpc_client.py
+++ b/playground/infrastructure/grpc_client.py
@@ -16,7 +16,7 @@
"""
Module contains the client to communicate with GRPC test Playground server
"""
-
+import logging
import uuid
import grpc
@@ -130,12 +130,13 @@ class GRPCClient:
response = await self._stub.GetCompileOutput(request)
return response.output
- async def get_graph(self, pipeline_uuid: str) -> str:
+ async def get_graph(self, pipeline_uuid: str, example_filepath: str) -> str:
"""
Get the graph of pipeline execution.
Args:
pipeline_uuid: uuid of the pipeline
+example_filepath: path to the file of the example
Returns:
graph: contain the graph of pipeline execution as a string
@@ -144,9 +145,11 @@ class GRPCClient:
request = api_pb2.GetGraphRequest(pipeline_uuid=pipeline_uuid)
try:
response = await self._stub.GetGraph(request)
+ if response.graph == "":
+logging.warning("Graph for %s wasn't generated", example_filepath)
return response.graph
except grpc.RpcError:
- # Some examples doesn't have graph (katas)
+ logging.warning("Graph for %s wasn't generated", example_filepath)
return ""
def _verify_pipeline_uuid(self, pipeline_uuid):
diff --git a/playground/infrastructure/helper.py
b/playground/infrastructure/helper.py
index f90cd87..8dd606a 100644
--- a/playground/infrastructure/helper.py
+++ b/playground/infrastructure/helper.py
@@ -46,7 +46,8 @@ Tag = namedtuple(
TagFields.pipeline_options,
TagFields.default_example,
TagFields.context_line
-], defaults=(None, None, False, None, None, False, None))
+],
+defaults=(None, None, False, None, None, False, None))
@dataclass
@@ -341,13 +342,13 @@ def _validate(tag: dict, supported_categories: List[str])
-> bool:
# check that context line's value is integer
context_line = tag.get(TagFields.context_line)
if not isinstance(context_line, int):
- logging.error(
- "tag's field context_line is incorrect: %s \n"
- "context_line variable should be integer format, "
- "but tag contains: %s",
- tag,
- context_line)
- valid = False
+logging.error(
+"Tag's field context_line is incorrect: %s \n"
+"context_line variable should be integer format, "
+"but tag contains: %s",
+tag,
+context_line)
+valid = False
return valid
[beam] branch master updated: Merge pull request #16701 from [BEAM-13786] [Playground] [Bugfix] Update CI/CD to verify only single-file examples
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new cf8af05 Merge pull request #16701 from [BEAM-13786] [Playground]
[Bugfix] Update CI/CD to verify only single-file examples
cf8af05 is described below
commit cf8af0588d67de0740334dcf6d6e4488a4394320
Author: Pavel Avilov
AuthorDate: Tue Feb 8 00:24:57 2022 +0300
Merge pull request #16701 from [BEAM-13786] [Playground] [Bugfix] Update
CI/CD to verify only single-file examples
* Update CI/CD to verify only single-file examples
* Refactoring code
* list comprehension to filter
---
playground/infrastructure/cd_helper.py | 11 +++
playground/infrastructure/ci_cd.py | 7 ---
playground/infrastructure/ci_helper.py | 8 +---
3 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/playground/infrastructure/cd_helper.py
b/playground/infrastructure/cd_helper.py
index 1f46401..e2e8599 100644
--- a/playground/infrastructure/cd_helper.py
+++ b/playground/infrastructure/cd_helper.py
@@ -41,14 +41,17 @@ class CDHelper:
It is used to save beam examples/katas/tests and their output on the GCS.
"""
-
def store_examples(self, examples: List[Example]):
"""
Store beam examples and their output in the Google Cloud.
+
+Outputs for multifile examples are left empty.
"""
-logging.info("Start of executing Playground examples ...")
-asyncio.run(self._get_outputs(examples))
-logging.info("Finish of executing Playground examples")
+single_file_examples = list(filter(
+ lambda example: example.tag.multifile is False, examples))
+logging.info("Start of executing only single-file Playground examples ...")
+asyncio.run(self._get_outputs(single_file_examples))
+logging.info("Finish of executing single-file Playground examples")
logging.info("Start of sending Playground examples to the bucket ...")
self._save_to_cloud_storage(examples)
diff --git a/playground/infrastructure/ci_cd.py
b/playground/infrastructure/ci_cd.py
index 1d94e0c..3517113 100644
--- a/playground/infrastructure/ci_cd.py
+++ b/playground/infrastructure/ci_cd.py
@@ -51,7 +51,7 @@ categories_file = os.getenv("BEAM_EXAMPLE_CATEGORIES")
def _ci_step(examples: List[Example]):
"""
- CI step to verify all beam examples/tests/katas
+ CI step to verify single-file beam examples/tests/katas
"""
ci_helper = CIHelper()
@@ -84,9 +84,10 @@ def _run_ci_cd(step: config.Config.CI_CD_LITERAL, sdk: Sdk):
logging.info("Number of found Playground examples: %s", len(examples))
if step == config.Config.CI_STEP_NAME:
-logging.info("Start of verification Playground examples ...")
+logging.info(
+"Start of verification only single_file Playground examples ...")
_ci_step(examples=examples)
-logging.info("Finish of verification Playground examples")
+logging.info("Finish of verification single_file Playground examples")
if step == config.Config.CD_STEP_NAME:
logging.info("Start of storing Playground examples ...")
_cd_step(examples=examples)
diff --git a/playground/infrastructure/ci_helper.py
b/playground/infrastructure/ci_helper.py
index 9ae3cc5..1792155 100644
--- a/playground/infrastructure/ci_helper.py
+++ b/playground/infrastructure/ci_helper.py
@@ -43,10 +43,12 @@ class CIHelper:
1. Find all beam examples starting from directory
os.getenv("BEAM_ROOT_DIR")
2. Group code of examples by their SDK.
-3. Run processing for all examples to verify examples' code.
+3. Run processing for single-file examples to verify examples' code.
"""
-await get_statuses(examples)
-await self._verify_examples(examples)
+single_file_examples = list(filter(
+ lambda example: example.tag.multifile is False, examples))
+await get_statuses(single_file_examples)
+await self._verify_examples(single_file_examples)
async def _verify_examples(self, examples: List[Example]):
"""
[beam] branch master updated: [BEAM-13820] Changed color of delete icon in pipeline options dropdown, removed unused imports
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new de7e9cd [BEAM-13820] Changed color of delete icon in pipeline options
dropdown, removed unused imports
new 63d9577 Merge pull request #16758 from [BEAM-13820] [Playground]
Outline delete icon in Pipeline options
de7e9cd is described below
commit de7e9cdfbdc39dac7a204c378cec5d6ec3bb462a
Author: Alexander
AuthorDate: Mon Feb 7 17:12:34 2022 +0400
[BEAM-13820] Changed color of delete icon in pipeline options dropdown,
removed unused imports
---
.../pipeline_options_dropdown/pipeline_options_dropdown_body.dart | 2 --
.../components/pipeline_options_dropdown/pipeline_options_form.dart | 6 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git
a/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_dropdown_body.dart
b/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_dropdown_body.dart
index f2ae442..aa6532a 100644
---
a/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_dropdown_body.dart
+++
b/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_dropdown_body.dart
@@ -18,7 +18,6 @@
import 'package:flutter/material.dart';
import 'package:flutter_gen/gen_l10n/app_localizations.dart';
-import 'package:playground/config/theme.dart';
import 'package:playground/constants/colors.dart';
import 'package:playground/constants/sizes.dart';
import
'package:playground/modules/editor/components/pipeline_options_dropdown/pipeline_option_model.dart';
@@ -26,7 +25,6 @@ import
'package:playground/modules/editor/components/pipeline_options_dropdown/p
import
'package:playground/modules/editor/components/pipeline_options_dropdown/pipeline_options_dropdown_separator.dart';
import
'package:playground/modules/editor/components/pipeline_options_dropdown/pipeline_options_form.dart';
import 'package:playground/modules/editor/parsers/run_options_parser.dart';
-import 'package:playground/modules/notifications/components/notification.dart';
const kOptionsTabIndex = 0;
const kRawTabIndex = 1;
diff --git
a/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_form.dart
b/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_form.dart
index 1d2179f..868ee4b 100644
---
a/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_form.dart
+++
b/playground/frontend/lib/modules/editor/components/pipeline_options_dropdown/pipeline_options_form.dart
@@ -20,6 +20,7 @@ import 'package:flutter/material.dart';
import 'package:flutter_gen/gen_l10n/app_localizations.dart';
import 'package:collection/collection.dart';
import 'package:playground/config/theme.dart';
+import 'package:playground/constants/colors.dart';
import 'package:playground/constants/sizes.dart';
import
'package:playground/modules/editor/components/pipeline_options_dropdown/pipeline_option_label.dart';
import
'package:playground/modules/editor/components/pipeline_options_dropdown/pipeline_option_model.dart';
@@ -72,7 +73,10 @@ class PipelineOptionsForm extends StatelessWidget {
child: IconButton(
iconSize: kIconSizeMd,
splashRadius: kIconButtonSplashRadius,
- icon: const Icon(Icons.delete_outlined),
+ icon: const Icon(
+Icons.delete_outlined,
+color: kLightPrimary,
+ ),
color: ThemeColors.of(context).grey1Color,
onPressed: () => onDelete(index),
),
[beam] branch master updated: [BEAM-13419] Check for initialization in dataflow runner (#16765)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 5e8c258 [BEAM-13419] Check for initialization in dataflow runner
(#16765)
5e8c258 is described below
commit 5e8c25838b90870c1ca604e3fe02887a6393d876
Author: Danny McCormick
AuthorDate: Mon Feb 7 15:51:22 2022 -0500
[BEAM-13419] Check for initialization in dataflow runner (#16765)
---
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 4
1 file changed, 4 insertions(+)
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflow.go
b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
index 89c3797..4cd877f 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflow.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
@@ -155,6 +155,10 @@ var unique int32
// Execute runs the given pipeline on Google Cloud Dataflow. It uses the
// default application credentials to submit the job.
func Execute(ctx context.Context, p *beam.Pipeline) (beam.PipelineResult,
error) {
+ if !beam.Initialized() {
+ panic("Beam has not been initialized. Call beam.Init() before
pipeline construction.")
+ }
+
// (1) Gather job options
project := gcpopts.GetProjectFromFlagOrEnvironment(ctx)
[beam] branch master updated: [BEAM-12976] Log projection pushdown optimizations.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 87f10bb [BEAM-12976] Log projection pushdown optimizations.
new 6595861 Merge pull request #16724 from ibzib/log-opt
87f10bb is described below
commit 87f10bbe7e342730c1113a294387d2e22150755e
Author: Kyle Weaver
AuthorDate: Thu Feb 3 16:13:18 2022 -0800
[BEAM-12976] Log projection pushdown optimizations.
---
.../core/construction/graph/ProjectionPushdownOptimizer.java | 10 ++
1 file changed, 10 insertions(+)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProjectionPushdownOptimizer.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProjectionPushdownOptimizer.java
index 9abb17f..4b6669a 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProjectionPushdownOptimizer.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProjectionPushdownOptimizer.java
@@ -37,9 +37,12 @@ import org.apache.beam.sdk.values.TaggedPValue;
import org.apache.beam.sdk.values.TupleTag;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** See {@link #optimize(Pipeline)}. */
public class ProjectionPushdownOptimizer {
+ private static final Logger LOG =
LoggerFactory.getLogger(ProjectionPushdownOptimizer.class);
/**
* Performs all known projection pushdown optimizations in-place on a
Pipeline.
@@ -78,6 +81,13 @@ public class ProjectionPushdownOptimizer {
// fields.
for (Entry>, Map,
FieldAccessDescriptor>>
entry : taggedFieldAccess.entrySet()) {
+ for (Entry, FieldAccessDescriptor> outputFields :
entry.getValue().entrySet()) {
+LOG.info(
+"Optimizing transform {}: output {} will contain reduced field set
{}",
+entry.getKey(),
+outputFields.getKey(),
+outputFields.getValue().fieldNamesAccessed());
+ }
PTransformMatcher matcher = application -> application.getTransform() ==
entry.getKey();
PushdownOverrideFactory overrideFactory =
new PushdownOverrideFactory<>(entry.getValue());
[beam] branch master updated (7d76d58 -> eef048b)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 7d76d58 Merge pull request #16646 from [BEAM-13643][Playground] Setup
running backend to verify SCIO SDK examples
add eef048b [BEAM-13015] Add state caching benchmark and move benchmarks
to their own module. (#16542)
No new revisions were added by this update.
Summary of changes:
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 88 +---
sdks/java/harness/build.gradle | 19 --
.../sql/perf-tests => harness/jmh}/build.gradle| 32 ++-
.../fn/harness/jmh}/ProcessBundleBenchmark.java| 222 -
.../jmh}/logging/BeamFnLoggingClientBenchmark.java | 4 +-
.../beam/fn/harness/jmh}/logging/package-info.java | 2 +-
.../apache/beam/fn/harness/jmh}/package-info.java | 2 +-
.../fn/harness/jmh/ProcessBundleBenchmarkTest.java | 56 ++
.../logging/BeamFnLoggingClientBenchmarkTest.java | 55 +
settings.gradle.kts| 1 +
10 files changed, 420 insertions(+), 61 deletions(-)
copy sdks/java/{extensions/sql/perf-tests => harness/jmh}/build.gradle (54%)
rename sdks/java/harness/{src/jmh/java/org/apache/beam/fn/harness =>
jmh/src/main/java/org/apache/beam/fn/harness/jmh}/ProcessBundleBenchmark.java
(57%)
rename sdks/java/harness/{src/jmh/java/org/apache/beam/fn/harness =>
jmh/src/main/java/org/apache/beam/fn/harness/jmh}/logging/BeamFnLoggingClientBenchmark.java
(97%)
rename sdks/java/harness/{src/jmh/java/org/apache/beam/fn/harness =>
jmh/src/main/java/org/apache/beam/fn/harness/jmh}/logging/package-info.java
(94%)
rename sdks/java/harness/{src/jmh/java/org/apache/beam/fn/harness =>
jmh/src/main/java/org/apache/beam/fn/harness/jmh}/package-info.java (95%)
create mode 100644
sdks/java/harness/jmh/src/test/java/org/apache/beam/fn/harness/jmh/ProcessBundleBenchmarkTest.java
create mode 100644
sdks/java/harness/jmh/src/test/java/org/apache/beam/fn/harness/jmh/logging/BeamFnLoggingClientBenchmarkTest.java
[beam] branch master updated (c182e4f -> 7d76d58)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c182e4f Merge pull request #16660 from [BEAM-13771][Playground] Send
multifile value to the frontend
add 7d76d58 Merge pull request #16646 from [BEAM-13643][Playground] Setup
running backend to verify SCIO SDK examples
No new revisions were added by this update.
Summary of changes:
.github/workflows/build_playground_backend.yml | 9 ++
.github/workflows/build_playground_frontend.yml| 5 ++--
playground/backend/containers/scio/Dockerfile | 2 +-
playground/backend/containers/scio/build.gradle| 2 +-
.../terraform/applications/backend-router/main.tf | 15 ++
.../{backend-java => backend-scio}/main.tf | 33 +++---
.../{backend-java => backend-scio}/variables.tf| 24 ++--
7 files changed, 48 insertions(+), 42 deletions(-)
copy playground/terraform/applications/{backend-java => backend-scio}/main.tf
(74%)
copy playground/terraform/applications/{backend-java =>
backend-scio}/variables.tf (84%)
[beam] branch master updated (93ef2f5 -> c182e4f)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 93ef2f5 Merge pull request #16746 from benWize/BEAM-12572-cron-jobs add c182e4f Merge pull request #16660 from [BEAM-13771][Playground] Send multifile value to the frontend No new revisions were added by this update. Summary of changes: playground/api/v1/api.proto| 3 +- playground/backend/internal/api/v1/api.pb.go | 398 +++-- .../internal/cloud_bucket/precompiled_objects.go | 1 + .../internal/utils/precompiled_objects_utils.go| 1 + .../utils/precompiled_objects_utils_test.go| 2 +- playground/frontend/lib/api/v1/api.pb.dart | 24 +- playground/frontend/lib/api/v1/api.pbjson.dart | 5 +- playground/infrastructure/api/v1/api_pb2.py| 87 ++--- 8 files changed, 278 insertions(+), 243 deletions(-)
[beam] branch master updated: [BEAM-12572] Change jobs to run as cron jobs
This is an automated email from the ASF dual-hosted git repository.
kileysok pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3f75777 [BEAM-12572] Change jobs to run as cron jobs
new 93ef2f5 Merge pull request #16746 from benWize/BEAM-12572-cron-jobs
3f75777 is described below
commit 3f75777c7d5153ecc54d103fbe1e8dbd6227826a
Author: Benjamin Gonzalez
AuthorDate: Fri Feb 4 15:47:36 2022 -0600
[BEAM-12572] Change jobs to run as cron jobs
---
.../job_PostCommit_Python_Examples_Dataflow.groovy | 37 -
.../job_PostCommit_Python_Examples_Direct.groovy | 39 +-
.../job_PostCommit_Python_Examples_Flink.groovy| 39 +-
.../job_PostCommit_Python_Examples_Spark.groovy| 46 +++---
4 files changed, 75 insertions(+), 86 deletions(-)
diff --git a/.test-infra/jenkins/job_PostCommit_Python_Examples_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_Examples_Dataflow.groovy
index 89a9e92..4024025 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_Examples_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_Examples_Dataflow.groovy
@@ -17,30 +17,27 @@
*/
import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
// This job runs the Python examples tests with DataflowRunner.
-job('beam_PostCommit_Python_Examples_Dataflow') {
- description('Runs the Python Examples with DataflowRunner.')
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Examples_Dataflow',
+'Run Python Examples_Dataflow', 'Python Dataflow Runner Examples', this) {
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
+ description('Runs the Python Examples with DataflowRunner')
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Python Dataflow Runner Examples',
- 'Run Python Examples_Dataflow')
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
- publishers {
-archiveJunit('**/pytest*.xml')
- }
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/pytest*.xml')
+ }
- // Execute shell command to run examples.
- steps {
-gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(":sdks:python:test-suites:dataflow:examplesPostCommit")
- commonJobProperties.setGradleSwitches(delegate)
+ // Execute shell command to run examples.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(":sdks:python:test-suites:dataflow:examplesPostCommit")
+ commonJobProperties.setGradleSwitches(delegate)
+}
+ }
}
- }
-}
\ No newline at end of file
diff --git a/.test-infra/jenkins/job_PostCommit_Python_Examples_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Python_Examples_Direct.groovy
index 6b3edbe..e420b04 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_Examples_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_Examples_Direct.groovy
@@ -17,30 +17,27 @@
*/
import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
// This job runs the Python examples tests with DirectRunner.
-job('beam_PostCommit_Python_Examples_Direct') {
- description('Runs the Python Examples with DirectRunner.')
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Examples_Direct',
+'Run Python Examples_Direct', 'Python Direct Runner Examples', this) {
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
+ description('Runs the Python Examples with DirectRunner')
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Python Direct Runner Examples',
- 'Run Python Examples_Direct')
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
- publishers {
-archiveJunit('**/pytest*.xml')
- }
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/pytest*.xml')
+ }
- // Execute shell command to run examples.
- steps {
-gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(":sdks:python:test-suites:direct:examplesPostCommit")
- commonJobProperties.setGradleSwitches(delegate)
-}
- }
-}
\ No newline at end of file
+ // Execute shell command to run examples.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(":sdks:python:test-suites:direct:examplesPostCommit")
+ commonJobProperties.setGradleSwitches(delegate)
+}
+ }
+}
\ No newline at end of file
diff --git a
[beam] branch master updated (d3b680e -> d27d709)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d3b680e Merge pull request #16653 from [BEAM-12164]: Add integration tests for spanner change streams add d27d709 Merge pull request #16728 from [BEAM-13823] Update docs for SnowflakeIO No new revisions were added by this update. Summary of changes: .../en/documentation/io/built-in/snowflake.md | 249 ++--- website/www/site/data/io_matrix.yaml | 3 + 2 files changed, 170 insertions(+), 82 deletions(-)
[beam] branch master updated (64581e0 -> d3b680e)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 64581e0 [BEAM-13767] Migrate a bundle of grade tasks to use
configuration avoidance API. (#16648)
add d3b680e Merge pull request #16653 from [BEAM-12164]: Add integration
tests for spanner change streams
No new revisions were added by this update.
Summary of changes:
.../action/ChildPartitionsRecordAction.java| 1 -
.../action/QueryChangeStreamAction.java| 2 -
.../spanner/changestreams/dofn/InitializeDoFn.java | 7 +-
.../it/ChangeStreamTestPipelineOptions.java} | 25 +-
.../changestreams/it/IntegrationTestEnv.java | 203 +++
.../changestreams/it/SpannerChangeStreamIT.java| 212 +++
.../it/SpannerChangeStreamOrderedWithinKeyIT.java | 393 +
7 files changed, 827 insertions(+), 16 deletions(-)
copy
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/{bigtable/BigtableTestOptions.java
=> spanner/changestreams/it/ChangeStreamTestPipelineOptions.java} (59%)
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/IntegrationTestEnv.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/SpannerChangeStreamIT.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/SpannerChangeStreamOrderedWithinKeyIT.java
[beam] branch master updated (74d94ea -> 64581e0)
This is an automated email from the ASF dual-hosted git repository. yichi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 74d94ea [BEAM-13193] Aggregates fn api outbound data/timers of different endpoints (#16439) add 64581e0 [BEAM-13767] Migrate a bundle of grade tasks to use configuration avoidance API. (#16648) No new revisions were added by this update. Summary of changes: build.gradle.kts | 66 +++ .../org/apache/beam/gradle/BeamModulePlugin.groovy | 95 +++--- runners/flink/job-server/flink_job_server.gradle | 4 +- runners/google-cloud-dataflow-java/build.gradle| 12 +-- runners/portability/java/build.gradle | 6 +- runners/samza/job-server/build.gradle | 4 +- runners/spark/job-server/spark_job_server.gradle | 4 +- sdks/go/container/build.gradle | 2 +- sdks/go/test/build.gradle | 4 +- sdks/python/build.gradle | 28 +++ sdks/python/test-suites/direct/common.gradle | 2 +- sdks/python/test-suites/direct/xlang/build.gradle | 4 +- sdks/python/test-suites/portable/common.gradle | 58 +++-- 13 files changed, 149 insertions(+), 140 deletions(-)
[beam] branch master updated (98e5fc5 -> 74d94ea)
This is an automated email from the ASF dual-hosted git repository. yichi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 98e5fc5 [BEAM-11971] Revert "Fix timer consistency in direct runner" (#16748) add 74d94ea [BEAM-13193] Aggregates fn api outbound data/timers of different endpoints (#16439) No new revisions were added by this update. Summary of changes: .../worker/fn/data/BeamFnDataGrpcService.java | 10 +- .../runners/fnexecution/data/FnDataService.java| 5 + .../runners/fnexecution/data/GrpcDataService.java | 8 +- .../data/BeamFnDataBufferingOutboundObserver.java | 102 -- .../sdk/fn/data/BeamFnDataOutboundAggregator.java | 389 + .../sdk/fn/data/BeamFnDataOutboundObserver.java| 85 + ...amFnDataSizeBasedBufferingOutboundObserver.java | 144 ...amFnDataTimeBasedBufferingOutboundObserver.java | 119 --- .../fn/data/BeamFnDataOutboundAggregatorTest.java | 386 ...DataSizeBasedBufferingOutboundObserverTest.java | 210 --- ...DataTimeBasedBufferingOutboundObserverTest.java | 178 -- .../beam/fn/harness/BeamFnDataWriteRunner.java | 123 ++- .../apache/beam/fn/harness/FnApiDoFnRunner.java| 42 +-- .../beam/fn/harness/PTransformRunnerFactory.java | 19 +- .../fn/harness/control/ProcessBundleHandler.java | 113 +++--- .../beam/fn/harness/data/BeamFnDataClient.java | 20 +- .../beam/fn/harness/data/BeamFnDataGrpcClient.java | 38 +- .../beam/fn/harness/data/BeamFnTimerClient.java| 40 --- .../fn/harness/data/BeamFnTimerGrpcClient.java | 48 --- .../fn/harness/state/FnApiTimerBundleTracker.java | 5 +- .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 187 -- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 78 - .../PTransformRunnerFactoryTestContext.java| 76 +++- .../harness/control/ProcessBundleHandlerTest.java | 131 +++ .../fn/harness/data/BeamFnDataGrpcClientTest.java | 19 +- .../fn/harness/data/FakeBeamFnTimerClient.java | 68 26 files changed, 1278 insertions(+), 1365 deletions(-) delete mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java create mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java create mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java delete mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataSizeBasedBufferingOutboundObserver.java delete mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataTimeBasedBufferingOutboundObserver.java create mode 100644 sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregatorTest.java delete mode 100644 sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataSizeBasedBufferingOutboundObserverTest.java delete mode 100644 sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataTimeBasedBufferingOutboundObserverTest.java delete mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnTimerClient.java delete mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnTimerGrpcClient.java delete mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/FakeBeamFnTimerClient.java
[beam] branch master updated (7e2b637 -> 98e5fc5)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7e2b637 Merge pull request #16752: [BEAM-13147] Avoid nullness issue during init of AwsModule (AWS Sdk v2) add 98e5fc5 [BEAM-11971] Revert "Fix timer consistency in direct runner" (#16748) No new revisions were added by this update. Summary of changes: .../apache/beam/runners/core/TimerInternals.java | 17 +- .../beam/runners/direct/DirectTimerInternals.java | 72 +++--- .../direct/ExecutorServiceParallelExecutor.java| 2 +- .../beam/runners/direct/QuiescenceDriver.java | 78 +++ .../direct/StatefulParDoEvaluatorFactory.java | 83 +++ .../beam/runners/direct/WatermarkManager.java | 250 - .../apache/beam/runners/local/ExecutionDriver.java | 2 +- .../org/apache/beam/sdk/transforms/ParDoTest.java | 10 +- 8 files changed, 282 insertions(+), 232 deletions(-)
[beam] branch master updated (7bafabc -> 7e2b637)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7bafabc Merge pull request #16726 from [BEAM-12164]: Parses change streams fields as json / strings add 6b09d10 [BEAM-13147] Avoid nullness issue during init of AwsModule (AWS Sdk v2) new 7e2b637 Merge pull request #16752: [BEAM-13147] Avoid nullness issue during init of AwsModule (AWS Sdk v2) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/aws2/options/AwsModule.java | 16 ++-- 1 file changed, 10 insertions(+), 6 deletions(-)
[beam] 01/01: Merge pull request #16752: [BEAM-13147] Avoid nullness issue during init of AwsModule (AWS Sdk v2)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7e2b6370b7c98a820c905a5c19abd3cc486b9d77 Merge: 7bafabc 6b09d10 Author: Kenn Knowles AuthorDate: Mon Feb 7 06:00:00 2022 -0800 Merge pull request #16752: [BEAM-13147] Avoid nullness issue during init of AwsModule (AWS Sdk v2) .../org/apache/beam/sdk/io/aws2/options/AwsModule.java | 16 ++-- 1 file changed, 10 insertions(+), 6 deletions(-)
