[beam] branch master updated (c57c983 -> d39cb0a)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c57c983 Merge pull request #16074 from [BEAM-13323][Playground] [Bugfix] Fix CI checks add d39cb0a Merge pull request #16050 from [BEAM-13307][Playground] Support Python SDK No new revisions were added by this update. Summary of changes: playground/backend/cmd/server/controller.go| 2 +- playground/backend/configs/SDK_PYTHON.json | 6 + .../backend/containers/{java => python}/Dockerfile | 18 + playground/backend/containers/python/app.yaml | 22 .../containers/{java => python}/build.gradle | 7 +++-- .../containers/{java => python}/settings.gradle| 2 +- .../internal/code_processing/code_processing.go| 23 ++--- .../internal/environment/environment_service.go| 2 +- .../backend/internal/executors/executor_test.go| 2 +- playground/backend/internal/fs_tool/fs.go | 2 ++ .../backend/internal/fs_tool/lc_constructor.go | 18 + .../internal/fs_tool/{go_fs.go => python_fs.go}| 9 +++ .../fs_tool/{go_fs_test.go => python_fs_test.go} | 30 ++ .../{preparator.go => python_preparators.go} | 9 --- .../setup_tools/run_builder/run_builder_setuper.go | 2 ++ .../run_builder/run_builder_setuper_test.go| 4 +-- .../backend/internal/utils/validators_utils.go | 2 ++ .../{go_validators.go => python_validators.go} | 6 ++--- settings.gradle.kts| 1 + 19 files changed, 111 insertions(+), 56 deletions(-) create mode 100644 playground/backend/configs/SDK_PYTHON.json copy playground/backend/containers/{java => python}/Dockerfile (73%) create mode 100644 playground/backend/containers/python/app.yaml copy playground/backend/containers/{java => python}/build.gradle (88%) copy playground/backend/containers/{java => python}/settings.gradle (93%) copy playground/backend/internal/fs_tool/{go_fs.go => python_fs.go} (73%) copy playground/backend/internal/fs_tool/{go_fs_test.go => python_fs_test.go} (63%) copy playground/backend/internal/preparators/{preparator.go => python_preparators.go} (81%) copy playground/backend/internal/validators/{go_validators.go => python_validators.go} (83%)
[beam] branch master updated (d39cb0a -> 645e102)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d39cb0a Merge pull request #16050 from [BEAM-13307][Playground] Support Python SDK add 645e102 Merge pull request #16071 from [BEAM-13306][Playground] Add using of Preparators No new revisions were added by this update. Summary of changes: .../{python_preparators.go => go_preparators.go} | 6 ++ .../compile_builder/compile_builder_setuper.go | 6 ++ .../utils/{validators_utils.go => preparators_utils.go} | 16 3 files changed, 16 insertions(+), 12 deletions(-) copy playground/backend/internal/preparators/{python_preparators.go => go_preparators.go} (84%) copy playground/backend/internal/utils/{validators_utils.go => preparators_utils.go} (70%)
[beam] branch master updated: [BEAM-11943] Fix wrong Klio logo being displayed (#16078)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new f6e4aef [BEAM-11943] Fix wrong Klio logo being displayed (#16078) f6e4aef is described below commit f6e4aef7cc22ea934bde471fd2e2ea2c110721d4 Author: Alexander Dahl AuthorDate: Mon Nov 29 16:27:55 2021 +0100 [BEAM-11943] Fix wrong Klio logo being displayed (#16078) --- website/www/site/data/en/quotes.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/data/en/quotes.yaml b/website/www/site/data/en/quotes.yaml index 38700e5..16766fe 100644 --- a/website/www/site/data/en/quotes.yaml +++ b/website/www/site/data/en/quotes.yaml @@ -15,7 +15,7 @@ logoUrl: images/logos/powered-by/eBay.png - text: Developed at Spotify and built on top of Apache Beam, Klio is an open source framework for data processing pipelines for audio and other media files. icon: icons/quote-icon.svg - logoUrl: images/logos/powered-by/scio.png + logoUrl: images/logos/powered-by/klio.png - text: Oriel Research Therapeutics (ORT) is a startup company that utilizes Apache Beam to process over 1 million samples of genomic data to detect Leukemia, Sepsis and other medical conditions. icon: icons/quote-icon.svg logoUrl: images/logos/powered-by/oriel.png
[beam] branch release-2.35.0 updated (9d3a8c4 -> 3cbeaa0)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 9d3a8c4 [BEAM-13300] [BEAM-13073] Cherry-pick #16045 onto the 2.35.0 release branch. #16053 add 9621626 [BEAM-12829] Never copy .gogradle/** for Python. new 3cbeaa0 [BEAM-12829] Never copy .gogradle/** for Python. #16060 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: [BEAM-12829] Never copy .gogradle/** for Python. #16060
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch release-2.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 3cbeaa065bc9646628bc43f450598e1e8f71c7d3 Merge: 9d3a8c4 9621626 Author: tvalentyn AuthorDate: Mon Nov 29 09:50:38 2021 -0800 [BEAM-12829] Never copy .gogradle/** for Python. #16060 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 + 1 file changed, 1 insertion(+)
[beam] branch asf-site updated: Publishing website 2021/11/29 18:02:47 at commit f6e4aef
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 66e7c16 Publishing website 2021/11/29 18:02:47 at commit f6e4aef 66e7c16 is described below commit 66e7c162a80a73fdee38abc867076e9978374955 Author: jenkins AuthorDate: Mon Nov 29 18:02:48 2021 + Publishing website 2021/11/29 18:02:47 at commit f6e4aef --- website/generated-content/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/index.html b/website/generated-content/index.html index 8f60a60..e2ef0fd 100644 --- a/website/generated-content/index.html +++ b/website/generated-content/index.html @@ -25,7 +25,7 @@ function openMenu(){addPlaceholder();blockScroll();}SourcesBeam reads your data from a diverse set of supported sources, no matter if it’s on-prem or in the cloud.ProcessingBeam executes your business logic for Batch and Streaming use cases.& MOREChoose your runnerA Beam pipeline can execute in the most popular distributed data processing systems such as Spark, Flink or Samza. -& MOREChoose your languageYou can write Apache Beam pipelines in your programming language of choice: Java, Python and Go. Learn More.Stay up to date with Beamhttp://www.apache.org>The Apache Software Foundation | Privacy Policy | RSS FeedApache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.
[beam] branch master updated: Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new b09a028 Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference b09a028 is described below commit b09a028121b7ffca6bb6409e419df61d6c6c3dc1 Author: Pablo AuthorDate: Mon Nov 29 13:17:06 2021 -0500 Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference * [BEAM-960] support coder inference for JdbcIO * [BEAM-1675] Deprecate withCoder iin JdbcIO * Update JdbcIO integration test * Address comment --- .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 112 +++-- .../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 4 +- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 19 3 files changed, 101 insertions(+), 34 deletions(-) diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java index 83f620f..0f6b9c3 100644 --- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java +++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java @@ -47,7 +47,9 @@ import java.util.stream.IntStream; import javax.sql.DataSource; import org.apache.beam.sdk.annotations.Experimental; import org.apache.beam.sdk.annotations.Experimental.Kind; +import org.apache.beam.sdk.coders.CannotProvideCoderException; import org.apache.beam.sdk.coders.Coder; +import org.apache.beam.sdk.coders.CoderRegistry; import org.apache.beam.sdk.coders.RowCoder; import org.apache.beam.sdk.coders.VoidCoder; import org.apache.beam.sdk.io.jdbc.JdbcIO.WriteFn.WriteFnSpec; @@ -84,6 +86,8 @@ import org.apache.beam.sdk.values.PCollectionView; import org.apache.beam.sdk.values.PDone; import org.apache.beam.sdk.values.Row; import org.apache.beam.sdk.values.TypeDescriptor; +import org.apache.beam.sdk.values.TypeDescriptors; +import org.apache.beam.sdk.values.TypeDescriptors.TypeVariableExtractor; import org.apache.commons.dbcp2.BasicDataSource; import org.apache.commons.dbcp2.DataSourceConnectionFactory; import org.apache.commons.dbcp2.PoolableConnectionFactory; @@ -118,7 +122,6 @@ import org.slf4j.LoggerFactory; *.withUsername("username") *.withPassword("password")) * .withQuery("select id,name from Person") - * .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of())) * .withRowMapper(new JdbcIO.RowMapper>() { * public KV mapRow(ResultSet resultSet) throws Exception { * return KV.of(resultSet.getInt(1), resultSet.getString(2)); @@ -136,7 +139,6 @@ import org.slf4j.LoggerFactory; * "com.mysql.jdbc.Driver", "jdbc:mysql://hostname:3306/mydb", * "username", "password")) * .withQuery("select id,name from Person where name = ?") - * .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of())) * .withStatementPreparator(new JdbcIO.StatementPreparator() { * public void setParameters(PreparedStatement preparedStatement) throws Exception { * preparedStatement.setString(1, "Darwin"); @@ -202,7 +204,6 @@ import org.slf4j.LoggerFactory; * .withLowerBound(0) * .withUpperBound(1000) * .withNumPartitions(5) - * .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of())) * .withRowMapper(new JdbcIO.RowMapper>() { *public KV mapRow(ResultSet resultSet) throws Exception { * return KV.of(resultSet.getInt(1), resultSet.getString(2)); @@ -226,7 +227,6 @@ import org.slf4j.LoggerFactory; * .withLowerBound(0) * .withUpperBound(1000) * .withNumPartitions(5) - * .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of())) * .withRowMapper(new JdbcIO.RowMapper>() { *public KV mapRow(ResultSet resultSet) throws Exception { * return KV.of(resultSet.getInt(1), resultSet.getString(2)); @@ -737,6 +737,11 @@ public class JdbcIO { return toBuilder().setRowMapper(rowMapper).build(); } +/** + * @deprecated + * {@link JdbcIO} is able to infer aprppriate coders from other parameters. + */ +@Deprecated public Read withCoder(Coder coder) { checkArgument(coder != null, "coder can not be null"); return toBuilder().setCoder(coder).build(); @@ -764,27 +769,28 @@ public class JdbcIO { public PCollection expand(PBegin input) { checkArgument(getQuery() != null, "withQuery() is required"); checkArgument(getRowMapper() != null, "withRowMapper() is required"); - checkArgument(getCoder() != null, "withCoder() is required"); checkArgument( (getDataSourceProviderFn() != null), "withDataSourceConfiguration() or withDataSourceProviderFn() is required"); - return input - .
[beam] branch master updated (b09a028 -> d500129)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b09a028 Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference add d500129 [BEAM-13284] Respect expiration time for all Redis write methods. (#16052) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/redis/RedisIO.java | 16 ++-- .../org/apache/beam/sdk/io/redis/RedisIOTest.java | 45 ++ 2 files changed, 57 insertions(+), 4 deletions(-)
[beam] branch master updated: Increase timeout for Dataflow Streaming VR
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new f05bcfe Increase timeout for Dataflow Streaming VR new 3769d75 Merge pull request #16040 from kileys/BEAM-13238-timeout f05bcfe is described below commit f05bcfe9be1815bd063cc19bb5fd16ca36b30688 Author: kileys AuthorDate: Mon Nov 22 19:24:58 2021 + Increase timeout for Dataflow Streaming VR --- .../job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy index 2735db9..8a5afb3 100644 --- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy +++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy @@ -27,7 +27,7 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflo description('Runs the ValidatesRunner suite on the Dataflow runner forcing streaming mode.') - commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 480) + commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 540) // Publish all test results to Jenkins publishers {
[beam] branch master updated (3769d75 -> 9160ba2)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3769d75 Merge pull request #16040 from kileys/BEAM-13238-timeout add 9160ba2 [BEAM-12697] Add SBE module and initial classes (#15733) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/annotations/Experimental.java | 3 + .../sbe/build.gradle} | 25 +- .../beam/sdk/extensions/sbe/SbeLogicalTypes.java | 263 + .../beam/sdk/extensions/sbe}/package-info.java | 6 +- .../sdk/extensions/sbe/SbeLogicalTypesTest.java| 99 settings.gradle.kts| 1 + 6 files changed, 380 insertions(+), 17 deletions(-) copy sdks/java/{io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/data/text/SnowflakeString.java => extensions/sbe/build.gradle} (65%) create mode 100644 sdks/java/extensions/sbe/src/main/java/org/apache/beam/sdk/extensions/sbe/SbeLogicalTypes.java copy sdks/java/{io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis => extensions/sbe/src/main/java/org/apache/beam/sdk/extensions/sbe}/package-info.java (87%) create mode 100644 sdks/java/extensions/sbe/src/test/java/org/apache/beam/sdk/extensions/sbe/SbeLogicalTypesTest.java
[beam] branch master updated (9160ba2 -> 792a057)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9160ba2 [BEAM-12697] Add SBE module and initial classes (#15733) add 792a057 [Playground][Beam-13316][Bugfix] Fix Playground Frontend Precommit (#16075) No new revisions were added by this update. Summary of changes: .github/workflows/build_playground_frontend.yml | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (792a057 -> 2155c8c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 792a057 [Playground][Beam-13316][Bugfix] Fix Playground Frontend Precommit (#16075) add 2155c8c [BEAM-13115][Playground] Security – Mock Storage (#16047) No new revisions were added by this update. Summary of changes: playground/terraform/applications/backend/main.tf | 9 + playground/terraform/applications/backend/variables.tf | 12 2 files changed, 21 insertions(+)
[beam] branch master updated: [BEAM-13337] Periodically delete stale spanner databases.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 7a5bdcd [BEAM-13337] Periodically delete stale spanner databases. new 2b5087a Merge pull request #16082 from ibzib/BEAM-13337 7a5bdcd is described below commit 7a5bdcd3c8a5d990a82bdb24af2bdac285a23d60 Author: Kyle Weaver AuthorDate: Mon Nov 29 13:45:09 2021 -0800 [BEAM-13337] Periodically delete stale spanner databases. --- .test-infra/tools/stale_dataflow_jobs_cleaner.sh | 7 +++ 1 file changed, 7 insertions(+) diff --git a/.test-infra/tools/stale_dataflow_jobs_cleaner.sh b/.test-infra/tools/stale_dataflow_jobs_cleaner.sh index e563048..e6df705 100755 --- a/.test-infra/tools/stale_dataflow_jobs_cleaner.sh +++ b/.test-infra/tools/stale_dataflow_jobs_cleaner.sh @@ -27,3 +27,10 @@ if [[ ${STALE_JOBS} ]]; then else echo "No stale jobs found." fi + +# Delete spanner databases older than 1 day. +gcloud spanner databases list \ +--instance beam-test \ +--filter="createTime < $(date --iso-8601=s -d '1 day ago')" \ +--format="value(name)" | \ +xargs -I{} gcloud spanner databases delete {} --instance beam-test --quiet
[beam] branch master updated: [BEAM-13220] Update release instructions.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8d75633 [BEAM-13220] Update release instructions. new 5fc55f3 Merge pull request #16065 from ibzib/BEAM-13220 8d75633 is described below commit 8d75633d0c082b1151405cf364c9370bfc95f596 Author: Kyle Weaver AuthorDate: Wed Nov 24 15:24:04 2021 -0800 [BEAM-13220] Update release instructions. The biggest change here is to run Java wordcount and mobile gaming examples on Jenkins instead of the bash script, since getting the script to work properly took too long. --- .../main/scripts/publish_github_release_notes.sh | 2 +- release/src/main/scripts/run_rc_validation.sh | 128 + release/src/main/scripts/script.config | 2 - .../site/content/en/contribute/release-guide.md| 20 ++-- 4 files changed, 14 insertions(+), 138 deletions(-) diff --git a/release/src/main/scripts/publish_github_release_notes.sh b/release/src/main/scripts/publish_github_release_notes.sh index 7f58648..09fec15 100755 --- a/release/src/main/scripts/publish_github_release_notes.sh +++ b/release/src/main/scripts/publish_github_release_notes.sh @@ -33,7 +33,7 @@ RELEASE_NOTES=$( ) # Escape notes' content to work with JSON -ESCAPED_NOTES=$(printf '%s' "${RELEASE_NOTES}" | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))') +ESCAPED_NOTES=$(printf '%s' "${RELEASE_NOTES}" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))') # Build JSON for the API request REQUEST_JSON="$(cat <<-EOF diff --git a/release/src/main/scripts/run_rc_validation.sh b/release/src/main/scripts/run_rc_validation.sh index baff09b..28ba2eb 100755 --- a/release/src/main/scripts/run_rc_validation.sh +++ b/release/src/main/scripts/run_rc_validation.sh @@ -207,133 +207,6 @@ kubectl version echo "" -echo "" -echo "Starting Java Quickstart===" -echo "[Current task] Java quickstart with direct runner" -if [[ "$java_quickstart_direct" = true ]]; then - echo "*" - echo "* Running Java Quickstart with DirectRunner" - echo "*" - ./gradlew :runners:direct-java:runQuickstartJavaDirect \ - -Prepourl=${REPO_URL} \ - -Pver=${RELEASE_VER} -else - echo "* Skip Java quickstart with direct runner" -fi - -echo "[Current task] Java quickstart with Flink local runner" -if [[ "$java_quickstart_flink_local" = true ]]; then - echo "*" - echo "* Running Java Quickstart with Flink local runner" - echo "*" - ./gradlew :runners:flink:1.13:runQuickstartJavaFlinkLocal \ - -Prepourl=${REPO_URL} \ - -Pver=${RELEASE_VER} -else - echo "* Skip Java quickstart with Flink local runner" -fi - -echo "[Current task] Java quickstart with Spark local runner" -if [[ "$java_quickstart_spark_local" = true ]]; then - echo "*" - echo "* Running Java Quickstart with Spark local runner" - echo "*" - ./gradlew :runners:spark:2:runQuickstartJavaSpark \ - -Prepourl=${REPO_URL} \ - -Pver=${RELEASE_VER} -else - echo "* Skip Java quickstart with Spark local runner" -fi - -echo "[Current task] Java quickstart with Dataflow runner" -if [[ "$java_quickstart_dataflow" = true && ! -z `which gcloud` ]]; then - echo "*" - echo "* Running Java Quickstart with DataflowRunner" - echo "*" - ./gradlew :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow \ - -Prepourl=${REPO_URL} \ - -Pver=${RELEASE_VER} \ - -PgcpProject=${USER_GCP_PROJECT} \ - -PgcsBucket=${USER_GCS_BUCKET:5} # skip 'gs://' prefix -else - echo "* Skip Java quickstart with Dataflow runner. Google Cloud SDK is required." -fi - -echo "[Current task] Java quickstart with Twister2 local runner" -if [[ "$java_quickstart_twister2_local" = true ]]; then - echo "*" - echo "* Running Java Quickstart with Twister2 local runner" - echo "*" - ./gradlew :runners:twister2:runQuickstartJavaTwister2 \ - -Prepourl=${REPO_URL} \ - -Pver=${RELEASE_VER} -else - echo "* Skip Java quickstart with Twister2 local runner" -fi - -echo "" -echo "Starting Java Mobile Game=" -if [[ ("$java_mobile_game_direct" = true || "$java_mobile_game_dataflow" = true) \ - && ! -z `which gcloud` ]]; then - MOBILE_GAME_DATASET=${USER}_java_v
[beam] branch master updated (5fc55f3 -> e6734b9)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5fc55f3 Merge pull request #16065 from ibzib/BEAM-13220 add e6734b9 [BEAM-13335] Use shorter, numerical indices for dataframe reads. (#16066) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/io.py | 39 + 1 file changed, 30 insertions(+), 9 deletions(-)
[beam] branch master updated (e6734b9 -> 3ce6138)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e6734b9 [BEAM-13335] Use shorter, numerical indices for dataframe reads. (#16066) add 3ce6138 Be more conservative about rebatching. (#16058) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/transforms.py | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-)
[beam] branch master updated (3ce6138 -> 33bcdf6)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3ce6138 Be more conservative about rebatching. (#16058) add 33bcdf6 [BEAM-12587] Allow None in Python's Any logical type. (#16055) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/convert_test.py | 7 +++ sdks/python/apache_beam/typehints/schemas.py | 8 ++-- sdks/python/apache_beam/typehints/schemas_test.py | 3 ++- 3 files changed, 15 insertions(+), 3 deletions(-)
[beam] branch revert-15969-combine-type-hints created (now d71203a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch revert-15969-combine-type-hints in repository https://gitbox.apache.org/repos/asf/beam.git. at d71203a Revert "Key-inferable type hints for CombinePerKey." This branch includes the following new commits: new d71203a Revert "Key-inferable type hints for CombinePerKey." The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Revert "Key-inferable type hints for CombinePerKey."
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch revert-15969-combine-type-hints in repository https://gitbox.apache.org/repos/asf/beam.git commit d71203a01307008114238a8b4506e67ae24a5ad7 Author: tvalentyn AuthorDate: Mon Nov 29 15:19:04 2021 -0800 Revert "Key-inferable type hints for CombinePerKey." --- sdks/python/apache_beam/pipeline.py| 4 +--- sdks/python/apache_beam/transforms/core.py | 24 ++ .../apache_beam/transforms/ptransform_test.py | 13 3 files changed, 12 insertions(+), 29 deletions(-) diff --git a/sdks/python/apache_beam/pipeline.py b/sdks/python/apache_beam/pipeline.py index a512a18..25c02d4 100644 --- a/sdks/python/apache_beam/pipeline.py +++ b/sdks/python/apache_beam/pipeline.py @@ -892,9 +892,7 @@ class Pipeline(object): pcoll.element_type) if (isinstance(output.element_type, typehints.TupleHint.TupleConstraint) and -len(output.element_type.tuple_types) == 2 and -pcoll.element_type.tuple_types[0] == -output.element_type.tuple_types[0]): +len(output.element_type.tuple_types) == 2): output.requires_deterministic_key_coder = ( deterministic_key_coders and transform_node.full_label) for side_input in transform_node.transform.side_inputs: diff --git a/sdks/python/apache_beam/transforms/core.py b/sdks/python/apache_beam/transforms/core.py index bd999f8..c71c00c 100644 --- a/sdks/python/apache_beam/transforms/core.py +++ b/sdks/python/apache_beam/transforms/core.py @@ -2306,20 +2306,18 @@ class CombinePerKey(PTransformWithSideInputs): self.fn, *args, **kwargs) def default_type_hints(self): -result = self.fn.get_type_hints() -k = typehints.TypeVariable('K') -if result.input_types: - args, kwargs = result.input_types - args = (typehints.Tuple[k, args[0]], ) + args[1:] - result = result.with_input_types(*args, **kwargs) -else: - result = result.with_input_types(typehints.Tuple[k, typehints.Any]) -if result.output_types: - main_output_type = result.simple_output_type('') - result = result.with_output_types(typehints.Tuple[k, main_output_type]) +hints = self.fn.get_type_hints() +if hints.input_types: + K = typehints.TypeVariable('K') + args, kwargs = hints.input_types + args = (typehints.Tuple[K, args[0]], ) + args[1:] + hints = hints.with_input_types(*args, **kwargs) else: - result = result.with_output_types(typehints.Tuple[k, typehints.Any]) -return result + K = typehints.Any +if hints.output_types: + main_output_type = hints.simple_output_type('') + hints = hints.with_output_types(typehints.Tuple[K, main_output_type]) +return hints def to_runner_api_parameter( self, diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py b/sdks/python/apache_beam/transforms/ptransform_test.py index 6a139eb..ac4f632 100644 --- a/sdks/python/apache_beam/transforms/ptransform_test.py +++ b/sdks/python/apache_beam/transforms/ptransform_test.py @@ -1839,19 +1839,6 @@ class PTransformTypeCheckTestCase(TypeHintTestCase): assert_that(d, equal_to([6])) self.p.run() - def test_combine_properly_pipeline_type_checks_without_decorator(self): -def sum_ints(ints): - return sum(ints) - -d = ( -self.p -| beam.Create([1, 2, 3]) -| beam.Map(lambda x: ('key', x)) -| beam.CombinePerKey(sum_ints)) - -self.assertEqual(typehints.Tuple[str, typehints.Any], d.element_type) -self.p.run() - def test_combine_func_type_hint_does_not_take_iterable_using_decorator(self): @with_output_types(int) @with_input_types(a=int)
[beam] branch asf-site updated: Publishing website 2021/11/30 00:03:25 at commit 33bcdf6
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 924a7c7 Publishing website 2021/11/30 00:03:25 at commit 33bcdf6 924a7c7 is described below commit 924a7c77db1c79a5f8ccb07cfbd318fc2d5a100b Author: jenkins AuthorDate: Tue Nov 30 00:03:26 2021 + Publishing website 2021/11/30 00:03:25 at commit 33bcdf6 --- website/generated-content/contribute/index.xml | 18 +++--- .../contribute/release-guide/index.html| 16 website/generated-content/sitemap.xml | 2 +- 3 files changed, 20 insertions(+), 16 deletions(-) diff --git a/website/generated-content/contribute/index.xml b/website/generated-content/contribute/index.xml index 44366ae..4da81a2 100644 --- a/website/generated-content/contribute/index.xml +++ b/website/generated-content/contribute/index.xml @@ -498,11 +498,11 @@ If you don’t have a GPG key associated with your Apache account, please createThis will list your GPG keys. One of these should reflect your Apache account, for example:
--- -pub 2048R/845E6689 2016-02-23 +pub rsa4096/845E6689845E6689 2016-02-23 uid Nomen Nescio <anonym...@apache.org> -sub 2048R/BA4D50BE 2016-02-23 +sub rsa4096/BA4D50BEBA4D50BE 2016-02-23
Here, the key ID is the 8-digit hex string in the
+pub
line:845E6689
.Here, the key ID is the 16-digit hex string in the
pub
line:845E6689845E6689
.Access to Apache Nexus repository
@@ -911,7 +911,7 @@ is perfectly safe since the script does not depend on the current working tree.&Usage
-./beam/release/src/main/scripts/build_release_candidate.sh +
./beam/release/src/main/scripts/build_release_candidate.sh --release "${RELEASE_VERSION}" --rc "${RC_NUM}" --github-user "${GITHUB_USER}"
@@ -1164,8 +1164,6 @@ versions to run all of the tests. See Python installation tips in Check whether validations succeed by following console output instructions. Terminate streaming jobs and java injector. +Run Java quickstart (wordcount) and mobile game examples with the staged artifacts. The easiest way to do this is by running the tests on Jenkins. +Log in to Jenkins. +Go to https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/. +Click “Build with Parameters”. +Set +snapshot_version
to2.xx.0
, and setsnapshot_url
to point to the staged artifacts in Maven central (https://repository.apache.org/content/repositories/orgapachebeam-/).Click “Build”. Sign up spreadsheet. Vote in the release thread. @@ -1482,7 +1486,7 @@ git push https://github.com/apache/beam "$VERSION_TAG"After pushing the tag, the tag should be visible on Github’s Tags page.
Publish release to Github
Once the tag is uploaded, publish the release notes to Github, as follows:
-cd beam/release/src/main/scripts && ./publish_github_release_notes.sh +
./beam/release/src/main/scripts/publish_github_release_notes.sh
Note this script reads the release notes from the blog post, so you should make sure to run this from master after merging the blog post PR.
After running the script, the release notes should be visible on Github’s Releases page.
PMC-Only Finalization
diff --git a/website/generated-content/contribute/release-guide/index.html b/website/generated-content
[beam] annotated tag sdks/v2.35.0-RC1 updated (48fdc28 -> 46de4fc)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to annotated tag sdks/v2.35.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag sdks/v2.35.0-RC1 was modified! *** from 48fdc28 (commit) to 46de4fc (tag) tagging 48fdc289bcd428af2454f6b5b7339bc6ec326124 (commit) replaces jupyterlab-sidepanel-v2.0.0 by Valentyn Tymofieiev on Mon Nov 29 17:38:36 2021 -0800 - Log - Go SDK v2.35.0-RC1 --- No new revisions were added by this update. Summary of changes:
[beam] annotated tag v2.35.0-RC1 updated (48fdc28 -> 30d54af)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to annotated tag v2.35.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.35.0-RC1 was modified! *** from 48fdc28 (commit) to 30d54af (tag) tagging 48fdc289bcd428af2454f6b5b7339bc6ec326124 (commit) replaces jupyterlab-sidepanel-v2.0.0 by Valentyn Tymofieiev on Mon Nov 29 17:38:36 2021 -0800 - Log - v2.35.0-RC1 --- No new revisions were added by this update. Summary of changes:
[beam] branch nightly-refs/heads/master updated (c57c983 -> 33bcdf6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git. from c57c983 Merge pull request #16074 from [BEAM-13323][Playground] [Bugfix] Fix CI checks add d39cb0a Merge pull request #16050 from [BEAM-13307][Playground] Support Python SDK add 645e102 Merge pull request #16071 from [BEAM-13306][Playground] Add using of Preparators add f6e4aef [BEAM-11943] Fix wrong Klio logo being displayed (#16078) add b09a028 Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference add d500129 [BEAM-13284] Respect expiration time for all Redis write methods. (#16052) add f05bcfe Increase timeout for Dataflow Streaming VR add 3769d75 Merge pull request #16040 from kileys/BEAM-13238-timeout add 9160ba2 [BEAM-12697] Add SBE module and initial classes (#15733) add 792a057 [Playground][Beam-13316][Bugfix] Fix Playground Frontend Precommit (#16075) add 2155c8c [BEAM-13115][Playground] Security – Mock Storage (#16047) add 7a5bdcd [BEAM-13337] Periodically delete stale spanner databases. add 2b5087a Merge pull request #16082 from ibzib/BEAM-13337 add 8d75633 [BEAM-13220] Update release instructions. add 5fc55f3 Merge pull request #16065 from ibzib/BEAM-13220 add e6734b9 [BEAM-13335] Use shorter, numerical indices for dataframe reads. (#16066) add 3ce6138 Be more conservative about rebatching. (#16058) add 33bcdf6 [BEAM-12587] Allow None in Python's Any logical type. (#16055) No new revisions were added by this update. Summary of changes: .github/workflows/build_playground_frontend.yml| 3 + ..._Java_ValidatesRunner_Dataflow_Streaming.groovy | 2 +- .test-infra/tools/stale_dataflow_jobs_cleaner.sh | 7 + playground/backend/cmd/server/controller.go| 2 +- playground/backend/configs/SDK_PYTHON.json | 6 + .../backend/containers/{java => python}/Dockerfile | 18 +- playground/backend/containers/python/app.yaml | 22 ++ .../containers/{java => python}/build.gradle | 7 +- .../containers/{java => python}/settings.gradle| 2 +- .../internal/code_processing/code_processing.go| 23 +- .../internal/environment/environment_service.go| 2 +- .../backend/internal/executors/executor_test.go| 2 +- playground/backend/internal/fs_tool/fs.go | 2 + .../backend/internal/fs_tool/lc_constructor.go | 18 ++ .../internal/fs_tool/{go_fs.go => python_fs.go}| 9 +- .../fs_tool/{go_fs_test.go => python_fs_test.go} | 30 ++- .../{preparator.go => go_preparators.go} | 7 +- .../{preparator.go => python_preparators.go} | 9 +- .../compile_builder/compile_builder_setuper.go | 6 + .../setup_tools/run_builder/run_builder_setuper.go | 2 + .../run_builder/run_builder_setuper_test.go| 4 +- .../{validators_utils.go => preparators_utils.go} | 16 +- .../backend/internal/utils/validators_utils.go | 2 + .../{go_validators.go => python_validators.go} | 6 +- playground/terraform/applications/backend/main.tf | 9 + .../terraform/applications/backend/variables.tf| 12 + .../main/scripts/publish_github_release_notes.sh | 2 +- release/src/main/scripts/run_rc_validation.sh | 128 +- release/src/main/scripts/script.config | 2 - .../apache/beam/sdk/annotations/Experimental.java | 3 + .../sbe/build.gradle} | 25 +- .../beam/sdk/extensions/sbe/SbeLogicalTypes.java | 263 + .../beam/sdk/extensions/sbe}/package-info.java | 6 +- .../sdk/extensions/sbe/SbeLogicalTypesTest.java| 99 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 112 ++--- .../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 4 +- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 19 ++ .../java/org/apache/beam/sdk/io/redis/RedisIO.java | 16 +- .../org/apache/beam/sdk/io/redis/RedisIOTest.java | 45 sdks/python/apache_beam/dataframe/convert_test.py | 7 + sdks/python/apache_beam/dataframe/io.py| 39 ++- sdks/python/apache_beam/dataframe/transforms.py| 23 +- sdks/python/apache_beam/typehints/schemas.py | 8 +- sdks/python/apache_beam/typehints/schemas_test.py | 3 +- settings.gradle.kts| 2 + .../site/content/en/contribute/release-guide.md| 20 +- website/www/site/data/en/quotes.yaml | 2 +- 47 files changed, 776 insertions(+), 280 deletions(-) create mode 100644 playground/backend/configs/SDK_PYTHON.json copy playground/backend/containers/{java => python}/Dockerfile (73%) create mode 100644 playground/backend/containers/python/app.yaml copy playground/backend/containers/{java => python}/build.gradle (88%) copy playground/backend/containers/{java => python}