[beam] branch master updated (c57c983 -> d39cb0a)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c57c983  Merge pull request #16074 from [BEAM-13323][Playground] 
[Bugfix] Fix CI checks
 add d39cb0a  Merge pull request #16050 from [BEAM-13307][Playground] 
Support Python SDK

No new revisions were added by this update.

Summary of changes:
 playground/backend/cmd/server/controller.go|  2 +-
 playground/backend/configs/SDK_PYTHON.json |  6 +
 .../backend/containers/{java => python}/Dockerfile | 18 +
 playground/backend/containers/python/app.yaml  | 22 
 .../containers/{java => python}/build.gradle   |  7 +++--
 .../containers/{java => python}/settings.gradle|  2 +-
 .../internal/code_processing/code_processing.go| 23 ++---
 .../internal/environment/environment_service.go|  2 +-
 .../backend/internal/executors/executor_test.go|  2 +-
 playground/backend/internal/fs_tool/fs.go  |  2 ++
 .../backend/internal/fs_tool/lc_constructor.go | 18 +
 .../internal/fs_tool/{go_fs.go => python_fs.go}|  9 +++
 .../fs_tool/{go_fs_test.go => python_fs_test.go}   | 30 ++
 .../{preparator.go => python_preparators.go}   |  9 ---
 .../setup_tools/run_builder/run_builder_setuper.go |  2 ++
 .../run_builder/run_builder_setuper_test.go|  4 +--
 .../backend/internal/utils/validators_utils.go |  2 ++
 .../{go_validators.go => python_validators.go} |  6 ++---
 settings.gradle.kts|  1 +
 19 files changed, 111 insertions(+), 56 deletions(-)
 create mode 100644 playground/backend/configs/SDK_PYTHON.json
 copy playground/backend/containers/{java => python}/Dockerfile (73%)
 create mode 100644 playground/backend/containers/python/app.yaml
 copy playground/backend/containers/{java => python}/build.gradle (88%)
 copy playground/backend/containers/{java => python}/settings.gradle (93%)
 copy playground/backend/internal/fs_tool/{go_fs.go => python_fs.go} (73%)
 copy playground/backend/internal/fs_tool/{go_fs_test.go => python_fs_test.go} 
(63%)
 copy playground/backend/internal/preparators/{preparator.go => 
python_preparators.go} (81%)
 copy playground/backend/internal/validators/{go_validators.go => 
python_validators.go} (83%)

[beam] branch master updated (d39cb0a -> 645e102)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d39cb0a  Merge pull request #16050 from [BEAM-13307][Playground] 
Support Python SDK
 add 645e102  Merge pull request #16071 from [BEAM-13306][Playground] Add 
using of Preparators

No new revisions were added by this update.

Summary of changes:
 .../{python_preparators.go => go_preparators.go} |  6 ++
 .../compile_builder/compile_builder_setuper.go   |  6 ++
 .../utils/{validators_utils.go => preparators_utils.go}  | 16 
 3 files changed, 16 insertions(+), 12 deletions(-)
 copy playground/backend/internal/preparators/{python_preparators.go => 
go_preparators.go} (84%)
 copy playground/backend/internal/utils/{validators_utils.go => 
preparators_utils.go} (70%)

[beam] branch master updated: [BEAM-11943] Fix wrong Klio logo being displayed (#16078)

[bhulette]

This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new f6e4aef  [BEAM-11943] Fix wrong Klio logo being displayed (#16078)
f6e4aef is described below

commit f6e4aef7cc22ea934bde471fd2e2ea2c110721d4
Author: Alexander Dahl 
AuthorDate: Mon Nov 29 16:27:55 2021 +0100

[BEAM-11943] Fix wrong Klio logo being displayed (#16078)
---
 website/www/site/data/en/quotes.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/website/www/site/data/en/quotes.yaml 
b/website/www/site/data/en/quotes.yaml
index 38700e5..16766fe 100644
--- a/website/www/site/data/en/quotes.yaml
+++ b/website/www/site/data/en/quotes.yaml
@@ -15,7 +15,7 @@
   logoUrl: images/logos/powered-by/eBay.png
 - text: Developed at Spotify and built on top of Apache Beam, Klio is an open 
source framework for data processing pipelines for audio and other media files.
   icon: icons/quote-icon.svg
-  logoUrl: images/logos/powered-by/scio.png
+  logoUrl: images/logos/powered-by/klio.png
 - text: Oriel Research Therapeutics (ORT) is a startup company that utilizes 
Apache Beam to process over 1 million samples of genomic data to detect 
Leukemia, Sepsis and other medical conditions.
   icon: icons/quote-icon.svg
   logoUrl: images/logos/powered-by/oriel.png

[beam] branch release-2.35.0 updated (9d3a8c4 -> 3cbeaa0)

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch release-2.35.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9d3a8c4  [BEAM-13300] [BEAM-13073] Cherry-pick #16045 onto the 2.35.0 
release branch. #16053
 add 9621626  [BEAM-12829] Never copy .gogradle/** for Python.
 new 3cbeaa0   [BEAM-12829] Never copy .gogradle/** for Python. #16060

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
 1 file changed, 1 insertion(+)

[beam] 01/01: [BEAM-12829] Never copy .gogradle/** for Python. #16060

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch release-2.35.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3cbeaa065bc9646628bc43f450598e1e8f71c7d3
Merge: 9d3a8c4 9621626
Author: tvalentyn 
AuthorDate: Mon Nov 29 09:50:38 2021 -0800

 [BEAM-12829] Never copy .gogradle/** for Python. #16060

 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
 1 file changed, 1 insertion(+)

[beam] branch asf-site updated: Publishing website 2021/11/29 18:02:47 at commit f6e4aef

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 66e7c16  Publishing website 2021/11/29 18:02:47 at commit f6e4aef
66e7c16 is described below

commit 66e7c162a80a73fdee38abc867076e9978374955
Author: jenkins 
AuthorDate: Mon Nov 29 18:02:48 2021 +

Publishing website 2021/11/29 18:02:47 at commit f6e4aef
---
 website/generated-content/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/website/generated-content/index.html 
b/website/generated-content/index.html
index 8f60a60..e2ef0fd 100644
--- a/website/generated-content/index.html
+++ b/website/generated-content/index.html
@@ -25,7 +25,7 @@ function 
openMenu(){addPlaceholder();blockScroll();}SourcesBeam reads your data from a diverse set of supported 
sources, no matter if it’s on-prem or in the cloud.ProcessingBeam executes your business logic for Batch and 
Streaming use cases.& 
MOREChoose your runnerA Beam pipeline can execute in the 
most popular distributed data processing systems such as Spark, Flink or 
Samza.
-& 
MOREChoose your languageYou can write Apache Beam 
pipelines in your programming language of choice: Java, Python and Go. Learn 
More.Stay up to date with Beamhttp://www.apache.org>The Apache Software Foundation
 | Privacy Policy
 | RSS FeedApache Beam, Apache, Beam, the Beam 
logo, and the Apache feather logo are either registered trademarks or 
trademarks of The Apache Software Foundation. All other products or name brands 
are trademarks of their respective holders, including The Apache Software 
Foundation.

[beam] branch master updated: Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO coder inference

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new b09a028  Merge pull request #15954 from [BEAM-960][BEAM-1675] 
Improvements to JdbcIO coder inference
b09a028 is described below

commit b09a028121b7ffca6bb6409e419df61d6c6c3dc1
Author: Pablo 
AuthorDate: Mon Nov 29 13:17:06 2021 -0500

Merge pull request #15954 from [BEAM-960][BEAM-1675] Improvements to JdbcIO 
coder inference

* [BEAM-960] support coder inference for JdbcIO

* [BEAM-1675] Deprecate withCoder iin JdbcIO

* Update JdbcIO integration test

* Address comment
---
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   | 112 +++--
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java |   4 +-
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java|  19 
 3 files changed, 101 insertions(+), 34 deletions(-)

diff --git 
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java 
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 83f620f..0f6b9c3 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -47,7 +47,9 @@ import java.util.stream.IntStream;
 import javax.sql.DataSource;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.coders.RowCoder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import org.apache.beam.sdk.io.jdbc.JdbcIO.WriteFn.WriteFnSpec;
@@ -84,6 +86,8 @@ import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.Row;
 import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.sdk.values.TypeDescriptors.TypeVariableExtractor;
 import org.apache.commons.dbcp2.BasicDataSource;
 import org.apache.commons.dbcp2.DataSourceConnectionFactory;
 import org.apache.commons.dbcp2.PoolableConnectionFactory;
@@ -118,7 +122,6 @@ import org.slf4j.LoggerFactory;
  *.withUsername("username")
  *.withPassword("password"))
  *   .withQuery("select id,name from Person")
- *   .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
  *   .withRowMapper(new JdbcIO.RowMapper>() {
  * public KV mapRow(ResultSet resultSet) throws Exception 
{
  *   return KV.of(resultSet.getInt(1), resultSet.getString(2));
@@ -136,7 +139,6 @@ import org.slf4j.LoggerFactory;
  *   "com.mysql.jdbc.Driver", "jdbc:mysql://hostname:3306/mydb",
  *   "username", "password"))
  *   .withQuery("select id,name from Person where name = ?")
- *   .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
  *   .withStatementPreparator(new JdbcIO.StatementPreparator() {
  * public void setParameters(PreparedStatement preparedStatement) throws 
Exception {
  *   preparedStatement.setString(1, "Darwin");
@@ -202,7 +204,6 @@ import org.slf4j.LoggerFactory;
  *  .withLowerBound(0)
  *  .withUpperBound(1000)
  *  .withNumPartitions(5)
- *  .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
  *  .withRowMapper(new JdbcIO.RowMapper>() {
  *public KV mapRow(ResultSet resultSet) throws Exception {
  *  return KV.of(resultSet.getInt(1), resultSet.getString(2));
@@ -226,7 +227,6 @@ import org.slf4j.LoggerFactory;
  *  .withLowerBound(0)
  *  .withUpperBound(1000)
  *  .withNumPartitions(5)
- *  .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
  *  .withRowMapper(new JdbcIO.RowMapper>() {
  *public KV mapRow(ResultSet resultSet) throws Exception {
  *  return KV.of(resultSet.getInt(1), resultSet.getString(2));
@@ -737,6 +737,11 @@ public class JdbcIO {
   return toBuilder().setRowMapper(rowMapper).build();
 }
 
+/**
+ * @deprecated
+ * {@link JdbcIO} is able to infer aprppriate coders from other 
parameters.
+ */
+@Deprecated
 public Read withCoder(Coder coder) {
   checkArgument(coder != null, "coder can not be null");
   return toBuilder().setCoder(coder).build();
@@ -764,27 +769,28 @@ public class JdbcIO {
 public PCollection expand(PBegin input) {
   checkArgument(getQuery() != null, "withQuery() is required");
   checkArgument(getRowMapper() != null, "withRowMapper() is required");
-  checkArgument(getCoder() != null, "withCoder() is required");
   checkArgument(
   (getDataSourceProviderFn() != null),
   "withDataSourceConfiguration() or withDataSourceProviderFn() is 
required");
 
-  return input
-  .

[beam] branch master updated (b09a028 -> d500129)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b09a028  Merge pull request #15954 from [BEAM-960][BEAM-1675] 
Improvements to JdbcIO coder inference
 add d500129  [BEAM-13284] Respect expiration time for all Redis write 
methods. (#16052)

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/beam/sdk/io/redis/RedisIO.java | 16 ++--
 .../org/apache/beam/sdk/io/redis/RedisIOTest.java  | 45 ++
 2 files changed, 57 insertions(+), 4 deletions(-)

[beam] branch master updated: Increase timeout for Dataflow Streaming VR

[ibzib]

This is an automated email from the ASF dual-hosted git repository.

ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new f05bcfe  Increase timeout for Dataflow Streaming VR
 new 3769d75  Merge pull request #16040 from kileys/BEAM-13238-timeout
f05bcfe is described below

commit f05bcfe9be1815bd063cc19bb5fd16ca36b30688
Author: kileys 
AuthorDate: Mon Nov 22 19:24:58 2021 +

Increase timeout for Dataflow Streaming VR
---
 .../job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
 
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
index 2735db9..8a5afb3 100644
--- 
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
+++ 
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
@@ -27,7 +27,7 @@ 
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflo
 
   description('Runs the ValidatesRunner suite on the Dataflow runner 
forcing streaming mode.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 480)
+  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 540)
 
   // Publish all test results to Jenkins
   publishers {

[beam] branch master updated (3769d75 -> 9160ba2)

[bhulette]

This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3769d75  Merge pull request #16040 from kileys/BEAM-13238-timeout
 add 9160ba2  [BEAM-12697] Add SBE module and initial classes (#15733)

No new revisions were added by this update.

Summary of changes:
 .../apache/beam/sdk/annotations/Experimental.java  |   3 +
 .../sbe/build.gradle}  |  25 +-
 .../beam/sdk/extensions/sbe/SbeLogicalTypes.java   | 263 +
 .../beam/sdk/extensions/sbe}/package-info.java |   6 +-
 .../sdk/extensions/sbe/SbeLogicalTypesTest.java|  99 
 settings.gradle.kts|   1 +
 6 files changed, 380 insertions(+), 17 deletions(-)
 copy 
sdks/java/{io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/data/text/SnowflakeString.java
 => extensions/sbe/build.gradle} (65%)
 create mode 100644 
sdks/java/extensions/sbe/src/main/java/org/apache/beam/sdk/extensions/sbe/SbeLogicalTypes.java
 copy 
sdks/java/{io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis
 => 
extensions/sbe/src/main/java/org/apache/beam/sdk/extensions/sbe}/package-info.java
 (87%)
 create mode 100644 
sdks/java/extensions/sbe/src/test/java/org/apache/beam/sdk/extensions/sbe/SbeLogicalTypesTest.java

[beam] branch master updated (9160ba2 -> 792a057)

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9160ba2  [BEAM-12697] Add SBE module and initial classes (#15733)
 add 792a057  [Playground][Beam-13316][Bugfix] Fix Playground Frontend 
Precommit (#16075)

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build_playground_frontend.yml | 3 +++
 1 file changed, 3 insertions(+)

[beam] branch master updated (792a057 -> 2155c8c)

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 792a057  [Playground][Beam-13316][Bugfix] Fix Playground Frontend 
Precommit (#16075)
 add 2155c8c  [BEAM-13115][Playground] Security – Mock Storage (#16047)

No new revisions were added by this update.

Summary of changes:
 playground/terraform/applications/backend/main.tf  |  9 +
 playground/terraform/applications/backend/variables.tf | 12 
 2 files changed, 21 insertions(+)

[beam] branch master updated: [BEAM-13337] Periodically delete stale spanner databases.

[ibzib]

This is an automated email from the ASF dual-hosted git repository.

ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 7a5bdcd  [BEAM-13337] Periodically delete stale spanner databases.
 new 2b5087a  Merge pull request #16082 from ibzib/BEAM-13337
7a5bdcd is described below

commit 7a5bdcd3c8a5d990a82bdb24af2bdac285a23d60
Author: Kyle Weaver 
AuthorDate: Mon Nov 29 13:45:09 2021 -0800

[BEAM-13337] Periodically delete stale spanner databases.
---
 .test-infra/tools/stale_dataflow_jobs_cleaner.sh | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/.test-infra/tools/stale_dataflow_jobs_cleaner.sh 
b/.test-infra/tools/stale_dataflow_jobs_cleaner.sh
index e563048..e6df705 100755
--- a/.test-infra/tools/stale_dataflow_jobs_cleaner.sh
+++ b/.test-infra/tools/stale_dataflow_jobs_cleaner.sh
@@ -27,3 +27,10 @@ if [[ ${STALE_JOBS} ]]; then
 else
   echo "No stale jobs found."
 fi
+
+# Delete spanner databases older than 1 day.
+gcloud spanner databases list \
+--instance beam-test \
+--filter="createTime < $(date --iso-8601=s -d '1 day ago')" \
+--format="value(name)" | \
+xargs -I{} gcloud spanner databases delete {} --instance beam-test --quiet

[beam] branch master updated: [BEAM-13220] Update release instructions.

[ibzib]

This is an automated email from the ASF dual-hosted git repository.

ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 8d75633  [BEAM-13220] Update release instructions.
 new 5fc55f3  Merge pull request #16065 from ibzib/BEAM-13220
8d75633 is described below

commit 8d75633d0c082b1151405cf364c9370bfc95f596
Author: Kyle Weaver 
AuthorDate: Wed Nov 24 15:24:04 2021 -0800

[BEAM-13220] Update release instructions.

The biggest change here is to run Java wordcount and mobile gaming examples 
on Jenkins instead of the bash script, since getting the script to work 
properly took too long.
---
 .../main/scripts/publish_github_release_notes.sh   |   2 +-
 release/src/main/scripts/run_rc_validation.sh  | 128 +
 release/src/main/scripts/script.config |   2 -
 .../site/content/en/contribute/release-guide.md|  20 ++--
 4 files changed, 14 insertions(+), 138 deletions(-)

diff --git a/release/src/main/scripts/publish_github_release_notes.sh 
b/release/src/main/scripts/publish_github_release_notes.sh
index 7f58648..09fec15 100755
--- a/release/src/main/scripts/publish_github_release_notes.sh
+++ b/release/src/main/scripts/publish_github_release_notes.sh
@@ -33,7 +33,7 @@ RELEASE_NOTES=$(
 )
 
 # Escape notes' content to work with JSON
-ESCAPED_NOTES=$(printf '%s' "${RELEASE_NOTES}" | python -c 'import json,sys; 
print(json.dumps(sys.stdin.read()))')
+ESCAPED_NOTES=$(printf '%s' "${RELEASE_NOTES}" | python3 -c 'import json,sys; 
print(json.dumps(sys.stdin.read()))')
 
 # Build JSON for the API request
 REQUEST_JSON="$(cat <<-EOF
diff --git a/release/src/main/scripts/run_rc_validation.sh 
b/release/src/main/scripts/run_rc_validation.sh
index baff09b..28ba2eb 100755
--- a/release/src/main/scripts/run_rc_validation.sh
+++ b/release/src/main/scripts/run_rc_validation.sh
@@ -207,133 +207,6 @@ kubectl version
 
 
 echo ""
-echo ""
-echo "Starting Java Quickstart==="
-echo "[Current task] Java quickstart with direct runner"
-if [[ "$java_quickstart_direct" = true ]]; then
-  echo "*"
-  echo "* Running Java Quickstart with DirectRunner"
-  echo "*"
-  ./gradlew :runners:direct-java:runQuickstartJavaDirect \
-  -Prepourl=${REPO_URL} \
-  -Pver=${RELEASE_VER}
-else
-  echo "* Skip Java quickstart with direct runner"
-fi
-
-echo "[Current task] Java quickstart with Flink local runner"
-if [[ "$java_quickstart_flink_local" = true ]]; then
-  echo "*"
-  echo "* Running Java Quickstart with Flink local runner"
-  echo "*"
-  ./gradlew :runners:flink:1.13:runQuickstartJavaFlinkLocal \
-  -Prepourl=${REPO_URL} \
-  -Pver=${RELEASE_VER}
-else
-  echo "* Skip Java quickstart with Flink local runner"
-fi
-
-echo "[Current task] Java quickstart with Spark local runner"
-if [[ "$java_quickstart_spark_local" = true ]]; then
-  echo "*"
-  echo "* Running Java Quickstart with Spark local runner"
-  echo "*"
-  ./gradlew :runners:spark:2:runQuickstartJavaSpark \
-  -Prepourl=${REPO_URL} \
-  -Pver=${RELEASE_VER}
-else
-  echo "* Skip Java quickstart with Spark local runner"
-fi
-
-echo "[Current task] Java quickstart with Dataflow runner"
-if [[ "$java_quickstart_dataflow" = true && ! -z `which gcloud` ]]; then
-  echo "*"
-  echo "* Running Java Quickstart with DataflowRunner"
-  echo "*"
-  ./gradlew :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow \
-  -Prepourl=${REPO_URL} \
-  -Pver=${RELEASE_VER} \
-  -PgcpProject=${USER_GCP_PROJECT} \
-  -PgcsBucket=${USER_GCS_BUCKET:5}  # skip 'gs://' prefix
-else
-  echo "* Skip Java quickstart with Dataflow runner. Google Cloud SDK is 
required."
-fi
-
-echo "[Current task] Java quickstart with Twister2 local runner"
-if [[ "$java_quickstart_twister2_local" = true ]]; then
-  echo "*"
-  echo "* Running Java Quickstart with Twister2 local runner"
-  echo "*"
-  ./gradlew :runners:twister2:runQuickstartJavaTwister2 \
-  -Prepourl=${REPO_URL} \
-  -Pver=${RELEASE_VER}
-else
-  echo "* Skip Java quickstart with Twister2 local runner"
-fi
-
-echo ""
-echo "Starting Java Mobile Game="
-if [[ ("$java_mobile_game_direct" = true || "$java_mobile_game_dataflow" = 
true) \
-  && ! -z `which gcloud` ]]; then
-  MOBILE_GAME_DATASET=${USER}_java_v

[beam] branch master updated (5fc55f3 -> e6734b9)

[bhulette]

This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5fc55f3  Merge pull request #16065 from ibzib/BEAM-13220
 add e6734b9  [BEAM-13335] Use shorter, numerical indices for dataframe 
reads. (#16066)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/dataframe/io.py | 39 +
 1 file changed, 30 insertions(+), 9 deletions(-)

[beam] branch master updated (e6734b9 -> 3ce6138)

[bhulette]

This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e6734b9  [BEAM-13335] Use shorter, numerical indices for dataframe 
reads. (#16066)
 add 3ce6138  Be more conservative about rebatching. (#16058)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/dataframe/transforms.py | 23 +--
 1 file changed, 17 insertions(+), 6 deletions(-)

[beam] branch master updated (3ce6138 -> 33bcdf6)

[bhulette]

This is an automated email from the ASF dual-hosted git repository.

bhulette pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3ce6138  Be more conservative about rebatching. (#16058)
 add 33bcdf6  [BEAM-12587] Allow None in Python's Any logical type. (#16055)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/dataframe/convert_test.py | 7 +++
 sdks/python/apache_beam/typehints/schemas.py  | 8 ++--
 sdks/python/apache_beam/typehints/schemas_test.py | 3 ++-
 3 files changed, 15 insertions(+), 3 deletions(-)

[beam] branch revert-15969-combine-type-hints created (now d71203a)

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a change to branch revert-15969-combine-type-hints
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at d71203a  Revert "Key-inferable type hints for CombinePerKey."

This branch includes the following new commits:

 new d71203a  Revert "Key-inferable type hints for CombinePerKey."

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[beam] 01/01: Revert "Key-inferable type hints for CombinePerKey."

[tvalentyn]

This is an automated email from the ASF dual-hosted git repository.

tvalentyn pushed a commit to branch revert-15969-combine-type-hints
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d71203a01307008114238a8b4506e67ae24a5ad7
Author: tvalentyn 
AuthorDate: Mon Nov 29 15:19:04 2021 -0800

Revert "Key-inferable type hints for CombinePerKey."
---
 sdks/python/apache_beam/pipeline.py|  4 +---
 sdks/python/apache_beam/transforms/core.py | 24 ++
 .../apache_beam/transforms/ptransform_test.py  | 13 
 3 files changed, 12 insertions(+), 29 deletions(-)

diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index a512a18..25c02d4 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -892,9 +892,7 @@ class Pipeline(object):
   pcoll.element_type)
 if (isinstance(output.element_type,
typehints.TupleHint.TupleConstraint) and
-len(output.element_type.tuple_types) == 2 and
-pcoll.element_type.tuple_types[0] ==
-output.element_type.tuple_types[0]):
+len(output.element_type.tuple_types) == 2):
   output.requires_deterministic_key_coder = (
   deterministic_key_coders and transform_node.full_label)
 for side_input in transform_node.transform.side_inputs:
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index bd999f8..c71c00c 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -2306,20 +2306,18 @@ class CombinePerKey(PTransformWithSideInputs):
 self.fn, *args, **kwargs)
 
   def default_type_hints(self):
-result = self.fn.get_type_hints()
-k = typehints.TypeVariable('K')
-if result.input_types:
-  args, kwargs = result.input_types
-  args = (typehints.Tuple[k, args[0]], ) + args[1:]
-  result = result.with_input_types(*args, **kwargs)
-else:
-  result = result.with_input_types(typehints.Tuple[k, typehints.Any])
-if result.output_types:
-  main_output_type = result.simple_output_type('')
-  result = result.with_output_types(typehints.Tuple[k, main_output_type])
+hints = self.fn.get_type_hints()
+if hints.input_types:
+  K = typehints.TypeVariable('K')
+  args, kwargs = hints.input_types
+  args = (typehints.Tuple[K, args[0]], ) + args[1:]
+  hints = hints.with_input_types(*args, **kwargs)
 else:
-  result = result.with_output_types(typehints.Tuple[k, typehints.Any])
-return result
+  K = typehints.Any
+if hints.output_types:
+  main_output_type = hints.simple_output_type('')
+  hints = hints.with_output_types(typehints.Tuple[K, main_output_type])
+return hints
 
   def to_runner_api_parameter(
   self,
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py 
b/sdks/python/apache_beam/transforms/ptransform_test.py
index 6a139eb..ac4f632 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -1839,19 +1839,6 @@ class PTransformTypeCheckTestCase(TypeHintTestCase):
 assert_that(d, equal_to([6]))
 self.p.run()
 
-  def test_combine_properly_pipeline_type_checks_without_decorator(self):
-def sum_ints(ints):
-  return sum(ints)
-
-d = (
-self.p
-| beam.Create([1, 2, 3])
-| beam.Map(lambda x: ('key', x))
-| beam.CombinePerKey(sum_ints))
-
-self.assertEqual(typehints.Tuple[str, typehints.Any], d.element_type)
-self.p.run()
-
   def test_combine_func_type_hint_does_not_take_iterable_using_decorator(self):
 @with_output_types(int)
 @with_input_types(a=int)

[beam] branch asf-site updated: Publishing website 2021/11/30 00:03:25 at commit 33bcdf6

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 924a7c7  Publishing website 2021/11/30 00:03:25 at commit 33bcdf6
924a7c7 is described below

commit 924a7c77db1c79a5f8ccb07cfbd318fc2d5a100b
Author: jenkins 
AuthorDate: Tue Nov 30 00:03:26 2021 +

Publishing website 2021/11/30 00:03:25 at commit 33bcdf6
---
 website/generated-content/contribute/index.xml | 18 +++---
 .../contribute/release-guide/index.html| 16 
 website/generated-content/sitemap.xml  |  2 +-
 3 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/website/generated-content/contribute/index.xml 
b/website/generated-content/contribute/index.xml
index 44366ae..4da81a2 100644
--- a/website/generated-content/contribute/index.xml
+++ b/website/generated-content/contribute/index.xml
@@ -498,11 +498,11 @@ If you don’t have a GPG key associated with your Apache 
account, please create