[beam] branch nightly-refs/heads/master updated (f31cad0 -> f43789a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git. from f31cad0 Merge pull request #16373 from [BEAM-13515] [Playground] Hiding lines in an example that are not necessary add 421bc80 Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms" add 4f29108 BEAM-13611 reactivating jdbcio xlang test add a00634d Merge pull request #16569 from Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for … add d43c755 Merge pull request #16371 from [BEAM-13518][Playground] Beam Playground quickstart page on the Beam website add 77b410c Update Java FnAPI beam master (#16572) add 93f41ce [BEAM-13699] Replace fnv with maphash. (#16573) add 2c22d8b [BEAM-13693] Bump beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours (#16576) add 586739d [BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema add ab6374e Merge pull request #16380 from talatuyarer/BEAM-13577-select-uniquifyNames-nullablity add bc39244 [BEAM-13689] Output token elements when BQ batch writes complete. add acd1d52 Merge pull request #16561 from [BEAM-13689] Output TableDestination elements when BQ batch writes complete. add 50e7235 [BEAM-10206] Remove Fatalf calls in non-test goroutines for tests/benchmarks (#16575) add e3c24f0 [BEAM-13430] Re-add provided configuration (#16552) add 0909fb8 edited README and comments in Python multi-lang pipes examples add 750b74e Merge pull request #16566: edited README and comments in Python multi-lang pipes examples add c074b0f [BEAM-12164]: Add SDF for reading change stream records add f43789a Merge pull request #16514 from [BEAM-12164]: Add SDF for reading change stream records No new revisions were added by this update. Summary of changes: ..._Java_ValidatesRunner_Dataflow_Streaming.groovy | 2 +- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +- examples/kotlin/build.gradle | 2 +- examples/multi-language/README.md | 37 ++- examples/multi-language/python/addprefix.py| 4 +- examples/multi-language/python/javacount.py| 6 +- .../multi-language/python/javadatagenerator.py | 6 +- runners/direct-java/build.gradle | 4 +- runners/flink/flink_runner.gradle | 3 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../worker/legacy-worker/build.gradle | 9 +- runners/portability/java/build.gradle | 1 + runners/spark/spark_runner.gradle | 12 +- sdks/go/pkg/beam/core/runtime/exec/hash.go | 5 +- sdks/go/pkg/beam/core/runtime/exec/hash_test.go| 4 +- sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 12 +- sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 10 +- sdks/java/container/build.gradle | 2 +- sdks/java/container/common.gradle | 8 +- sdks/java/core/build.gradle| 10 +- .../apache/beam/sdk/schemas/transforms/Select.java | 10 +- sdks/java/extensions/sorter/build.gradle | 6 +- sdks/java/extensions/sql/build.gradle | 16 +- sdks/java/extensions/sql/datacatalog/build.gradle | 6 +- .../sdk/extensions/sql/BeamSqlDslArrayTest.java| 53 +++- .../rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java | 32 +- sdks/java/extensions/sql/zetasql/build.gradle | 2 +- sdks/java/fn-execution/build.gradle| 2 +- sdks/java/harness/build.gradle | 2 +- sdks/java/io/debezium/build.gradle | 2 +- .../beam/sdk/io/gcp/bigquery/BatchLoads.java | 165 +- .../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 2 +- .../sdk/io/gcp/bigquery/StreamingWriteTables.java | 4 +- .../beam/sdk/io/gcp/bigquery/WriteRename.java | 22 +- .../beam/sdk/io/gcp/bigquery/WriteResult.java | 71 - .../spanner/changestreams/TimestampConverter.java | 64 .../changestreams/action/ActionFactory.java| 131 .../action/ChildPartitionsRecordAction.java| 230 ++ .../action/DataChangeRecordAction.java | 115 +++ .../action/HeartbeatRecordAction.java | 110 +++ .../action/QueryChangeStreamAction.java| 299 ++ .../spanner/changestreams/action/package-info.java | 23 ++ .../dofn/ReadChangeStreamPartitionDoFn.java| 230 ++ .../ReadChangeStreamPartitionRangeTracker.java | 80 + .../changestreams/restriction/package-info.java| 23 ++ .../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 94 +- .../changestreams/TimestampConverterTest.java | 69 +
[beam] branch master updated (1d594f3 -> a75ccc8)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1d594f3 Merge pull request #16546 from [BEAM-13661] [BEAM-13704] [Playground] Update tags for examples/katas/unit-tests add a75ccc8 Merge pull request #16369 from [BEAM-13558] [Playground] Hide the Graph tab and SCIO from SDK options No new revisions were added by this update. Summary of changes: playground/frontend/lib/modules/output/components/output.dart | 2 +- .../frontend/lib/modules/output/components/output_area.dart | 4 ++-- .../modules/output/components/output_header/output_tabs.dart | 11 ++- .../frontend/lib/modules/sdk/components/sdk_selector.dart | 7 +-- 4 files changed, 14 insertions(+), 10 deletions(-)
[beam] branch master updated (f850308 -> 1d594f3)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f850308 Merge pull request #16539 from [BEAM-13677][Playground]Update GitHub Actions to Build Playground Containers In Proper Way add 1d594f3 Merge pull request #16546 from [BEAM-13661] [BEAM-13704] [Playground] Update tags for examples/katas/unit-tests No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/examples/DebuggingWordCount.java | 1 + .../java/src/main/java/org/apache/beam/examples/MinimalWordCount.java | 3 ++- examples/java/src/main/java/org/apache/beam/examples/WordCount.java| 1 + .../src/test/java/org/apache/beam/examples/complete/TfIdfTest.java | 1 - .../java/org/apache/beam/examples/complete/game/GameStatsTest.java | 1 - .../java/org/apache/beam/examples/cookbook/DistinctExampleTest.java| 1 - .../beam/learning/katas/commontransforms/aggregation/count/Task.java | 1 - .../beam/learning/katas/commontransforms/aggregation/max/Task.java | 1 - .../beam/learning/katas/commontransforms/aggregation/mean/Task.java| 1 - .../beam/learning/katas/commontransforms/aggregation/min/Task.java | 1 - .../beam/learning/katas/commontransforms/aggregation/sum/Task.java | 1 - .../beam/learning/katas/commontransforms/filter/filter/Task.java | 1 - .../apache/beam/learning/katas/commontransforms/filter/pardo/Task.java | 1 - .../org/apache/beam/learning/katas/commontransforms/withkeys/Task.java | 1 - .../org/apache/beam/learning/katas/coretransforms/branching/Task.java | 1 - .../katas/coretransforms/combine/binarycombinefnlambda/Task.java | 1 - .../learning/katas/coretransforms/combine/binarycombinefn/Task.java| 1 - .../beam/learning/katas/coretransforms/combine/combineperkey/Task.java | 1 - .../beam/learning/katas/coretransforms/combine/combinefn/Task.java | 1 - .../apache/beam/learning/katas/coretransforms/combine/simple/Task.java | 1 - .../org/apache/beam/learning/katas/coretransforms/composite/Task.java | 1 - .../org/apache/beam/learning/katas/coretransforms/flatten/Task.java| 1 - .../org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java | 1 - .../beam/learning/katas/coretransforms/map/flatmapelements/Task.java | 1 - .../beam/learning/katas/coretransforms/map/mapelements/Task.java | 1 - .../beam/learning/katas/coretransforms/map/pardoonetomany/Task.java| 1 - .../org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java | 1 - .../org/apache/beam/learning/katas/coretransforms/partition/Task.java | 1 - .../org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java | 1 - .../src/org/apache/beam/learning/katas/examples/wordcount/Task.java| 1 - .../src/org/apache/beam/learning/katas/intro/hello/Task.java | 2 +- .../src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java | 1 - learning/katas/python/Common Transforms/Aggregation/Count/task.py | 1 - learning/katas/python/Common Transforms/Aggregation/Largest/task.py| 1 - learning/katas/python/Common Transforms/Aggregation/Mean/task.py | 1 - learning/katas/python/Common Transforms/Aggregation/Smallest/task.py | 1 - learning/katas/python/Common Transforms/Aggregation/Sum/task.py| 1 - learning/katas/python/Common Transforms/Filter/Filter/task.py | 1 - learning/katas/python/Common Transforms/Filter/ParDo/task.py | 1 - learning/katas/python/Common Transforms/WithKeys/WithKeys/task.py | 1 - learning/katas/python/Core Transforms/Branching/Branching/task.py | 1 - .../katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.py | 1 - learning/katas/python/Core Transforms/Combine/Combine PerKey/task.py | 1 - learning/katas/python/Core Transforms/Combine/CombineFn/task.py| 1 - learning/katas/python/Core Transforms/Combine/Simple Function/task.py | 1 - .../Core Transforms/Composite Transform/Composite Transform/task.py| 1 - learning/katas/python/Core Transforms/Flatten/Flatten/task.py | 1 - learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.py| 1 - learning/katas/python/Core Transforms/Map/FlatMap/task.py | 1 - learning/katas/python/Core Transforms/Map/Map/task.py | 1 - learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.py | 1 - learning/katas/python/Core Transforms/Map/ParDo/task.py| 1 - learning/katas/python/Core Transforms/Partition/Partition/task.py | 1 - learning/katas/python/Core Transforms/Side Input/Side Input/task.py| 1 - learning/katas/python/Core Transforms/Side Output/Side Output/task.py | 1 - learning/katas/python/Examples/Word Count/Word Count/task.py | 1 - learning/katas/python/Introduction/Hello Beam/Hello Beam/task.py | 2 +-
[beam] branch master updated: Update GH Actions to use proper variables names and proper triggers
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c54b4b3 Update GH Actions to use proper variables names and proper
triggers
new f850308 Merge pull request #16539 from [BEAM-13677][Playground]Update
GitHub Actions to Build Playground Containers In Proper Way
c54b4b3 is described below
commit c54b4b3ef7714de8c345e52365dfdbde9c678856
Author: Sergey Kalinin
AuthorDate: Mon Jan 17 20:34:48 2022 +
Update GH Actions to use proper variables names and proper triggers
---
.github/workflows/build_playground_backend.yml | 52 +++--
.github/workflows/build_playground_frontend.yml | 23 +++
2 files changed, 47 insertions(+), 28 deletions(-)
diff --git a/.github/workflows/build_playground_backend.yml
b/.github/workflows/build_playground_backend.yml
index 574a259..512c4ed 100644
--- a/.github/workflows/build_playground_backend.yml
+++ b/.github/workflows/build_playground_backend.yml
@@ -57,35 +57,45 @@ jobs:
- uses: hashicorp/setup-terraform@v1
with:
terraform_version: ${{ env.TERRAFORM_VERSION }}
-if: startsWith(github.ref, 'ref/tags/')
+if: startsWith(github.ref, 'refs/tags/')
+ - name: Set Docker Tag
+run: echo "DOCKERTAG=${GITHUB_SHA}" >> $GITHUB_ENV
+ - name: Set Docker Tag If Github Tag was trigger
+run: echo "DOCKERTAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
+if: startsWith(github.ref, 'refs/tags/')
- name: Setup GCP account
-run: echo ${{ secrets.GCP_ACCESS_KEY }} | base64 -d >
/tmp/gcp_access.json
-if: startsWith(github.ref, 'ref/tags/')
+run: |
+ echo "${{ secrets.PLAYGROUND_GCP_ACCESS_KEY }}" | base64 -d >
/tmp/gcp_access.json
+if: startsWith(github.ref, 'refs/tags/')
- name: Login to Docker Registry
-run: cat /tmp/gcp_access.json | docker login -u _json_key
--password-stdin https://${{ secrets.REGISTRY_NAME }}
-if: startsWith(github.ref, 'ref/tags/')
+run: cat /tmp/gcp_access.json | docker login -u _json_key
--password-stdin https://${{ secrets.PLAYGROUND_REGISTRY_NAME }}
+if: startsWith(github.ref, 'refs/tags/')
- name: Preapre Build Java
-run: ./gradlew playground:backend:containers:java:dockerPush
-Pdocker-repository-root='${{ secrets.REGISTRY_NAME}}/${{ secrets.PROJECT_ID
}}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{
env.BEAM_VERSION }}'
-if: startsWith(github.ref, 'ref/tags/')
+run: |
+export
+./gradlew playground:backend:containers:java:dockerTagPush
-Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{
secrets.PLAYGROUND_PROJECT_ID }}/playground-repository'
-Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{
env.DOCKERTAG }}
+if: startsWith(github.ref, 'refs/tags/')
- name: Preapre Build Python
-run: ./gradlew playground:backend:containers:python:dockerPush
-Pdocker-repository-root='${{ secrets.REGISTRY_NAME}}/${{ secrets.PROJECT_ID
}}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{
env.BEAM_VERSION }}'
- - name: Preapre Build Go
-run: ./gradlew playground:backend:containers:go:dockerPush
-Pdocker-repository-root='${{ secrets.REGISTRY_NAME}}/${{ secrets.PROJECT_ID
}}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{
env.BEAM_VERSION }}'
-if: startsWith(github.ref, 'ref/tags/')
+run: ./gradlew playground:backend:containers:python:dockerTagPush
-Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{
secrets.PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{
env.DOCKERTAG }}
+if: startsWith(github.ref, 'refs/tags/')
+ - name: Preapre Build Go
+run: ./gradlew playground:backend:containers:go:dockerTagPush
-Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{
secrets.PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{
env.DOCKERTAG }}
+if: startsWith(github.ref, 'refs/tags/')
- name: Deploy Backend Application Java
env:
GOOGLE_APPLICATION_CREDENTIALS: /tmp/gcp_access.json
-run: terraform init && terraform apply -auto-approve
-var="project_id=${{ secrets.PROJECT_ID }}" -var="docker_registry_address=${{
secrets.REGISTRY_NAME}}/${{ secrets.PROJECT_ID }}/playground-repository"
-working-directory: playground/terraform/applications/backend
-if: startsWith(github.ref, 'ref/tags/')
- - name: Deploy Backend Application Python
-env:
- GOOGLE_APPLICATION_CREDENTIALS: /tmp/gcp_access.json
-run: terraform init && terraform apply -auto-approve
-var="project_id=${{ secrets.PROJECT_ID }}"
[beam] branch master updated (f43789a -> 5d5ffad)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from f43789a Merge pull request #16514 from [BEAM-12164]: Add SDF for
reading change stream records
add 5d5ffad Merge pull request #16540 from [BEAM-13678][Playground]Update
Github Action To Deploy Examples
No new revisions were added by this update.
Summary of changes:
.github/workflows/playground_deploy_examples.yml | 27 +++---
playground/infrastructure/helm/Chart.yaml | 2 +-
.../helm/templates/service-python.yaml | 2 +-
.../{Chart.yaml => templates/serviceaccount.yaml} | 16 -
4 files changed, 31 insertions(+), 16 deletions(-)
copy playground/infrastructure/helm/{Chart.yaml =>
templates/serviceaccount.yaml} (74%)
[beam] branch master updated: [BEAM-12164]: Add SDF for reading change stream records
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c074b0f [BEAM-12164]: Add SDF for reading change stream records
new f43789a Merge pull request #16514 from [BEAM-12164]: Add SDF for
reading change stream records
c074b0f is described below
commit c074b0f7535e7f12b467864ca9441226f06dd57e
Author: Thiago Nunes
AuthorDate: Fri Jan 14 15:16:13 2022 +1100
[BEAM-12164]: Add SDF for reading change stream records
Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions
from change streams and process them accordingly. This component
receives a change stream name, a partition, a start time and an end time
to query. It then initiates a change stream query with the received
parameters.
Within a change stream, 3 types of records can be received:
1. A Data record
2. A Heartbeat record
3. A Child partitions record
Upon receiving #1, the function updates the watermark with the record's
commit timestamp and emits the record into the output PCollection.
Upon receiving #2, the function updates the watermark with the record's
timestamp, but it does not emit any record into the PCollection.
Upon receiving #3, the function updates the watermark with the record's
timestamp and writes the new child partitions into the metadata table.
These partitions will be later scheduled by the DetectNewPartitions
component.
Once the change stream query for the element partition finishes, it
marks the partition as finished in the metadata table and terminates.
---
.../spanner/changestreams/TimestampConverter.java | 64
.../changestreams/action/ActionFactory.java| 131
.../action/ChildPartitionsRecordAction.java| 230 ++
.../action/DataChangeRecordAction.java | 115 +++
.../action/HeartbeatRecordAction.java | 110 +++
.../action/QueryChangeStreamAction.java| 299 ++
.../spanner/changestreams/action/package-info.java | 23 ++
.../dofn/ReadChangeStreamPartitionDoFn.java| 230 ++
.../ReadChangeStreamPartitionRangeTracker.java | 80 +
.../changestreams/restriction/package-info.java| 23 ++
.../changestreams/TimestampConverterTest.java | 69 +
.../action/ChildPartitionsRecordActionTest.java| 244 +++
.../action/DataChangeRecordActionTest.java | 91 ++
.../action/HeartbeatRecordActionTest.java | 85 ++
.../action/QueryChangeStreamActionTest.java| 337 +
.../dofn/ReadChangeStreamPartitionDoFnTest.java| 171 +++
.../ReadChangeStreamPartitionRangeTrackerTest.java | 59
.../changestreams/util/TestTransactionAnswer.java | 42 +++
18 files changed, 2403 insertions(+)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/TimestampConverter.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/TimestampConverter.java
new file mode 100644
index 000..ab7b9e1
--- /dev/null
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/TimestampConverter.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.spanner.changestreams;
+
+import com.google.cloud.Timestamp;
+import java.math.BigDecimal;
+
+/** Util class to manage timestamp conversions. */
+public class TimestampConverter {
+
+ /** The number of microseconds in a {@link Timestamp#MAX_VALUE}. */
+ public static final long MAX_MICROS = timestampToMicros(Timestamp.MAX_VALUE);
+
+ /**
+ * Converts a {@link Timestamp} to its number of microseconds. Note there is
precision loss here.
+ *
+ * @param timestamp the timestamp to be converted
+ * @return the number of microseconds in the given timestamp
+ */
+ public static long timestampToMicros(Timestamp timestamp) {
+
[beam] 01/01: Merge pull request #16566: edited README and comments in Python multi-lang pipes examples
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 750b74e8bd8d77d30d3f88c4357c46101f677d5f Merge: e3c24f0 0909fb8 Author: Chamikara Jayalath AuthorDate: Thu Jan 20 18:02:54 2022 -0800 Merge pull request #16566: edited README and comments in Python multi-lang pipes examples examples/multi-language/README.md | 37 ++ examples/multi-language/python/addprefix.py| 4 +-- examples/multi-language/python/javacount.py| 6 ++-- .../multi-language/python/javadatagenerator.py | 6 ++-- 4 files changed, 31 insertions(+), 22 deletions(-)
[beam] branch master updated (e3c24f0 -> 750b74e)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e3c24f0 [BEAM-13430] Re-add provided configuration (#16552) add 0909fb8 edited README and comments in Python multi-lang pipes examples new 750b74e Merge pull request #16566: edited README and comments in Python multi-lang pipes examples The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: examples/multi-language/README.md | 37 ++ examples/multi-language/python/addprefix.py| 4 +-- examples/multi-language/python/javacount.py| 6 ++-- .../multi-language/python/javadatagenerator.py | 6 ++-- 4 files changed, 31 insertions(+), 22 deletions(-)
[beam] 02/02: Set Dataflow container to release version.
This is an automated email from the ASF dual-hosted git repository.
emilyye pushed a commit to branch release-2.36.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 5076eb1fd69365ff998f776caaff9690623e7b0d
Author: Emily Ye
AuthorDate: Thu Jan 20 17:11:31 2022 -0800
Set Dataflow container to release version.
---
runners/google-cloud-dataflow-java/build.gradle | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index 13d07ce..8dd8f29 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -54,8 +54,8 @@ processResources {
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '8',
'dataflow.fnapi_environment_major_version' : '8',
-'dataflow.legacy_container_version' : 'beam-master-20211213',
-'dataflow.fnapi_container_version' : 'beam-master-20210119',
+'dataflow.legacy_container_version' : 'beam-2.36.0',
+'dataflow.fnapi_container_version' : 'beam-2.36.0',
'dataflow.container_base_repository' : 'gcr.io/cloud-dataflow/v1beta3',
]
}
[beam] 01/02: (release recut) Revert "Moving to 2.37.0-SNAPSHOT on master branch."
This is an automated email from the ASF dual-hosted git repository.
emilyye pushed a commit to branch release-2.36.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 79120fdaaa8ec86e209194853d4802786dd51049
Author: Emily Ye
AuthorDate: Thu Jan 20 17:08:21 2022 -0800
(release recut) Revert "Moving to 2.37.0-SNAPSHOT on master branch."
This reverts commit 28ea3f81a59a7b96945b13ee2af4bc5bc8d3fbd7.
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 2 +-
gradle.properties | 4 ++--
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/python/apache_beam/version.py| 2 +-
4 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index d551554..008a4cb 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -378,7 +378,7 @@ class BeamModulePlugin implements Plugin {
// Automatically use the official release version if we are performing a
release
// otherwise append '-SNAPSHOT'
-project.version = '2.37.0'
+project.version = '2.36.0'
if (!isRelease(project)) {
project.version += '-SNAPSHOT'
}
diff --git a/gradle.properties b/gradle.properties
index 5876c34..bc0dc86 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -24,8 +24,8 @@ offlineRepositoryRoot=offline-repository
signing.gnupg.executable=gpg
signing.gnupg.useLegacyGpg=true
-version=2.37.0-SNAPSHOT
-sdk_version=2.37.0.dev
+version=2.36.0-SNAPSHOT
+sdk_version=2.36.0.dev
javaVersion=1.8
diff --git a/sdks/go/pkg/beam/core/core.go b/sdks/go/pkg/beam/core/core.go
index 0f23e86..506672f 100644
--- a/sdks/go/pkg/beam/core/core.go
+++ b/sdks/go/pkg/beam/core/core.go
@@ -27,5 +27,5 @@ const (
// SdkName is the human readable name of the SDK for UserAgents.
SdkName = "Apache Beam SDK for Go"
// SdkVersion is the current version of the SDK.
- SdkVersion = "2.37.0.dev"
+ SdkVersion = "2.36.0.dev"
)
diff --git a/sdks/python/apache_beam/version.py
b/sdks/python/apache_beam/version.py
index f80b339..fb7b7d1 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -17,4 +17,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.37.0.dev'
+__version__ = '2.36.0.dev'
[beam] branch release-2.36.0 updated (d5d0bc3 -> 5076eb1)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch release-2.36.0 in repository https://gitbox.apache.org/repos/asf/beam.git. discard d5d0bc3 Merge pull request #16462 from ihji/cherry-pick-13599 omit 3fb0ff6 [BEAM-13599] Fix overflow error in Python Datastore RampupThrottlingFn discard bd46e7b [release-2.36.0][BEAM-13541] More intelligent caching of CoGBK values. (#16354, #16407) (#16421) omit d5e75c5 [release-2.36.0][BEAM-13575] Sickbay test that is flaky to restore precommit test signal (#16410) (#16435) omit 99e455e [release-2.36.0][BEAM-13459] Fix sdk_container_builder too many values to unpack error (#16422) omit f62cabf Set Dataflow container to release version. add 28ea3f8 Moving to 2.37.0-SNAPSHOT on master branch. add 12fbe03 Release website update for Beam 2.35.0 (#16115) add 0af3689 [BEAM-13430] Re-enable checkerframework for the project excluding the two classes it started to crash on. (#16389) add e2189b3 [adhoc] Use predefined min/max combiners in SQS reader to gather stats add d8784ac Merge pull request #16390: [adhoc] Use predefined min/max combiners in SQS reader to gather stats add 52715c0 [BEAM-13009] Fix tests to ensure absence of duplicates only per request to avoid issues when running with low targetParellelism add fe456b7 Merge pull request #16367: [BEAM-13009] Fix tests to ensure absence of duplicates only per request add df26931 [BEAM-13430] Fix class not found error for example integration tests and dataflow postcommit (#16388) add eaacf70 [BEAM-13430] Re-enable dependency analysis for modules. (#16395) add 3a74728 Fix class not found in perf integration tests. add 598f444 Update buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy add 2a9e8b6 Merge pull request #16398 from y1chi/perf-it add e55f54e [BEAM-13484][Playground] Improve Terraform Scripts For Deploy Infrastructure (#16378) add 3ab26ce [BEAM-13541] More intelligent caching of CoGBK values. (#16354) add 0e69485 Merge pull request #15769 from [BEAM-13031] [Playground] Code editor - Support Localization add 503adc0 [BEAM-13555] playground - quick fix for safari add 304680d Merge pull request #16362 from [BEAM-13555] [Bugfix] [Playground] change html renderer for safari to fix freezes add bbcfa4f [BEAM-13541] Fix spotbugs issue with DLS in tail (#16407) add b9e8a78 [BEAM-13588] Add missing module for PVR tests and also sickbay known broken test. (#16408) add fdbb90b [BEAM-13575] Sickbay test that is flaky to restore precommit test signal (#16410) add 34118c6 [BEAM-12092] Bump jedis to version 3.8.0 (#16403) add 7487bf0 [BEAM-13498, BEAM-13522] Sickbay broken test and test category for Spark (#16411) add df8b83b [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions add 2cd5013 Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions add 0fd6504 [BEAM-13509] Stop sharing SQS client between readers of same source. add c4fc398 Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source. add 17b62ad Merge pull request #16404: [BEAM-13586] Fix NPE in DefaultS3ClientBuilderFactory add 36ce02b [BEAM-13480] Sickbay PubSubIntegrationTest.test_streaming_data_only on Dataflow (#16255) add 3d272e7 Fix sdk_container_builder too many values to unpack error add 1636a3a Merge pull request #16417 from y1chi/python_val add 43a17a3 [BEAM-13430] Swap to use "mainClass" instead of "main" since it was deprecated. (#16400) add e6c73b4 [BEAM-13430] Replace deprecated "appendix" with "archiveAppendix" (#16401) add 061533c [BEAM-13591] Bump log4j2 version to 2.17.1 add 5a11778 Merge pull request #16415 from deadwind4/BEAM-13591 add 122ae34 [BEAM-13015] Add jamm as a java agent to the Java SDK harness container (#16412) add 2d4a17c [BEAM-13430] Partially revert https://github.com/apache/beam/commit/eaacf709a861ea1c260e3856b720266348b1aecc (#16419) add 7b5c4cd [BEAM-12391] update avro sink to close the opened file handle, instead of getting it by reference from the writer. This ensures that when we load the python writer implementation we don't run into an attribute error add f63451e Merge pull request #16246: [BEAM-12391] update avro sink to close the opened file handle directly add 6045ee1 [BEAM-13482] Python fully qualified name external transforms. add d13c21f Add Python expansion service entry point. add 0c2f5a5 Merge pull request #16266 [BEAM-13482] Python fully qualified name external transforms. add 818428f Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms add 9d2fcf3 Add Flink runner support for
[beam] branch master updated (50e7235 -> e3c24f0)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 50e7235 [BEAM-10206] Remove Fatalf calls in non-test goroutines for tests/benchmarks (#16575) add e3c24f0 [BEAM-13430] Re-add provided configuration (#16552) No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy| 8 +++- examples/kotlin/build.gradle | 2 +- runners/direct-java/build.gradle | 4 ++-- runners/flink/flink_runner.gradle | 3 +-- .../worker/legacy-worker/build.gradle | 9 +++-- runners/portability/java/build.gradle | 1 + runners/spark/spark_runner.gradle | 12 sdks/java/container/build.gradle | 2 +- sdks/java/container/common.gradle | 8 +--- sdks/java/core/build.gradle | 10 -- sdks/java/extensions/sorter/build.gradle | 6 ++ sdks/java/extensions/sql/build.gradle | 16 ++-- sdks/java/extensions/sql/datacatalog/build.gradle | 6 ++ sdks/java/extensions/sql/zetasql/build.gradle | 2 +- sdks/java/fn-execution/build.gradle | 2 +- sdks/java/harness/build.gradle| 2 +- sdks/java/io/debezium/build.gradle| 2 +- sdks/java/io/hadoop-common/build.gradle | 9 +++-- sdks/java/io/hadoop-format/build.gradle | 19 --- sdks/java/io/hbase/build.gradle | 2 +- sdks/java/io/hcatalog/build.gradle| 13 ++--- sdks/java/io/kafka/build.gradle | 10 +- sdks/java/io/parquet/build.gradle | 4 ++-- sdks/java/io/rabbitmq/build.gradle| 2 +- sdks/java/testing/tpcds/build.gradle | 2 +- 25 files changed, 74 insertions(+), 82 deletions(-)
[beam] branch master updated (acd1d52 -> 50e7235)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from acd1d52 Merge pull request #16561 from [BEAM-13689] Output TableDestination elements when BQ batch writes complete. add 50e7235 [BEAM-10206] Remove Fatalf calls in non-test goroutines for tests/benchmarks (#16575) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 12 ++-- sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 10 +- 2 files changed, 19 insertions(+), 3 deletions(-)
[beam] branch master updated: [BEAM-13689] Output token elements when BQ batch writes complete.
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bc39244 [BEAM-13689] Output token elements when BQ batch writes
complete.
new acd1d52 Merge pull request #16561 from [BEAM-13689] Output
TableDestination elements when BQ batch writes complete.
bc39244 is described below
commit bc39244aae33b1558e5388082d196f3a7d441921
Author: Steve Niemitz
AuthorDate: Mon Feb 1 09:06:08 2021 -0500
[BEAM-13689] Output token elements when BQ batch writes complete.
---
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 165 -
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 2 +-
.../sdk/io/gcp/bigquery/StreamingWriteTables.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 22 ++-
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 71 +++--
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 94 +++-
6 files changed, 262 insertions(+), 96 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 6297223..98310eb 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -46,6 +46,7 @@ import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.GroupIntoBatches;
+import org.apache.beam.sdk.transforms.Keys;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
@@ -356,30 +357,41 @@ class BatchLoads
PCollection> tempTables =
writeTempTables(partitions.get(multiPartitionsTag),
tempLoadJobIdPrefixView);
-tempTables
-// Now that the load job has happened, we want the rename to happen
immediately.
-.apply(
-"Window Into Global Windows",
-Window.>into(new
GlobalWindows())
-
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1
-.apply("Add Void Key", WithKeys.of((Void) null))
-.setCoder(KvCoder.of(VoidCoder.of(), tempTables.getCoder()))
-.apply("GroupByKey", GroupByKey.create())
-.apply("Extract Values", Values.create())
-.apply(
-"WriteRenameTriggered",
-ParDo.of(
-new WriteRename(
-bigQueryServices,
-copyJobIdPrefixView,
-writeDisposition,
-createDisposition,
-maxRetryJobs,
-kmsKey,
-loadJobProjectId))
-.withSideInputs(copyJobIdPrefixView));
-writeSinglePartition(partitions.get(singlePartitionTag),
loadJobIdPrefixView);
-return writeResult(p);
+PCollection successfulMultiPartitionWrites =
+tempTables
+// Now that the load job has happened, we want the rename to
happen immediately.
+.apply(
+"Window Into Global Windows",
+Window.>into(new
GlobalWindows())
+
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1
+.apply("Add Void Key", WithKeys.of((Void) null))
+.setCoder(KvCoder.of(VoidCoder.of(), tempTables.getCoder()))
+.apply("GroupByKey", GroupByKey.create())
+.apply("Extract Values", Values.create())
+.apply(
+"WriteRenameTriggered",
+ParDo.of(
+new WriteRename(
+bigQueryServices,
+copyJobIdPrefixView,
+writeDisposition,
+createDisposition,
+maxRetryJobs,
+kmsKey,
+loadJobProjectId))
+.withSideInputs(copyJobIdPrefixView));
+
+PCollection successfulSinglePartitionWrites =
+writeSinglePartition(partitions.get(singlePartitionTag),
loadJobIdPrefixView)
+.apply(
+"RewindowSinglePartitionResults",
+Window.into(new GlobalWindows())
+
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1;
+
+PCollectionList allSuccessfulWrites =
+
PCollectionList.of(successfulMultiPartitionWrites).and(successfulSinglePartitionWrites);
+
+return writeResult(p, allSuccessfulWrites.apply(Flatten.pCollections()));
}
// Expand
[beam] branch master updated: [BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 586739d [BEAM-13577] Beam Select's uniquifyNames function loses
nullability of Complex types while inferring schema
new ab6374e Merge pull request #16380 from
talatuyarer/BEAM-13577-select-uniquifyNames-nullablity
586739d is described below
commit 586739d854b5f6503f03469bee278de0bbd72ce9
Author: Talat Uyarer
AuthorDate: Thu Jan 13 21:21:04 2022 -0800
[BEAM-13577] Beam Select's uniquifyNames function loses nullability of
Complex types while inferring schema
---
.../apache/beam/sdk/schemas/transforms/Select.java | 10 ++--
.../sdk/extensions/sql/BeamSqlDslArrayTest.java| 53 +++---
.../rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java | 32 -
3 files changed, 84 insertions(+), 11 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
index 597aa62..763b7cf 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
@@ -226,12 +226,16 @@ public class Select {
.withNullable(fieldType.getNullable())
.withMetadata(fieldType.getAllMetadata());
case ARRAY:
-return
FieldType.array(uniquifyNames(fieldType.getCollectionElementType()));
+return
FieldType.array(uniquifyNames(fieldType.getCollectionElementType()))
+.withNullable(fieldType.getNullable());
case ITERABLE:
-return
FieldType.iterable(uniquifyNames(fieldType.getCollectionElementType()));
+return
FieldType.iterable(uniquifyNames(fieldType.getCollectionElementType()))
+.withNullable(fieldType.getNullable());
case MAP:
return FieldType.map(
-uniquifyNames(fieldType.getMapKeyType()),
uniquifyNames(fieldType.getMapValueType()));
+uniquifyNames(fieldType.getMapKeyType()),
+uniquifyNames(fieldType.getMapValueType()))
+.withNullable(fieldType.getNullable());
default:
return fieldType;
}
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
index f072467..65d6d72 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.extensions.sql;
import java.util.Arrays;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
@@ -37,7 +38,8 @@ public class BeamSqlDslArrayTest {
private static final Schema INPUT_SCHEMA =
Schema.builder()
.addInt32Field("f_int")
- .addArrayField("f_stringArr", Schema.FieldType.STRING)
+ .addStringField("f_string")
+ .addNullableField("f_stringArr",
FieldType.array(Schema.FieldType.STRING))
.build();
@Rule public final TestPipeline pipeline = TestPipeline.create();
@@ -91,6 +93,41 @@ public class BeamSqlDslArrayTest {
}
@Test
+ public void testProjectArrayFieldWithCoGBKJoin() {
+PCollection input = pCollectionOf2Elements();
+
+Schema resultType =
+Schema.builder()
+.addNullableField("f_stringArr",
FieldType.array(Schema.FieldType.STRING))
+.build();
+
+// When we use longer enough IN clause, Calcite calculate physical plan
with BeamCoGBKJoin.
+// This SQL push Calcite to use BeamCoGBKJoin for simple select statement.
+PCollection result =
+input.apply(
+"sqlQuery",
+SqlTransform.query(
+"SELECT f_stringArr FROM PCOLLECTION WHERE f_string IN ('A',
'B', "
++
"'ABCAABAAAGAG','ABCAABAAAGCB','ABCAABAAAGCJ','ABCAABAAAGEB','ABCAABAAAGEK',"
++
"'ABCAABAAAGFB','ABCAABAAAGFG','ABCAABABAGBJ','ABCAABABBKIF','ABCAABABCAIK',"
++
"'ABCAABAEJAAF','ABCAABAEJAED','ABCAABAEJAEE','ABCAABAEJAEF','ABCIABAAAGGJ',"
++
"'ABCIABAAAGKB','ABCIABAAAJBC','ABCIABAAAJCD','ABCIABAAAJEK','ABCIABAAAJFE',"
++
"'ABCIABAAAJGE','ABCIABAAAJGF','ABCIABAAAJGG','ABCIABAAAJJK','ABCIABAABAGK',"
++
"'ABCIABAABAKD','ABCIABAABBDI','ABCIABAABBEI','ABCIABAABFBB','ABCIABAABFBJ',"
++
[beam] branch master updated: [BEAM-13693] Bump beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours (#16576)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2c22d8b [BEAM-13693] Bump
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours
(#16576)
2c22d8b is described below
commit 2c22d8b914098c8512095fd36b82c8244e8576ed
Author: Brian Hulette
AuthorDate: Thu Jan 20 11:37:51 2022 -0800
[BEAM-13693] Bump beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming
timeout to 12 hours (#16576)
---
.../job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
index 8a5afb3..5d5c060 100644
---
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy
@@ -27,7 +27,7 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflo
description('Runs the ValidatesRunner suite on the Dataflow runner
forcing streaming mode.')
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 540)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 720)
// Publish all test results to Jenkins
publishers {
[beam] branch master updated: [BEAM-13699] Replace fnv with maphash. (#16573)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 93f41ce [BEAM-13699] Replace fnv with maphash. (#16573)
93f41ce is described below
commit 93f41ce1e3d245d7decd1d9b7c4768e51a90b39b
Author: Robert Burke
AuthorDate: Thu Jan 20 11:16:30 2022 -0800
[BEAM-13699] Replace fnv with maphash. (#16573)
---
sdks/go/pkg/beam/core/runtime/exec/hash.go | 5 ++---
sdks/go/pkg/beam/core/runtime/exec/hash_test.go | 4 ++--
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/sdks/go/pkg/beam/core/runtime/exec/hash.go
b/sdks/go/pkg/beam/core/runtime/exec/hash.go
index db2d0b6..70fbfa0 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/hash.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/hash.go
@@ -19,7 +19,7 @@ import (
"encoding/binary"
"fmt"
"hash"
- "hash/fnv"
+ "hash/maphash"
"math"
"reflect"
@@ -35,8 +35,7 @@ type elementHasher interface {
}
func makeElementHasher(c *coder.Coder, wc *coder.WindowCoder) elementHasher {
- // TODO(lostluck): move to a faster hashing library once we can take
dependencies easily.
- hasher := fnv.New64a()
+ hasher := {}
we := MakeWindowEncoder(wc)
switch c.Kind {
case coder.Bytes:
diff --git a/sdks/go/pkg/beam/core/runtime/exec/hash_test.go
b/sdks/go/pkg/beam/core/runtime/exec/hash_test.go
index 7d2c3de..528dfd0 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/hash_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/hash_test.go
@@ -18,7 +18,7 @@ package exec
import (
"encoding/json"
"fmt"
- "hash/fnv"
+ "hash/maphash"
"reflect"
"strings"
"testing"
@@ -32,7 +32,7 @@ import (
func BenchmarkPrimitives(b *testing.B) {
var value FullValue
- myHash := fnv.New64a()
+ myHash := {}
wfn := window.NewGlobalWindows()
we := MakeWindowEncoder(wfn.Coder())
b.Run("int", func(b *testing.B) {
[beam] branch master updated (d43c755 -> 77b410c)
This is an automated email from the ASF dual-hosted git repository. emilyye pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d43c755 Merge pull request #16371 from [BEAM-13518][Playground] Beam Playground quickstart page on the Beam website add 77b410c Update Java FnAPI beam master (#16572) No new revisions were added by this update. Summary of changes: runners/google-cloud-dataflow-java/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (a00634d -> d43c755)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a00634d Merge pull request #16569 from Revert "Merge pull request
#15863 from [BEAM-13184] Autosharding for …
add d43c755 Merge pull request #16371 from [BEAM-13518][Playground] Beam
Playground quickstart page on the Beam website
No new revisions were added by this update.
Summary of changes:
website/www/site/assets/scss/_global.sass | 3 +
.../content/en/get-started/try-beam-playground.md | 102 +
.../partials/section-menu/en/get-started.html | 1 +
.../{button-colab.html => playground.html} | 20 ++--
4 files changed, 114 insertions(+), 12 deletions(-)
create mode 100644
website/www/site/content/en/get-started/try-beam-playground.md
copy website/www/site/layouts/shortcodes/{button-colab.html =>
playground.html} (59%)
[beam] branch master updated (f31cad0 -> a00634d)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f31cad0 Merge pull request #16373 from [BEAM-13515] [Playground] Hiding lines in an example that are not necessary new 421bc80 Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms" new 4f29108 BEAM-13611 reactivating jdbcio xlang test new a00634d Merge pull request #16569 from Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for … The 34290 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 107 +++-- .../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 39 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 30 -- .../io/external/xlang_jdbcio_it_test.py| 1 - 4 files changed, 16 insertions(+), 161 deletions(-)
