[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 62691601173 Updating config from bot
62691601173 is described below
commit 62691601173d4ed245ec182bc2b5850a87191610
Author: github-actions
AuthorDate: Wed Jun 28 05:35:47 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index 9951a38b15f..704b5f6c501 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -1,7 +1,7 @@
{
"label": "go",
"dateOfLastReviewAssignment": {
-"lostluck": 1687896365292,
+"lostluck": 1687930544686,
"jrmccluskey": 1687930539930,
"youngoli": 1657688896155,
"damccorm": 1680501930289,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4284ec3d897 Updating config from bot
4284ec3d897 is described below
commit 4284ec3d8979797c9ef4986bae17a71f3f63e7e9
Author: github-actions
AuthorDate: Wed Jun 28 05:35:46 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27281.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27281.json
b/scripts/ci/pr-bot/state/pr-state/pr-27281.json
new file mode 100644
index 000..8ea94810368
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27281.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "lostluck"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 66ba0f05121 Updating config from bot
66ba0f05121 is described below
commit 66ba0f051214ec21080f320115827f20e7f03fd9
Author: github-actions
AuthorDate: Wed Jun 28 05:35:43 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index e70edea29c2..9951a38b15f 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -2,7 +2,7 @@
"label": "go",
"dateOfLastReviewAssignment": {
"lostluck": 1687896365292,
-"jrmccluskey": 1687844114098,
+"jrmccluskey": 1687930539930,
"youngoli": 1657688896155,
"damccorm": 1680501930289,
"riteshghorse": 1687898082790
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ce1233b4f47 Updating config from bot
ce1233b4f47 is described below
commit ce1233b4f470f3418030ed5f61c727b71c78c886
Author: github-actions
AuthorDate: Wed Jun 28 05:35:41 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27282.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27282.json
b/scripts/ci/pr-bot/state/pr-state/pr-27282.json
new file mode 100644
index 000..1c2c2e31975
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27282.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch nightly-refs/heads/master updated (2b309e87e08 -> 6cd15e7bed1)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 2b309e87e08 Bump worker image dependency (#27253) add d0806a6807e [#22737] Add line about timer support for Go (#27263) add aba3225d0f7 Bump actions/checkout from 2 to 3 (#27259) add 3844587819c Add Tour of Beam Page (#27244) add 0db241d457b Bump transformers (#27106) add 2a06a270ac5 Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks (#27262) add 4ce16432090 [prism] Fix linting issue for flag var (#27248) add c1a5ce7bf0a Add notebook for image processing using beam (#27034) add 8f21e9e9990 Enable remote caching for Beam builds (#27015) add ee9d37daa97 Remove some null checks for optional SplunkIO parameters (#27177) add 8695d704a59 don't error-log expected offset mismatch errors (#27251) add 61651c3394c Update Release Guide DockerHub beammaintainers team to beam (#27264) add 270e4fe2e2e Add error metrics and DLQ support to GCS/File write schema transforms (#27039) add 2f555c51f18 Disable FirestoreIT tests on Dataflow (#27267) add 664f355ee9c Fix perf tool tests (#27270) add 084585abc81 Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks (#27265) add eac3d63738a Use a more mnemonic variable name (#27217) add bc14623d02f Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266) add 0c55d382988 Start expansion service containers before the controller container add 4ff952afce7 Merge pull request #27273: Start expansion service containers before the controller container add a2ebc8a95f1 Cleanup for GA launch of ReadChangeStream (#27249) add 5f1eae62293 Decrease run log failures severity (#27230) add 6cd15e7bed1 BigTable write SchemaTransform and Python wrapper (#27146) No new revisions were added by this update. Summary of changes: .github/workflows/build_runner_image.yml | 2 +- .github/workflows/run_perf_alert_tool.yml | 6 + .test-infra/jenkins/CommonJobProperties.groovy | 1 + ...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +- CHANGES.md | 2 + .../beam-ml/image_processing_tensorflow.ipynb | 951 + .../internal/code_processing/code_processing.go| 5 +- .../backend/internal/errors/lifecycle_error.go | 13 + runners/google-cloud-dataflow-java/build.gradle| 4 + sdks/go.mod| 12 +- sdks/go.sum| 26 +- sdks/go/cmd/prism/prism.go | 4 +- .../AvroWriteSchemaTransformFormatProvider.java| 45 +- .../CsvWriteSchemaTransformFormatProvider.java | 16 +- .../FileWriteSchemaTransformFormatProvider.java| 6 +- .../FileWriteSchemaTransformFormatProviders.java | 55 ++ .../FileWriteSchemaTransformProvider.java | 18 +- .../JsonWriteSchemaTransformFormatProvider.java| 41 +- .../ParquetWriteSchemaTransformFormatProvider.java | 45 +- .../XmlWriteSchemaTransformFormatProvider.java | 37 +- ...AvroWriteSchemaTransformFormatProviderTest.java | 79 +- .../CsvWriteSchemaTransformFormatProviderTest.java | 9 +- ...FileWriteSchemaTransformFormatProviderTest.java | 4 +- ...ileWriteSchemaTransformFormatProvidersTest.java | 86 ++ ...JsonWriteSchemaTransformFormatProviderTest.java | 60 ++ ...quetWriteSchemaTransformFormatProviderTest.java | 75 ++ .../XmlWriteSchemaTransformFormatProviderTest.java | 43 + .../bigquery/StorageApiWritesShardedRecords.java | 33 +- .../beam/sdk/io/gcp/bigtable/BigtableIO.java | 133 ++- .../BigtableWriteSchemaTransformProvider.java | 253 ++ .../changestreams/action/ChangeStreamAction.java | 41 +- .../action/ReadChangeStreamPartitionAction.java| 31 +- .../dao/BigtableChangeStreamAccessor.java | 44 +- .../changestreams/dao/ChangeStreamDao.java | 14 +- .../gcp/bigtable/changestreams/dao/DaoFactory.java | 15 +- .../changestreams/dao/MetadataTableAdminDao.java | 5 + .../changestreams/dao/MetadataTableDao.java| 47 +- .../changestreams/dofn/InitializeDoFn.java | 35 +- .../dofn/ReadChangeStreamPartitionDoFn.java| 11 +- .../BigtableWriteSchemaTransformProviderIT.java| 413 + .../action/ChangeStreamActionTest.java | 12 +- .../ReadChangeStreamPartitionActionTest.java | 33 +- .../changestreams/dao/MetadataTableDaoTest.java| 31 + .../changestreams/dofn/InitializeDoFnTest.java | 71 +- .../dofn/ReadChangeStreamPartitionDoFnTest.java| 5 +- .../sdk/io/gcp/firestore/it/BaseFirestoreIT.java | 3 - .../org/apache/beam/sdk/io/splunk/SplunkIO.java| 24 - .../docker-compose/docker-compose.yml | 3 + .../inference/runinference_metrics/set
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.32.0 created (now a03891751b4)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.32.0 in repository https://gitbox.apache.org/repos/asf/beam.git at a03891751b4 Bump cloud.google.com/go/pubsub from 1.31.0 to 1.32.0 in /sdks No new revisions were added by this update.
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/datastore-1.12.0 created (now 1bbe5462c14)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/datastore-1.12.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 1bbe5462c14 Bump cloud.google.com/go/datastore from 1.11.0 to 1.12.0 in /sdks No new revisions were added by this update.
[beam] branch asf-site updated: Publishing website 2023/06/28 04:17:16 at commit 6cd15e7
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 6c1240c1a59 Publishing website 2023/06/28 04:17:16 at commit 6cd15e7 6c1240c1a59 is described below commit 6c1240c1a591c9440943ae1d1093b81b4bc8b6c8 Author: jenkins AuthorDate: Wed Jun 28 04:17:16 2023 + Publishing website 2023/06/28 04:17:16 at commit 6cd15e7 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 2250366d508..aebe633b3ba 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/categories/blog/2023-06-28T00:57:00+04:00/blog/2023-06-28T00:57:00+04:00/categories/2023-06-28T00:57:00+04:00/blog/managing-beam-dependencies-in-java/2023-06-28T00:57:00+04:00 [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/categories/blog/2023-06-28T01:14:17+00:00/blog/2023-06-28T01:14:17+00:00/categories/2023-06-28T01:14:17+00:00/blog/managing-beam-dependencies-in-java/2023-06-28T01:14:17+00:00 [...] \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 58d512ae7bb Updating config from bot
58d512ae7bb is described below
commit 58d512ae7bb7f4d214983484c994f7d8e7f3c85c
Author: github-actions
AuthorDate: Wed Jun 28 03:38:33 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27279.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27279.json
b/scripts/ci/pr-bot/state/pr-state/pr-27279.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27279.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: BigTable write SchemaTransform and Python wrapper (#27146)
This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6cd15e7bed1 BigTable write SchemaTransform and Python wrapper (#27146)
6cd15e7bed1 is described below
commit 6cd15e7bed1206cea55a7fd02a715e595ed2a966
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Wed Jun 28 01:14:17 2023 +
BigTable write SchemaTransform and Python wrapper (#27146)
* python wrapper
* schematransform; started some tests
* schematransform and tests done
* python wrapper and tests done
* add tests for _DirectRowMutationsToBeamRow DoFn
* use expansion service
* style fix
* style fix
* use secrets token in instance name to prevent race condition between py37
& py311 tests
* use typing typehints
* style fix
* add missing import
* test fix
* resolve merge issues
* style fix
* raise timeout of dataflow tests
* fix setting timeout
* address reviewer comments
* address reviewer comments
* trivial change
* remove unused imports; lint fixes
* add unittest main call
---
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
.../BigtableWriteSchemaTransformProvider.java | 253 +
.../BigtableWriteSchemaTransformProviderIT.java| 413 +
sdks/python/apache_beam/io/gcp/bigtableio.py | 135 +--
.../apache_beam/io/gcp/bigtableio_it_test.py | 377 +++
sdks/python/apache_beam/io/gcp/bigtableio_test.py | 247 +++-
6 files changed, 1304 insertions(+), 123 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
index d1676fcae46..d1ee27088c7 100644
---
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
@@ -32,7 +32,7 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Dataflow',
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 180)
// Publish all test results to Jenkins
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
new file mode 100644
index 000..f57ea46dcdb
--- /dev/null
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
@@ -0,0 +1,253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigtable;
+
+import static java.util.Optional.ofNullable;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import com.google.bigtable.v2.Mutation;
+import com.google.bigtable.v2.TimestampRange;
+import com.google.protobuf.ByteString;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import
org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteSchemaTransformProvider.BigtableWriteSchemaTransformConfiguration;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transfor
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 9d06db3c365 Updating config from bot 9d06db3c365 is described below commit 9d06db3c365265c3ccd67d9a020d2013b67a05f9 Author: github-actions AuthorDate: Tue Jun 27 23:06:00 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index c196967a723..545871c7ccc 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -9,7 +9,7 @@ "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1687628089801, -"jrmccluskey": 1687586793239, +"jrmccluskey": 1687907156627, "riteshghorse": 1687793708877, "liferoad": 1687887309460 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 43f0061ae98 Updating config from bot
43f0061ae98 is described below
commit 43f0061ae985dfcacaad34791a65c7e5b0c3472d
Author: github-actions
AuthorDate: Tue Jun 27 23:05:58 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27277.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27277.json
b/scripts/ci/pr-bot/state/pr-state/pr-27277.json
new file mode 100644
index 000..959c289c017
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27277.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (a2ebc8a95f1 -> 5f1eae62293)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a2ebc8a95f1 Cleanup for GA launch of ReadChangeStream (#27249) add 5f1eae62293 Decrease run log failures severity (#27230) No new revisions were added by this update. Summary of changes: .../backend/internal/code_processing/code_processing.go | 5 - playground/backend/internal/errors/lifecycle_error.go | 13 + 2 files changed, 17 insertions(+), 1 deletion(-)
[beam] branch master updated: Cleanup for GA launch of ReadChangeStream (#27249)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a2ebc8a95f1 Cleanup for GA launch of ReadChangeStream (#27249)
a2ebc8a95f1 is described below
commit a2ebc8a95f1a86d03278dd161f0d8466e4f01c4c
Author: Jack Dingilian
AuthorDate: Tue Jun 27 16:54:19 2023 -0400
Cleanup for GA launch of ReadChangeStream (#27249)
This includes a series of small changes to release the connector:
- Remove option for users to override heartbeat duration
- Add option to skip creating metadata table and standalone utility to
create metadata table
- Add hard timeout to mutateRow requests as workaround for hanging requests
- Remove excessive RCSP logging for debugging during preview
- Increase read rows timeout to better accomodate large tables
- Remove side effects from InitializeDoFn
- Lower ReadChangeStream deadline to closer align with checkpoint duration
- Add release note to CHANGES.md
---
CHANGES.md | 1 +
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 133 -
.../changestreams/action/ChangeStreamAction.java | 41 +--
.../action/ReadChangeStreamPartitionAction.java| 31 +
.../dao/BigtableChangeStreamAccessor.java | 44 +--
.../changestreams/dao/ChangeStreamDao.java | 14 +--
.../gcp/bigtable/changestreams/dao/DaoFactory.java | 15 ++-
.../changestreams/dao/MetadataTableAdminDao.java | 5 +
.../changestreams/dao/MetadataTableDao.java| 47 +++-
.../changestreams/dofn/InitializeDoFn.java | 35 +-
.../dofn/ReadChangeStreamPartitionDoFn.java| 11 +-
.../action/ChangeStreamActionTest.java | 12 +-
.../ReadChangeStreamPartitionActionTest.java | 33 +++--
.../changestreams/dao/MetadataTableDaoTest.java| 31 +
.../changestreams/dofn/InitializeDoFnTest.java | 71 ++-
.../dofn/ReadChangeStreamPartitionDoFnTest.java| 5 +-
16 files changed, 271 insertions(+), 258 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 6c55e276f91..aee9b96d9f4 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -59,6 +59,7 @@
## I/Os
* Support for X source added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
+* Support for Bigtable Change Streams added in Java
`BigtableIO.ReadChangeStream`
([#27183](https://github.com/apache/beam/issues/27183))
## New Features / Improvements
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 319c3139021..06497458a66 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -48,6 +48,7 @@ import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
import org.apache.beam.sdk.io.gcp.bigtable.changestreams.ChangeStreamMetrics;
import org.apache.beam.sdk.io.gcp.bigtable.changestreams.UniqueIdGenerator;
import org.apache.beam.sdk.io.gcp.bigtable.changestreams.action.ActionFactory;
+import
org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao.BigtableChangeStreamAccessor;
import org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao.DaoFactory;
import
org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao.MetadataTableAdminDao;
import
org.apache.beam.sdk.io.gcp.bigtable.changestreams.dofn.DetectNewPartitionsDoFn;
@@ -310,7 +311,6 @@ public class BigtableIO {
*
*
* {@link BigtableIO.ReadChangeStream#withStartTime} which defaults to
now.
- * {@link BigtableIO.ReadChangeStream#withHeartbeatDuration} with
defaults to 1 seconds.
* {@link BigtableIO.ReadChangeStream#withMetadataTableProjectId}
which defaults to value
* from {@link BigtableIO.ReadChangeStream#withProjectId}
* {@link BigtableIO.ReadChangeStream#withMetadataTableInstanceId}
which defaults to value
@@ -1797,8 +1797,6 @@ public class BigtableIO {
RESUME_OR_NEW,
// Same as RESUME_OR_NEW except if previous pipeline doesn't exist, don't
start.
RESUME_OR_FAIL,
-// Start a new pipeline. Overriding existing pipeline with the same name.
-NEW,
// This skips cleaning up previous pipeline metadata and starts a new
pipeline. This should
// only be used to skip cleanup in tests
@VisibleForTesting
@@ -1827,8 +1825,6 @@ public class BigtableIO {
abstract @Nullable Instant getEndTime();
-abstract @Nullable Duration getHeartbeatDuration();
-
abstract @Nullable String getChangeStreamName();
abstract @Nullable ExistingPipelineOptions getExistingPipelineOptions();
@@ -1837,6 +1833,8 @@ pu
[beam] 01/01: Merge pull request #27273: Start expansion service containers before the controller container
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4ff952afce7a3d0f422749f663c0ed6f702905a1 Merge: bc14623d02f 0c55d382988 Author: Chamikara Jayalath AuthorDate: Tue Jun 27 13:53:47 2023 -0700 Merge pull request #27273: Start expansion service containers before the controller container sdks/java/transform-service/docker-compose/docker-compose.yml | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (bc14623d02f -> 4ff952afce7)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from bc14623d02f Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266) add 0c55d382988 Start expansion service containers before the controller container new 4ff952afce7 Merge pull request #27273: Start expansion service containers before the controller container The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/transform-service/docker-compose/docker-compose.yml | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 06b2d80b528 Updating config from bot
06b2d80b528 is described below
commit 06b2d80b528c13ec31b9919749ddbcce3c1b23e3
Author: github-actions
AuthorDate: Tue Jun 27 20:36:47 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27276.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27276.json
b/scripts/ci/pr-bot/state/pr-state/pr-27276.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27276.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 885122a7261 Updating config from bot 885122a7261 is described below commit 885122a72613650305ec7dbbe2b6580aa1638c8a Author: github-actions AuthorDate: Tue Jun 27 20:34:47 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index c5f0e11ed8d..e70edea29c2 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -5,6 +5,6 @@ "jrmccluskey": 1687844114098, "youngoli": 1657688896155, "damccorm": 1680501930289, -"riteshghorse": 1687800926548 +"riteshghorse": 1687898082790 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 79a7d95f303 Updating config from bot
79a7d95f303 is described below
commit 79a7d95f3035588528f4d0d92ae1b485811242b6
Author: github-actions
AuthorDate: Tue Jun 27 20:34:45 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27274.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27274.json
b/scripts/ci/pr-bot/state/pr-state/pr-27274.json
new file mode 100644
index 000..aa3638a74ec
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27274.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "riteshghorse"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new bc14623d02f Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266) bc14623d02f is described below commit bc14623d02fae00a37979fe8dcbcf94cd8aa6397 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Jun 27 16:06:55 2023 -0400 Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.51.2 to 1.52.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.51.2...bigquery/v1.52.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 6 +++--- sdks/go.sum | 14 +++--- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index ffb42c0e25c..b31880973d8 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -23,7 +23,7 @@ module github.com/apache/beam/sdks/v2 go 1.19 require ( - cloud.google.com/go/bigquery v1.51.2 + cloud.google.com/go/bigquery v1.52.0 cloud.google.com/go/bigtable v1.18.1 cloud.google.com/go/datastore v1.11.0 cloud.google.com/go/profiler v0.3.1 @@ -154,8 +154,8 @@ require ( github.com/zeebo/xxh3 v1.0.2 // indirect go.opencensus.io v0.24.0 // indirect golang.org/x/crypto v0.10.0 // indirect - golang.org/x/mod v0.9.0 // indirect - golang.org/x/tools v0.7.0 // indirect + golang.org/x/mod v0.10.0 // indirect + golang.org/x/tools v0.9.1 // indirect golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 493dbb4dc71..0f320b259ea 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -13,15 +13,15 @@ cloud.google.com/go v0.110.2/go.mod h1:k04UEeEtb6ZBRTv3dZz4CeJC3jKGxyhl0sAiVVqux cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= -cloud.google.com/go/bigquery v1.51.2 h1:p6SZQJBh64rNJB/9V5O0jvMBI8O/XV5rJKlhmmCU+2o= -cloud.google.com/go/bigquery v1.51.2/go.mod h1:6YYSJ37dAY1HyMDq/+XByPmzsC52MgzNXhxjlTzIVCM= +cloud.google.com/go/bigquery v1.52.0 h1:JKLNdxI0N+TIUWD6t9KN646X27N5dQWq9dZbbTWZ8hc= +cloud.google.com/go/bigquery v1.52.0/go.mod h1:3b/iXjRQGU4nKa87cXeg6/gogLjO8C6PmuM8i5Bi/u4= cloud.google.com/go/bigtable v1.18.1 h1:SxQk9Bj6OKxeiuvevG/KBjqGn/7X8heZbWfK0tYkFd8= cloud.google.com/go/bigtable v1.18.1/go.mod h1:NAVyfJot9jlo+KmgWLUJ5DJGwNDoChzAcrecLpmuAmY= cloud.google.com/go/compute v1.19.3 h1:DcTwsFgGev/wV5+q8o2fzgcHOaac+DKGC91ZlvpsQds= cloud.google.com/go/compute v1.19.3/go.mod h1:qxvISKp/gYnXkSAD1ppcSOveRAmzxicEv/JlizULFrI= cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= -cloud.google.com/go/datacatalog v1.13.0 h1:4H5IJiyUE0X6ShQBqgFFZvGGcrwGVndTwUSLP4c52gw= +cloud.google.com/go/datacatalog v1.14.0 h1:ScW+U7bcoNYdS4xuVfnNdt2nR2j7esPyFJEZFW87ZzY= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= cloud.google.com/go/datastore v1.11.0 h1:iF6I/HaLs3Ado8uRKMvZRvF/ZLkWaWE9i8AiHzbC774= @@ -508,8 +508,8 @@ golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzB golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs= -golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= +golang.org/x/mod v0.10.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.52.0 deleted (was abc3d9ddcdb)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.52.0 in repository https://gitbox.apache.org/repos/asf/beam.git was abc3d9ddcdb Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new aa9de4bb88d Updating config from bot
aa9de4bb88d is described below
commit aa9de4bb88d3d7b525426e567b50809dc72c6f60
Author: github-actions
AuthorDate: Tue Jun 27 20:06:07 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27266.json | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27266.json
b/scripts/ci/pr-bot/state/pr-state/pr-27266.json
index 242a48d7d3b..8ea94810368 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-27266.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27266.json
@@ -1,7 +1,9 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "lostluck"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2679f632104 Updating config from bot
2679f632104 is described below
commit 2679f6321048114443a84d5b748531c0f937d710
Author: github-actions
AuthorDate: Tue Jun 27 20:06:09 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index 34ee50433c3..c5f0e11ed8d 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -1,7 +1,7 @@
{
"label": "go",
"dateOfLastReviewAssignment": {
-"lostluck": 1687498550530,
+"lostluck": 1687896365292,
"jrmccluskey": 1687844114098,
"youngoli": 1657688896155,
"damccorm": 1680501930289,
[beam] branch master updated: Use a more mnemonic variable name (#27217)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eac3d63738a Use a more mnemonic variable name (#27217)
eac3d63738a is described below
commit eac3d63738a0188ab9399f97738d65296b1f2bb8
Author: jubebo <38796540+jub...@users.noreply.github.com>
AuthorDate: Tue Jun 27 22:02:18 2023 +0200
Use a more mnemonic variable name (#27217)
Co-authored-by: Julian Berzborn
---
sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py
b/sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py
index 4c25aa62e0b..cf6c26d1a7f 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py
@@ -67,7 +67,7 @@ def generate_user_type_from_bq_schema(the_table_schema):
the_table_schema)
if the_schema == {}:
raise ValueError("Encountered an empty schema")
- dict_of_tuples = []
+ field_names_and_types = []
for i in range(len(the_schema['fields'])):
if the_schema['fields'][i]['type'] in BIG_QUERY_TO_PYTHON_TYPES:
typ = bq_field_to_type(
@@ -76,9 +76,9 @@ def generate_user_type_from_bq_schema(the_table_schema):
raise ValueError(
f"Encountered "
f"an unsupported type: {the_schema['fields'][i]['type']!r}")
-# TODO svetaksundhar@: Map remaining BQ types
-dict_of_tuples.append((the_schema['fields'][i]['name'], typ))
- sample_schema = beam.typehints.schemas.named_fields_to_schema(dict_of_tuples)
+field_names_and_types.append((the_schema['fields'][i]['name'], typ))
+ sample_schema = beam.typehints.schemas.named_fields_to_schema(
+ field_names_and_types)
usertype = beam.typehints.schemas.named_tuple_from_schema(sample_schema)
return usertype
[beam] branch dependabot/go_modules/sdks/google.golang.org/api-0.129.0 deleted (was 820946e4549)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/api-0.129.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 820946e4549 Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks (#27265)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 084585abc81 Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks (#27265) 084585abc81 is described below commit 084585abc81bd335c70a12fd57e8779edf83e886 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Jun 27 15:55:46 2023 -0400 Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks (#27265) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.128.0 to 0.129.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.128.0...v0.129.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 88a4ef2a215..ffb42c0e25c 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -57,7 +57,7 @@ require ( golang.org/x/sync v0.3.0 golang.org/x/sys v0.9.0 golang.org/x/text v0.10.0 - google.golang.org/api v0.128.0 + google.golang.org/api v0.129.0 google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc google.golang.org/grpc v1.56.1 google.golang.org/protobuf v1.31.0 @@ -118,7 +118,7 @@ require ( github.com/google/pprof v0.0.0-20221103000818-d260c55eee4c // indirect github.com/google/renameio/v2 v2.0.0 // indirect github.com/google/s2a-go v0.1.4 // indirect - github.com/googleapis/enterprise-certificate-proxy v0.2.4 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.2.5 // indirect github.com/googleapis/gax-go/v2 v2.11.0 // indirect github.com/gorilla/handlers v1.5.1 // indirect github.com/gorilla/mux v1.8.0 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 39eb5299298..493dbb4dc71 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -286,8 +286,8 @@ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/enterprise-certificate-proxy v0.2.4 h1:uGy6JWR/uMIILU8wbf+OkstIrNiMjGpEIyhx8f6W7s4= -github.com/googleapis/enterprise-certificate-proxy v0.2.4/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k= +github.com/googleapis/enterprise-certificate-proxy v0.2.5 h1:UR4rDjcgpgEnqpIEvkiqTYKBCKLNmlge2eVjoZfySzM= +github.com/googleapis/enterprise-certificate-proxy v0.2.5/go.mod h1:RxW0N9901Cko1VOCW3SXCpWP+mlIEkk2tP7jnHy9a3w= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.11.0 h1:9V9PWXEsWnPpQhu/PeQIkS4eGzMlTLGgt80cUUI8Ki4= @@ -661,8 +661,8 @@ google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= -google.golang.org/api v0.128.0 h1:RjPESny5CnQRn9V6siglged+DZCgfu9l6mO9dkX9VOg= -google.golang.org/api v0.128.0/go.mod h1:Y611qgqaE92On/7g65MQgxYul3c0rEB894kniWLY750= +google.golang.org/api v0.129.0 h1:2XbdjjNfFPXQyufzQVwPf1RRnHH8Den2pfNE2jw7L8w= +google.golang.org/api v0.129.0/go.mod h1:dFjiXlanKwWE3612X97llhsoI36FAoIiRj3aTl5b/zE= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
[beam] branch dependabot/gradle/com.palantir.gradle.docker-gradle-docker-0.35.0 deleted (was ea9a510ac8f)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/gradle/com.palantir.gradle.docker-gradle-docker-0.35.0 in repository https://gitbox.apache.org/repos/asf/beam.git was ea9a510ac8f Bump com.palantir.gradle.docker:gradle-docker from 0.34.0 to 0.35.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Fix perf tool tests (#27270)
This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 664f355ee9c Fix perf tool tests (#27270)
664f355ee9c is described below
commit 664f355ee9c631afb4d80440e837e7b48028bbc8
Author: Anand Inguva <34158215+ananding...@users.noreply.github.com>
AuthorDate: Tue Jun 27 15:04:05 2023 -0400
Fix perf tool tests (#27270)
* Fix tests
* Don't publish data when running analyzer in PR
* Update .github/workflows/run_perf_alert_tool.yml
* Run perf analysis tests during schedules
* Don't authenticate during pull_request
---
.github/workflows/run_perf_alert_tool.yml | 6 ++
sdks/python/apache_beam/testing/analyzers/perf_analysis.py | 6 --
.../apache_beam/testing/analyzers/perf_analysis_test.py| 14 --
3 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/run_perf_alert_tool.yml
b/.github/workflows/run_perf_alert_tool.yml
index 510db83f869..1cdcd858e61 100644
--- a/.github/workflows/run_perf_alert_tool.yml
+++ b/.github/workflows/run_perf_alert_tool.yml
@@ -23,6 +23,10 @@ on:
workflow_dispatch:
schedule:
- cron: '5 22 * * *'
+ pull_request:
+branches: ['master']
+tags: 'v*'
+paths: ['sdks/python/apache_beam/testing/**']
jobs:
python_run_change_point_analysis:
name: Run Change Point Analysis.
@@ -37,6 +41,7 @@ jobs:
with:
python-version: 3.8
- name: Authenticate on GCP
+if: github.event_name != 'pull_request'
uses: google-github-actions/setup-gcloud@v0
with:
service_account_key: ${{ secrets.GCP_SA_KEY }}
@@ -55,6 +60,7 @@ jobs:
working-directory: ./sdks/python/apache_beam/testing/analyzers
shell: bash
run: python perf_analysis.py
+if: github.event_name != 'pull_request'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Run change point analysis tests.
diff --git a/sdks/python/apache_beam/testing/analyzers/perf_analysis.py
b/sdks/python/apache_beam/testing/analyzers/perf_analysis.py
index e5e8a3a7b2f..7f1ffbb944e 100644
--- a/sdks/python/apache_beam/testing/analyzers/perf_analysis.py
+++ b/sdks/python/apache_beam/testing/analyzers/perf_analysis.py
@@ -122,7 +122,6 @@ def run_change_point_analysis(params, test_name,
big_query_metrics_fetcher):
change_point_index=change_point_index,
timestamps=timestamps,
min_runs_between_change_points=min_runs_between_change_points)
- logging.debug("Performance alert is %s for test %s" % (is_alert, test_name))
if is_alert:
issue_number, issue_url = create_performance_alert(
metric_name, test_name, timestamps,
@@ -173,7 +172,10 @@ def run(config_file_path: Optional[str] = None) -> None:
big_query_metrics_fetcher = BigQueryMetricsFetcher()
for test_name, params in tests_config.items():
-run_change_point_analysis(params, test_name, big_query_metrics_fetcher)
+run_change_point_analysis(
+params=params,
+test_name=test_name,
+big_query_metrics_fetcher=big_query_metrics_fetcher)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/testing/analyzers/perf_analysis_test.py
b/sdks/python/apache_beam/testing/analyzers/perf_analysis_test.py
index 000175e6388..fabf185a41d 100644
--- a/sdks/python/apache_beam/testing/analyzers/perf_analysis_test.py
+++ b/sdks/python/apache_beam/testing/analyzers/perf_analysis_test.py
@@ -21,6 +21,7 @@ import os
import unittest
import mock
+import numpy as np
import pandas as pd
# pylint: disable=ungrouped-imports
@@ -54,7 +55,8 @@ def get_existing_issue_data(**kwargs):
# change point found at index 10. So passing 10 in the
# existing issue data in mock method.
return pd.DataFrame([{
- constants._CHANGE_POINT_TIMESTAMP_LABEL: 10, constants._ISSUE_NUMBER: 0
+ constants._CHANGE_POINT_TIMESTAMP_LABEL: 10,
+ constants._ISSUE_NUMBER: np.array([0])
}])
@@ -70,7 +72,7 @@ class TestChangePointAnalysis(unittest.TestCase):
] * 20
self.timestamps = list(range(5))
self.params = {
-'test_name': 'fake_test',
+'test_description': 'fake_description',
'metrics_dataset': 'fake_dataset',
'metrics_table': 'fake_table',
'project': 'fake_project',
@@ -102,7 +104,7 @@ class TestChangePointAnalysis(unittest.TestCase):
def test_validate_config(self):
test_keys = {
-'test_name',
+'test_description',
'metrics_dataset',
'metrics_table',
'project',
@@ -146,7 +148,7 @@ class TestChangePointAnalysis(unittest.TestCase):
def test_no_alerts_when_no_change_points(self):
is_alert = analysis.run_change_point_analysis(
params=self.params,
-test_id=self.test_id,
[beam] branch master updated (270e4fe2e2e -> 2f555c51f18)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 270e4fe2e2e Add error metrics and DLQ support to GCS/File write schema transforms (#27039) add 2f555c51f18 Disable FirestoreIT tests on Dataflow (#27267) No new revisions were added by this update. Summary of changes: runners/google-cloud-dataflow-java/build.gradle | 4 .../java/org/apache/beam/sdk/io/gcp/firestore/it/BaseFirestoreIT.java | 3 --- 2 files changed, 4 insertions(+), 3 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 034c004b2b8 Updating config from bot
034c004b2b8 is described below
commit 034c004b2b84ea1a23f9352d2afcc8993bdc5f5d
Author: github-actions
AuthorDate: Tue Jun 27 18:25:40 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27273.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27273.json
b/scripts/ci/pr-bot/state/pr-state/pr-27273.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27273.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a77c7048cb6 Updating config from bot
a77c7048cb6 is described below
commit a77c7048cb61af042a16e66ad1c1f3526a5eceda
Author: github-actions
AuthorDate: Tue Jun 27 18:06:40 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-26632.json | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-26632.json
b/scripts/ci/pr-bot/state/pr-state/pr-26632.json
index 242a48d7d3b..6272b4e797d 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-26632.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-26632.json
@@ -1,7 +1,9 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "AnandInguva"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1a993998413 Updating config from bot
1a993998413 is described below
commit 1a993998413cf737e46d0292c6f246659c831165
Author: github-actions
AuthorDate: Tue Jun 27 18:06:42 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json
b/scripts/ci/pr-bot/state/reviewers-for-label-python.json
index 8916f2f43b9..c196967a723 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json
@@ -1,7 +1,7 @@
{
"label": "python",
"dateOfLastReviewAssignment": {
-"AnandInguva": 1687566297442,
+"AnandInguva": 1687889199180,
"yeandy": 1665802753763,
"TheNeuralBit": 1667896849319,
"ryanthompson591": 1670002443548,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 89696a14b92 Updating config from bot
89696a14b92 is described below
commit 89696a14b9207adec32dea19ad82f9b449495080
Author: github-actions
AuthorDate: Tue Jun 27 17:35:23 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27257.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27257.json
b/scripts/ci/pr-bot/state/pr-state/pr-27257.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27257.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 8ac8591b60f Updating config from bot
8ac8591b60f is described below
commit 8ac8591b60f94e1456f520f5f8ae7b7579d8efc7
Author: github-actions
AuthorDate: Tue Jun 27 17:35:16 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index d72cdf65f94..a41775d1d35 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1687844122310,
+"damccorm": 1687887309460,
"Abacn": 1687844127128
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 990a7ab89e4 Updating config from bot 990a7ab89e4 is described below commit 990a7ab89e4837d3467ee147ccb7872f8223d6e7 Author: github-actions AuthorDate: Tue Jun 27 17:35:13 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 6baed90b3c6..8916f2f43b9 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -11,6 +11,6 @@ "damccorm": 1687628089801, "jrmccluskey": 1687586793239, "riteshghorse": 1687793708877, -"liferoad": 1687430099585 +"liferoad": 1687887309460 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2b3fab9e011 Updating config from bot
2b3fab9e011 is described below
commit 2b3fab9e011e226f95087ef986bd5362171a8ae3
Author: github-actions
AuthorDate: Tue Jun 27 17:35:11 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27270.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27270.json
b/scripts/ci/pr-bot/state/pr-state/pr-27270.json
new file mode 100644
index 000..5216b6274e8
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27270.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "liferoad",
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Add error metrics and DLQ support to GCS/File write schema transforms (#27039)
This is an automated email from the ASF dual-hosted git repository.
johncasey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 270e4fe2e2e Add error metrics and DLQ support to GCS/File write schema
transforms (#27039)
270e4fe2e2e is described below
commit 270e4fe2e2ed326eeb84eea132079202c7ffa184
Author: Nick Li <56149585+nickuncaged1...@users.noreply.github.com>
AuthorDate: Tue Jun 27 10:24:29 2023 -0700
Add error metrics and DLQ support to GCS/File write schema transforms
(#27039)
* init
* fixing
* fixed avro schema transform
* json error metrics
* parquet error metric
* xml error metrics
* fix comments
* spotless
* add tests
* spotless
* Change name of ErrorCounterFn
* add generic test for BeamRowMapperWithDlq
---
.../AvroWriteSchemaTransformFormatProvider.java| 45 +++
.../CsvWriteSchemaTransformFormatProvider.java | 16 ++--
.../FileWriteSchemaTransformFormatProvider.java| 6 +-
.../FileWriteSchemaTransformFormatProviders.java | 55 ++
.../FileWriteSchemaTransformProvider.java | 18 -
.../JsonWriteSchemaTransformFormatProvider.java| 41 ---
.../ParquetWriteSchemaTransformFormatProvider.java | 45 ---
.../XmlWriteSchemaTransformFormatProvider.java | 37 +++---
...AvroWriteSchemaTransformFormatProviderTest.java | 79 +++-
.../CsvWriteSchemaTransformFormatProviderTest.java | 9 ++-
...FileWriteSchemaTransformFormatProviderTest.java | 4 +-
...ileWriteSchemaTransformFormatProvidersTest.java | 86 ++
...JsonWriteSchemaTransformFormatProviderTest.java | 60 +++
...quetWriteSchemaTransformFormatProviderTest.java | 75 +++
.../XmlWriteSchemaTransformFormatProviderTest.java | 43 +++
15 files changed, 560 insertions(+), 59 deletions(-)
diff --git
a/sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/AvroWriteSchemaTransformFormatProvider.java
b/sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/AvroWriteSchemaTransformFormatProvider.java
index 34b74089405..71e5c2e8a8b 100644
---
a/sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/AvroWriteSchemaTransformFormatProvider.java
+++
b/sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/AvroWriteSchemaTransformFormatProvider.java
@@ -19,23 +19,32 @@ package org.apache.beam.sdk.io.fileschematransform;
import static
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformFormatProviders.getNumShards;
import static
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformFormatProviders.getShardNameTemplate;
+import static
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformProvider.ERROR_SCHEMA;
+import static
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformProvider.ERROR_TAG;
+import static
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformProvider.RESULT_TAG;
import com.google.auto.service.AutoService;
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.extensions.avro.coders.AvroCoder;
import org.apache.beam.sdk.extensions.avro.io.AvroIO;
import org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils;
+import
org.apache.beam.sdk.io.fileschematransform.FileWriteSchemaTransformFormatProviders.BeamRowMapperWithDlq;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
/** A {@link FileWriteSchemaTransformFormatProvider} for avro format. */
@AutoService(FileWriteSchemaTransformFormatProvider.class)
public class AvroWriteSchemaTransformFormatProvider
implements FileWriteSchemaTransformFormatProvider {
+ static final TupleTag ERROR_FN_OUPUT_TAG = new
TupleTag() {};
@Override
public String identifier() {
@@ -44,25 +53,31 @@ public class AvroWriteSchemaTransformFormatProvider
/**
* Builds a {@link PTransform} that transforms a {@link Row} {@link
PCollection} into result
- * {@link PCollection} file names written using {@link AvroIO.Write}.
+ * {@link PCollectionTuple} with two tags, one for file names written using
{@link AvroIO.Write},
+ * another for errored-out rows.
*/
@Override
- public PTransform, PCollection> buildTransfor
[beam] branch master updated (8695d704a59 -> 61651c3394c)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 8695d704a59 don't error-log expected offset mismatch errors (#27251) add 61651c3394c Update Release Guide DockerHub beammaintainers team to beam (#27264) No new revisions were added by this update. Summary of changes: website/www/site/content/en/contribute/release-guide.md | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-)
[beam] branch master updated (ee9d37daa97 -> 8695d704a59)
This is an automated email from the ASF dual-hosted git repository. johncasey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ee9d37daa97 Remove some null checks for optional SplunkIO parameters (#27177) add 8695d704a59 don't error-log expected offset mismatch errors (#27251) No new revisions were added by this update. Summary of changes: .../bigquery/StorageApiWritesShardedRecords.java | 33 +- 1 file changed, 19 insertions(+), 14 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new cfd0c191d44 Updating config from bot
cfd0c191d44 is described below
commit cfd0c191d44b24a56b0b688bf8646449d676af3a
Author: github-actions
AuthorDate: Tue Jun 27 16:06:46 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-26632.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-26632.json
b/scripts/ci/pr-bot/state/pr-state/pr-26632.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-26632.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b67d3b3663d Updating config from bot
b67d3b3663d is described below
commit b67d3b3663d9067e1ea0d009dd88b49e8aca1c07
Author: github-actions
AuthorDate: Tue Jun 27 16:06:25 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27265.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27265.json
b/scripts/ci/pr-bot/state/pr-state/pr-27265.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27265.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bff8c5c39e3 Updating config from bot
bff8c5c39e3 is described below
commit bff8c5c39e3a7de374b43743cf08f8b363a0f76b
Author: github-actions
AuthorDate: Tue Jun 27 16:06:21 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27266.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27266.json
b/scripts/ci/pr-bot/state/pr-state/pr-27266.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27266.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (8f21e9e9990 -> ee9d37daa97)
This is an automated email from the ASF dual-hosted git repository. johncasey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 8f21e9e9990 Enable remote caching for Beam builds (#27015) add ee9d37daa97 Remove some null checks for optional SplunkIO parameters (#27177) No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/splunk/SplunkIO.java| 24 -- 1 file changed, 24 deletions(-)
[beam] branch master updated: Enable remote caching for Beam builds (#27015)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8f21e9e9990 Enable remote caching for Beam builds (#27015)
8f21e9e9990 is described below
commit 8f21e9e999078118f31c26df821e42ec2eb9f5c4
Author: Clay Johnson
AuthorDate: Tue Jun 27 11:05:11 2023 -0500
Enable remote caching for Beam builds (#27015)
---
.test-infra/jenkins/CommonJobProperties.groovy | 1 +
settings.gradle.kts| 11 +--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy
b/.test-infra/jenkins/CommonJobProperties.groovy
index 4cb7f330b8b..93c81ff9af3 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -108,6 +108,7 @@ class CommonJobProperties {
credentialsBinding {
string("CODECOV_TOKEN", "beam-codecov-token")
string("COVERALLS_REPO_TOKEN", "beam-coveralls-token")
+usernamePassword("GRADLE_ENTERPRISE_CACHE_USERNAME",
"GRADLE_ENTERPRISE_CACHE_PASSWORD", "beam_cache_node_credentials")
}
timestamps()
colorizeOutput()
diff --git a/settings.gradle.kts b/settings.gradle.kts
index 591c1b78833..adc66beb04f 100644
--- a/settings.gradle.kts
+++ b/settings.gradle.kts
@@ -49,8 +49,15 @@ buildCache {
local {
isEnabled = true
}
- remote(gradleEnterprise.buildCache) {
-isEnabled = false
+ remote {
+url = uri("https://beam-cache.apache.org/cache/";)
+isAllowUntrustedServer = false
+credentials {
+ username = System.getenv("GRADLE_ENTERPRISE_CACHE_USERNAME")
+ password = System.getenv("GRADLE_ENTERPRISE_CACHE_PASSWORD")
+}
+isEnabled = true
+isPush = isCi
}
}
[beam] branch master updated: Add notebook for image processing using beam (#27034)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c1a5ce7bf0a Add notebook for image processing using beam (#27034)
c1a5ce7bf0a is described below
commit c1a5ce7bf0aee88ee8f827a72a15c4ce4b5b14d3
Author: Reeba Qureshi <64488642+reeba...@users.noreply.github.com>
AuthorDate: Tue Jun 27 21:33:52 2023 +0530
Add notebook for image processing using beam (#27034)
* Add notebook for image processing using beam
* Delete image_processing_tensorflow.ipynb
* add image processing use case using tensorflow
* Delete image_processing_tensorflow.ipynb
* Add image processing use case after suggestions
* Delete image_processing_beam.ipynb
* Add image processing with implemented suggestions
---
.../beam-ml/image_processing_tensorflow.ipynb | 951 +
1 file changed, 951 insertions(+)
diff --git a/examples/notebooks/beam-ml/image_processing_tensorflow.ipynb
b/examples/notebooks/beam-ml/image_processing_tensorflow.ipynb
new file mode 100644
index 000..0914653f1de
--- /dev/null
+++ b/examples/notebooks/beam-ml/image_processing_tensorflow.ipynb
@@ -0,0 +1,951 @@
+{
+ "cells": [
+{
+ "cell_type": "code",
+ "source": [
+"# @title ## Licensed to the Apache Software Foundation (ASF),
Version 2.0 (the \"License\")\n",
+"\n",
+"# Licensed to the Apache Software Foundation (ASF) under one\n",
+"# or more contributor license agreements. See the NOTICE file\n",
+"# distributed with this work for additional information\n",
+"# regarding copyright ownership. The ASF licenses this file\n",
+"# to you under the Apache License, Version 2.0 (the\n",
+"# \"License\"); you may not use this file except in compliance\n",
+"# with the License. You may obtain a copy of the License at\n",
+"#\n",
+"# http://www.apache.org/licenses/LICENSE-2.0\n";,
+"#\n",
+"# Unless required by applicable law or agreed to in writing,\n",
+"# software distributed under the License is distributed on an\n",
+"# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+"# KIND, either express or implied. See the License for the\n",
+"# specific language governing permissions and limitations\n",
+"# under the License"
+ ],
+ "metadata": {
+"id": "NsNImDL8TGM1"
+ },
+ "execution_count": null,
+ "outputs": []
+},
+{
+ "cell_type": "markdown",
+ "source": [
+"# Image Processing using Apache Beam\n",
+"\n",
+"\n",
+" \n",
+"https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/image_processing_tensorflow.ipynb\";>https://raw.githubusercontent.com/google/or-tools/main/tools/colab_32px.png\";
/>Run in Google Colab\n",
+" \n",
+" \n",
+"https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/image_processing_tensorflow.ipynb\";>https://raw.githubusercontent.com/google/or-tools/main/tools/github_32px.png\";
/>View source on GitHub\n",
+" \n",
+"\n",
+"\n"
+ ],
+ "metadata": {
+"id": "SwN0Rj4cJSg5"
+ }
+},
+{
+ "cell_type": "markdown",
+ "source": [
+"Image Processing is a machine learning technique to read, analyze and
extract meaningful information from images. It involves multiple steps such as
applying various preprocessing fuctions, getting predictions from a model,
storing the predictions in a useful format, etc. Apache Beam is a suitable tool
to handle these tasks and build a structured workflow. This notebook
demonstrates the use of Apache Beam in image processing and performs the
following:\n",
+"* Import and preprocess the CIFAR-10 dataset\n",
+"* Train a TensorFlow model to classify images\n",
+"* Store the model in Google Cloud and create a model handler\n",
+"* Build a Beam pipeline to:\n",
+" 1. Create a
[PCollection]('https://beam.apache.org/documentation/programming-guide/#pcollections')
of input images\n",
+" 2. Perform preprocessing
[transforms]('https://beam.apache.org/documentation/programming-guide/#transforms')\n",
+" 3. RunInference to get predictions from the previously trained
model\n",
+" 4. Store the results\n",
+"\n",
+"For more information on using Apache Beam for machine learning, have
a look at [AI/ML Pipelines using
Beam]('https://beam.apache.org/documentation/ml/overview/')."
+ ],
+ "metadata": {
+"id": "yxLoBQxocAOv"
+ }
+},
+{
+ "cell_type": "markdown",
+ "metadata": {
+"id": "OSZrRmHl9NQY"
+
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1800596e57f Updating config from bot
1800596e57f is described below
commit 1800596e57f60775b36c96fb05d573be037dfbe8
Author: github-actions
AuthorDate: Tue Jun 27 15:35:26 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 09689662dbe..291b6d517e0 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -3,7 +3,7 @@
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
"kennknowles": 1687809976492,
-"robertwb": 1687781676770,
+"robertwb": 1687880122793,
"kileys": 1674428678843,
"apilloud": 1678822446183,
"Abacn": 1687868107889,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 59ed4b5181b Updating config from bot
59ed4b5181b is described below
commit 59ed4b5181be2f09149f3287f019e06e3bee4734
Author: github-actions
AuthorDate: Tue Jun 27 15:35:24 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27177.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27177.json
b/scripts/ci/pr-bot/state/pr-state/pr-27177.json
index 5fc6dfb8e72..6321ad10c1d 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-27177.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27177.json
@@ -1,11 +1,11 @@
{
"commentedAboutFailingChecks": false,
"reviewersAssignedForLabels": {
-"java": "bvolpato",
+"java": "robertwb",
"io": "bvolpato"
},
"nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
- "committerAssigned": false
+ "committerAssigned": true
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1ca913315bf Updating config from bot
1ca913315bf is described below
commit 1ca913315bfdac0c86efa35e852c0a6f5e883d9f
Author: github-actions
AuthorDate: Tue Jun 27 15:31:37 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27267.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27267.json
b/scripts/ci/pr-bot/state/pr-state/pr-27267.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27267.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.52.0 created (now abc3d9ddcdb)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.52.0 in repository https://gitbox.apache.org/repos/asf/beam.git at abc3d9ddcdb Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks No new revisions were added by this update.
[beam] branch dependabot/go_modules/sdks/google.golang.org/api-0.129.0 created (now 820946e4549)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/api-0.129.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 820946e4549 Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks No new revisions were added by this update.
[beam] branch master updated: [prism] Fix linting issue for flag var (#27248)
This is an automated email from the ASF dual-hosted git repository.
jrmccluskey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4ce16432090 [prism] Fix linting issue for flag var (#27248)
4ce16432090 is described below
commit 4ce164320905d2447b715ae5818bc44fb1716dc5
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Tue Jun 27 11:23:48 2023 -0400
[prism] Fix linting issue for flag var (#27248)
---
sdks/go/cmd/prism/prism.go | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/sdks/go/cmd/prism/prism.go b/sdks/go/cmd/prism/prism.go
index 32c6fbadd52..f00a16c9b2d 100644
--- a/sdks/go/cmd/prism/prism.go
+++ b/sdks/go/cmd/prism/prism.go
@@ -31,7 +31,7 @@ import (
var (
jobManagerEndpoint = flag.String("jm_override", "", "set to only stand
up a web ui that refers to a seperate JobManagement endpoint")
- serveHttp = flag.Bool("serve_http", true, "enable or disable
the web ui")
+ serveHTTP = flag.Bool("serve_http", true, "enable or disable
the web ui")
)
func main() {
@@ -41,7 +41,7 @@ func main() {
if err != nil {
log.Fatalf("error creating job server: %v", err)
}
- if *serveHttp {
+ if *serveHTTP {
if err := prism.CreateWebServer(ctx, cli, prism.Options{Port:
8074}); err != nil {
log.Fatalf("error creating web server: %v", err)
}
[beam] branch dependabot/go_modules/sdks/google.golang.org/protobuf-1.31.0 deleted (was a81706f73cc)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/protobuf-1.31.0 in repository https://gitbox.apache.org/repos/asf/beam.git was a81706f73cc Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks (#27262)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 2a06a270ac5 Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks (#27262) 2a06a270ac5 is described below commit 2a06a270ac524b0046fa2faa203e84fdc86fb766 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Jun 27 11:22:30 2023 -0400 Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks (#27262) Bumps google.golang.org/protobuf from 1.30.0 to 1.31.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index e72fb5ef9e6..88a4ef2a215 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -60,7 +60,7 @@ require ( google.golang.org/api v0.128.0 google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc google.golang.org/grpc v1.56.1 - google.golang.org/protobuf v1.30.0 + google.golang.org/protobuf v1.31.0 gopkg.in/retry.v1 v1.0.3 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 diff --git a/sdks/go.sum b/sdks/go.sum index cfd565027b3..39eb5299298 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -721,8 +721,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= -google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 07652578297 Updating config from bot
07652578297 is described below
commit 07652578297ceddec41941c6598efbb5dd1250af
Author: github-actions
AuthorDate: Tue Jun 27 15:17:05 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27264.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27264.json
b/scripts/ci/pr-bot/state/pr-state/pr-27264.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27264.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/pip/sdks/python/apache_beam/examples/inference/runinference_metrics/transformers-4.30.0 deleted (was 715d157c31d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/sdks/python/apache_beam/examples/inference/runinference_metrics/transformers-4.30.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 715d157c31d Bump transformers The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump transformers (#27106)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 0db241d457b Bump transformers (#27106) 0db241d457b is described below commit 0db241d457bed5b8a991d6b48d0106cc409f4baf Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Jun 27 11:01:54 2023 -0400 Bump transformers (#27106) Bumps [transformers](https://github.com/huggingface/transformers) from 4.21.0 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](https://github.com/huggingface/transformers/compare/v4.21.0...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .../python/apache_beam/examples/inference/runinference_metrics/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py b/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py index d022d250be5..102699b6bec 100644 --- a/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py +++ b/sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py @@ -29,7 +29,7 @@ import setuptools from setuptools import find_packages REQUIREMENTS = [ -"apache-beam[gcp]==2.41.0", "transformers==4.21.0", "torch==1.13.1" +"apache-beam[gcp]==2.41.0", "transformers==4.30.0", "torch==1.13.1" ] setuptools.setup(
[beam] branch master updated: Add Tour of Beam Page (#27244)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 3844587819c Add Tour of Beam Page (#27244) 3844587819c is described below commit 3844587819cfe21c262db125332d77af511bb427 Author: Svetak Sundhar AuthorDate: Tue Jun 27 10:29:37 2023 -0400 Add Tour of Beam Page (#27244) * add tour of beam page to redirect. * add tour of beam page to redirect. * add tour of beam page to redirect. --- .../www/site/content/en/get-started/tour-of-beam.md | 21 + 1 file changed, 21 insertions(+) diff --git a/website/www/site/content/en/get-started/tour-of-beam.md b/website/www/site/content/en/get-started/tour-of-beam.md new file mode 100644 index 000..0deca6be0b9 --- /dev/null +++ b/website/www/site/content/en/get-started/tour-of-beam.md @@ -0,0 +1,21 @@ +--- +title: "The Tour of Beam" +--- + + + +# The Tour of Beam + +The "Tour of Beam" is an interactive way of learning to write Beam code with a sandbox, where you can write and run pipelines while walking through various concepts. Please [click here](https://tour.beam.apache.org/) to try it out. \ No newline at end of file
[beam] branch dependabot/github_actions/actions/checkout-3 deleted (was 811036d934c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/actions/checkout-3 in repository https://gitbox.apache.org/repos/asf/beam.git was 811036d934c Bump actions/checkout from 2 to 3 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump actions/checkout from 2 to 3 (#27259)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aba3225d0f7 Bump actions/checkout from 2 to 3 (#27259)
aba3225d0f7 is described below
commit aba3225d0f7deb0e20bc58bec3d994c273f7f1ba
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Tue Jun 27 09:15:03 2023 -0400
Bump actions/checkout from 2 to 3 (#27259)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
.github/workflows/build_runner_image.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/build_runner_image.yml
b/.github/workflows/build_runner_image.yml
index cdd2b34e3ce..4c8f9282a4e 100644
--- a/.github/workflows/build_runner_image.yml
+++ b/.github/workflows/build_runner_image.yml
@@ -31,7 +31,7 @@ jobs:
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 1f4ede97383 Updating config from bot 1f4ede97383 is described below commit 1f4ede97383db8c2c38cd6464fba18e76ac99f59 Author: github-actions AuthorDate: Tue Jun 27 12:15:15 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 2f577bddf34..0ea388975d7 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -4,7 +4,7 @@ "chamikaramj": 1687436043135, "johnjcasey": 1687430099585, "pabloem": 1687349652119, -"Abacn": 1687308600513, +"Abacn": 1687868109258, "ahmedabu98": 1687809976492, "bvolpato": 1687829756029, "manavgarg": 1687586793239
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a7c00fe90fa Updating config from bot a7c00fe90fa is described below commit a7c00fe90faddf2d8e6cc1fd691e4bb1391cbe79 Author: github-actions AuthorDate: Tue Jun 27 12:15:13 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 9034e778726..09689662dbe 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -6,7 +6,7 @@ "robertwb": 1687781676770, "kileys": 1674428678843, "apilloud": 1678822446183, -"Abacn": 1687586793239, +"Abacn": 1687868107889, "bvolpato": 1687829756028 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ffcf41d3d4b Updating config from bot
ffcf41d3d4b is described below
commit ffcf41d3d4bc8ad54bff8270f81fad7225814aa4
Author: github-actions
AuthorDate: Tue Jun 27 12:15:11 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-26948.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-26948.json
b/scripts/ci/pr-bot/state/pr-state/pr-26948.json
index e872c66b8d2..faeedc4416c 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-26948.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-26948.json
@@ -1,8 +1,8 @@
{
"commentedAboutFailingChecks": false,
"reviewersAssignedForLabels": {
-"java": "robertwb",
-"io": "chamikaramj"
+"java": "Abacn",
+"io": "Abacn"
},
"nextAction": "Reviewers",
"stopReviewerNotifications": false,
[beam] branch master updated: [#22737] Add line about timer support for Go (#27263)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d0806a6807e [#22737] Add line about timer support for Go (#27263) d0806a6807e is described below commit d0806a6807ee408e0a7cc6b42f3c2869fd7df022 Author: Robert Burke AuthorDate: Tue Jun 27 13:31:32 2023 +0200 [#22737] Add line about timer support for Go (#27263) --- CHANGES.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGES.md b/CHANGES.md index 69a893b49ed..6c55e276f91 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -65,6 +65,7 @@ * Allow prebuilding large images when using `--prebuild_sdk_container_engine=cloud_build`, like images depending on `tensorflow` or `torch` ([#27023](https://github.com/apache/beam/pull/27023)). * Disabled `pip` cache when installing packages on the workers. This reduces the size of prebuilt Python container images ([#27035](https://github.com/apache/beam/pull/27035)). * Select dedicated avro datum reader and writer (Java) ([#18874](https://github.com/apache/beam/issues/18874)). +* Timer API for the Go SDK (Go) ([#22737](https://github.com/apache/beam/issues/22737)). ## Breaking Changes
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 9bfc6c3e73d Updating config from bot 9bfc6c3e73d is described below commit 9bfc6c3e73dd625ccbdf0434951dcc626a41d878 Author: github-actions AuthorDate: Tue Jun 27 09:05:05 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-27230.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27230.json b/scripts/ci/pr-bot/state/pr-state/pr-27230.json index d1ca4634459..2389c0bb160 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-27230.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-27230.json @@ -4,7 +4,7 @@ "no-matching-label": "Abacn" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c331086c3df Updating config from bot
c331086c3df is described below
commit c331086c3df7048d59c860921dcfec80f8b98573
Author: github-actions
AuthorDate: Tue Jun 27 08:36:47 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json
b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json
index 52907f56446..b81cf5c1bdf 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json
@@ -1,7 +1,7 @@
{
"label": "no-matching-label",
"dateOfLastReviewAssignment": {
-"Abacn": 1686764088333,
+"Abacn": 1687855003583,
"AnandInguva": 1686773169139,
"chamikaramj": 1686846969452,
"damccorm": 1686920834837,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bad184389a3 Updating config from bot
bad184389a3 is described below
commit bad184389a33a983fd3802b62d88c30897f21ba1
Author: github-actions
AuthorDate: Tue Jun 27 08:36:45 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27230.json | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27230.json
b/scripts/ci/pr-bot/state/pr-state/pr-27230.json
index 242a48d7d3b..d1ca4634459 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-27230.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27230.json
@@ -1,7 +1,9 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"no-matching-label": "Abacn"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2a333c7a28a Updating config from bot
2a333c7a28a is described below
commit 2a333c7a28ae1fb437db9dde061870f58c242d3b
Author: github-actions
AuthorDate: Tue Jun 27 07:01:00 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-27263.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-27263.json
b/scripts/ci/pr-bot/state/pr-state/pr-27263.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-27263.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
