(beam) branch nightly-refs/heads/master updated (93a5bc76577 -> a435f452211)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 93a5bc76577 [Python] Use retry decorator for retrying (#31273) add 28a7199e80b [Java] Add job name to GCS custom audit info (#31316) add 4f0c5fb Done with python scripts/tools/bomupgrader.py 26.39.0 based on https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md (#31318) add d15d98c0026 Update code-change-guide.md (#31333) add d7165f940cc Upgrade cibuildwheel version. (#31321) add 44177d1a9f3 [Python] BigQuery handler for enrichment transform (#31295) add a435f452211 [Java] Disable soft delete policy when creating a default bucket for a project. (#31324) No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json | 1 + CHANGES.md | 2 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +- contributor-docs/code-change-guide.md | 4 +- .../container/license_scripts/dep_urls_java.yaml | 2 +- .../google-cloud-platform-core/build.gradle| 24 ++ .../sdk/extensions/gcp/options/GcpOptions.java | 56 +++- .../beam/sdk/extensions/gcp/util/GcsUtil.java | 45 .../gcp/util/RetryHttpRequestInitializer.java | 11 + .../beam/sdk/extensions/gcp/util/Transport.java| 32 ++- .../sdk/extensions/gcp/options/GcpOptionsIT.java | 93 +++ .../sdk/extensions/gcp/options/GcpOptionsTest.java | 12 + .../sdk/extensions/gcp/util/TransportTest.java | 35 ++- sdks/python/apache_beam/io/requestresponse.py | 121 ++--- sdks/python/apache_beam/transforms/enrichment.py | 14 +- .../transforms/enrichment_handlers/bigquery.py | 256 ++ .../enrichment_handlers/bigquery_it_test.py| 289 + .../enrichment_handlers/bigquery_test.py | 70 + sdks/python/build.gradle | 2 +- sdks/python/scripts/generate_pydoc.sh | 1 + 20 files changed, 995 insertions(+), 83 deletions(-) create mode 100644 sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsIT.java create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_test.py
(beam) branch master updated: Update hamcrest dependency documentation version (#31309)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f851cbb317e Update hamcrest dependency documentation version (#31309)
f851cbb317e is described below
commit f851cbb317e24e60ee2ab908bc8eb37b8010623d
Author: Maarten Vercruysse <67873169+mv...@users.noreply.github.com>
AuthorDate: Sat May 18 04:21:58 2024 +0200
Update hamcrest dependency documentation version (#31309)
---
.../www/site/content/en/documentation/pipelines/test-your-pipeline.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
b/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
index 1ab47dd540a..744b12aa162 100644
--- a/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
+++ b/website/www/site/content/en/documentation/pipelines/test-your-pipeline.md
@@ -107,8 +107,8 @@ Any Java code that uses `PAssert` must link in `JUnit` and
`Hamcrest`. If you're
{{< highlight java >}}
org.hamcrest
-hamcrest-all
-1.3
+hamcrest
+2.2
test
{{< /highlight >}}
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 84f9922b72c Updating config from bot
84f9922b72c is described below
commit 84f9922b72c7586a20128ce117b8bdbef4902092
Author: github-actions
AuthorDate: Sat May 18 00:49:29 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index 64cd6fc907d..4a8a1afa203 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -1,7 +1,7 @@
{
"label": "go",
"dateOfLastReviewAssignment": {
-"lostluck": 1715024110074,
+"lostluck": 1715993365583,
"jrmccluskey": 1715900747506,
"youngoli": 1657688896155,
"damccorm": 1680501930289,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new d5e0909215f Updating config from bot
d5e0909215f is described below
commit d5e0909215f08699bd903404a0c4e30d59ae16c5
Author: github-actions
AuthorDate: Sat May 18 00:49:26 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31340.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31340.json
b/scripts/ci/pr-bot/state/pr-state/pr-31340.json
new file mode 100644
index 000..8ea94810368
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31340.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "lostluck"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch asf-site updated: Publishing website 2024/05/17 23:37:24 at commit a435f45
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new a8d32eea96d Publishing website 2024/05/17 23:37:24 at commit a435f45 a8d32eea96d is described below commit a8d32eea96db396e1cce939a9ebb756e2cda712c Author: runner AuthorDate: Fri May 17 23:37:25 2024 + Publishing website 2024/05/17 23:37:24 at commit a435f45 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 12c2ffde532..f525ebf63e6 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-17T09:30:05-07:00/categories/blog/2024-05-17T09:30:05-07:00/blog/2024-05-17T09:30:05-07:00/categories/2024-05-17T09:30:05-07:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-05-17T16:05:06-04:00/categories/blog/2024-05-17T16:05:06-04:00/blog/2024-05-17T16:05:06-04:00/categories/2024-05-17T16:05:06-04:00/catego [...] \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 0b72b45f10e Updating config from bot 0b72b45f10e is described below commit 0b72b45f10e0ec712cd2d7bf104b290ff08f4726 Author: github-actions AuthorDate: Fri May 17 20:06:17 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-31337.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31337.json b/scripts/ci/pr-bot/state/pr-state/pr-31337.json index 5216b6274e8..570978fe1f4 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-31337.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-31337.json @@ -7,5 +7,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
(beam) branch master updated: [Java] Disable soft delete policy when creating a default bucket for a project. (#31324)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a435f452211 [Java] Disable soft delete policy when creating a default
bucket for a project. (#31324)
a435f452211 is described below
commit a435f452211d6e38a3c54687d24588a392aa87d6
Author: Shunping Huang
AuthorDate: Fri May 17 16:05:06 2024 -0400
[Java] Disable soft delete policy when creating a default bucket for a
project. (#31324)
* Disable soft delete policy when creating a default bucket for a project.
Also, getBucket() and removeBucket() are added into GcsUtil.
* Add random number to prefix to avoid collision on multiple test instances.
* Remove default bucket before creating in the integration test.
* Add one line in CHANGES.md to mention this change and fix a typo.
---
CHANGES.md | 1 +
.../google-cloud-platform-core/build.gradle| 24 ++
.../sdk/extensions/gcp/options/GcpOptions.java | 56 +
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 45 +++
.../sdk/extensions/gcp/options/GcpOptionsIT.java | 93 ++
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 12 +++
6 files changed, 217 insertions(+), 14 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index dd66e9192b6..a6c61da4510 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -72,6 +72,7 @@
Template variables can be passed with the (json-formatted)
`--jinja_variables` flag.
* DataFrame API now supports pandas 2.1.x and adds 12 more string functions
for Series.([#31185](https://github.com/apache/beam/pull/31185)).
* Added BigQuery handler for enrichment transform (Python)
([#31295](https://github.com/apache/beam/pull/31295))
+* Disable soft delete policy when creating the default bucket for a project
(Java) ([#31324](https://github.com/apache/beam/pull/31324)).
## Breaking Changes
diff --git a/sdks/java/extensions/google-cloud-platform-core/build.gradle
b/sdks/java/extensions/google-cloud-platform-core/build.gradle
index d4dfd46f745..4af856cc9ff 100644
--- a/sdks/java/extensions/google-cloud-platform-core/build.gradle
+++ b/sdks/java/extensions/google-cloud-platform-core/build.gradle
@@ -87,8 +87,32 @@ task integrationTestKms(type: Test) {
}
}
+// Note that no runner is specified here, so tests running under this task
should not be running
+// pipelines.
+task integrationTestNoKms(type: Test) {
+ group = "Verification"
+ def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
+ def gcpTempRoot = project.findProperty('gcpTempRoot') ?:
'gs://temp-storage-for-end-to-end-tests-cmek'
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+ "--project=${gcpProject}",
+ "--tempRoot=${gcpTempRoot}",
+ ])
+
+ // Disable Gradle cache: these ITs interact with live service that should
always be considered "out of date"
+ outputs.upToDateWhen { false }
+
+ include '**/*IT.class'
+ maxParallelForks 4
+ classpath = sourceSets.test.runtimeClasspath
+ testClassesDirs = sourceSets.test.output.classesDirs
+ useJUnit {
+excludeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
+}
+
task postCommit {
group = "Verification"
description = "Integration tests of GCP connectors using the DirectRunner."
dependsOn integrationTestKms
+ dependsOn integrationTestNoKms
}
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
index 3c65f0fa748..2686a53cbcd 100644
---
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
@@ -27,6 +27,7 @@ import com.google.api.client.util.Sleeper;
import com.google.api.services.cloudresourcemanager.CloudResourceManager;
import com.google.api.services.cloudresourcemanager.model.Project;
import com.google.api.services.storage.model.Bucket;
+import com.google.api.services.storage.model.Bucket.SoftDeletePolicy;
import com.google.auth.Credentials;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.cloud.hadoop.util.ChainingHttpRequestInitializer;
@@ -392,40 +393,67 @@ public interface GcpOptions extends
GoogleApiDebugOptions, PipelineOptions {
return tempLocation;
}
-/**
- * Creates a default bucket or verifies the existence and proper access
control of an existing
- * default bucket. Returns the location if successful.
- */
@VisibleForTesting
-static String
(beam) branch tvalentyn-gha updated (9d64f3c6a2a -> 9019925e3e3)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 9d64f3c6a2a drop: restrict tests to py312 only. discard 82e4a6bb0c0 Upgrade cibuildwheel and also install setuptools. discard 97991328554 Also install setuptools. discard e1ce6602619 Upgrade cibuildwheel. discard 485913128fe Support Python 3.12 in Beam. discard f4417e1395f Support Python 3.12 in Dataflow. discard 98010e16018 Build Python 3.12 wheels. discard ca99deabe27 Switch remaining GHA workflows to test Python 3.12. discard 02fe949438f Switch Py3.11 Gradle suites to use Python 3.12. discard 232633fae6d Gradle definitions for Python 3.12 discard 69abdd16576 Add a tox suite for Py312. discard b30085d10de Define Python 3.12 test environments in tox. add 93a5bc76577 [Python] Use retry decorator for retrying (#31273) add 28a7199e80b [Java] Add job name to GCS custom audit info (#31316) add 4f0c5fb Done with python scripts/tools/bomupgrader.py 26.39.0 based on https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md (#31318) add d15d98c0026 Update code-change-guide.md (#31333) add d7165f940cc Upgrade cibuildwheel version. (#31321) add 44177d1a9f3 [Python] BigQuery handler for enrichment transform (#31295) add 3bcd1539bc5 Define Python 3.12 test environments in tox. add 347e082c7c6 Add a tox suite for Py312. add 73624667677 Gradle definitions for Python 3.12 add 6971be752bc Switch Py3.11 Gradle suites to use Python 3.12. add e76cd89eac0 Switch remaining GHA workflows to test Python 3.12. add f99b3bc2210 Build Python 3.12 wheels. add a098bbb88d5 Support Python 3.12 in Dataflow. add 8d95889f9a4 Support Python 3.12 in Beam. add 5a743b49b93 Also install setuptools. add b89ba7776dd Upgrade cibuildwheel and also install setuptools. add 21312037d42 drop: restrict tests to py312 only. add 9019925e3e3 drop: maybe reword: don't set CIBW variable to use stdlib. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9d64f3c6a2a) \ N -- N -- N refs/heads/tvalentyn-gha (9019925e3e3) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json | 1 + .github/workflows/build_wheels.yml | 1 - CHANGES.md | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +- contributor-docs/code-change-guide.md | 4 +- .../container/license_scripts/dep_urls_java.yaml | 2 +- .../gcp/util/RetryHttpRequestInitializer.java | 11 + .../beam/sdk/extensions/gcp/util/Transport.java| 32 ++- .../sdk/extensions/gcp/util/TransportTest.java | 35 ++- sdks/python/apache_beam/io/requestresponse.py | 121 ++--- sdks/python/apache_beam/io/requestresponse_test.py | 61 +++-- sdks/python/apache_beam/transforms/enrichment.py | 14 +- .../transforms/enrichment_handlers/bigquery.py | 256 ++ .../enrichment_handlers/bigquery_it_test.py| 289 + .../enrichment_handlers/bigquery_test.py | 70 + sdks/python/build.gradle | 1 - sdks/python/scripts/generate_pydoc.sh | 1 + 17 files changed, 806 insertions(+), 102 deletions(-) create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_test.py
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 889d7cf79fe Updating config from bot 889d7cf79fe is described below commit 889d7cf79fe7c18662be0c3a390339d45f420cce Author: github-actions AuthorDate: Fri May 17 19:34:38 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 4dc27913a26..46bd2ef9e8f 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -11,7 +11,7 @@ "damccorm": 1715905285934, "jrmccluskey": 1715798129262, "riteshghorse": 1715775239490, -"liferoad": 1715711701469, +"liferoad": 1715974475601, "shunping": 1715715917227 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1dc0eb8a7b0 Updating config from bot
1dc0eb8a7b0 is described below
commit 1dc0eb8a7b090ec76c9d7ef2a033294582753294
Author: github-actions
AuthorDate: Fri May 17 19:34:39 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 79c1140133f..bd4f620acd1 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1715808906638,
+"damccorm": 1715974475601,
"Abacn": 1715871947470
}
}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 542853b1857 Updating config from bot
542853b1857 is described below
commit 542853b185778bce6be3cde6d95e854afcc43398
Author: github-actions
AuthorDate: Fri May 17 19:34:36 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31337.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31337.json
b/scripts/ci/pr-bot/state/pr-state/pr-31337.json
new file mode 100644
index 000..5216b6274e8
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31337.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "liferoad",
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated: [Python] BigQuery handler for enrichment transform (#31295)
This is an automated email from the ASF dual-hosted git repository.
riteshghorse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 44177d1a9f3 [Python] BigQuery handler for enrichment transform (#31295)
44177d1a9f3 is described below
commit 44177d1a9f32c3707f60a6a7a3b67334372a4291
Author: Ritesh Ghorse
AuthorDate: Fri May 17 14:58:42 2024 -0400
[Python] BigQuery handler for enrichment transform (#31295)
* bigquery enrichment with it
* unit tests
* more integration tests, move BE to RRIO
* trigger postcommit
* fix doc, project name
* refactor, improve doc comments
* refactor
* remove query_fn check
---
.github/trigger_files/beam_PostCommit_Python.json | 1 +
CHANGES.md | 1 +
sdks/python/apache_beam/io/requestresponse.py | 121 ++---
sdks/python/apache_beam/transforms/enrichment.py | 14 +-
.../transforms/enrichment_handlers/bigquery.py | 256 ++
.../enrichment_handlers/bigquery_it_test.py| 289 +
.../enrichment_handlers/bigquery_test.py | 70 +
sdks/python/scripts/generate_pydoc.sh | 1 +
8 files changed, 705 insertions(+), 48 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index c4edaa85a89..63bd5651def 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,3 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run"
}
+
diff --git a/CHANGES.md b/CHANGES.md
index bce9636237e..dd66e9192b6 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -71,6 +71,7 @@
* Beam YAML now supports the jinja templating syntax.
Template variables can be passed with the (json-formatted)
`--jinja_variables` flag.
* DataFrame API now supports pandas 2.1.x and adds 12 more string functions
for Series.([#31185](https://github.com/apache/beam/pull/31185)).
+* Added BigQuery handler for enrichment transform (Python)
([#31295](https://github.com/apache/beam/pull/31295))
## Breaking Changes
diff --git a/sdks/python/apache_beam/io/requestresponse.py
b/sdks/python/apache_beam/io/requestresponse.py
index 4458aa59c18..d7011e5a8ff 100644
--- a/sdks/python/apache_beam/io/requestresponse.py
+++ b/sdks/python/apache_beam/io/requestresponse.py
@@ -28,6 +28,8 @@ from datetime import timedelta
from typing import Any
from typing import Dict
from typing import Generic
+from typing import List
+from typing import Mapping
from typing import Optional
from typing import Tuple
from typing import TypeVar
@@ -42,6 +44,7 @@ from apache_beam.coders import coders
from apache_beam.io.components.adaptive_throttler import AdaptiveThrottler
from apache_beam.metrics import Metrics
from apache_beam.ml.inference.vertex_ai_inference import MSEC_TO_SEC
+from apache_beam.transforms.util import BatchElements
from apache_beam.utils import retry
RequestT = TypeVar('RequestT')
@@ -143,6 +146,10 @@ class Caller(contextlib.AbstractContextManager,
"""
return ""
+ def batch_elements_kwargs(self) -> Mapping[str, Any]:
+"""Returns a kwargs suitable for `beam.BatchElements`."""
+return {}
+
class ShouldBackOff(abc.ABC):
"""
@@ -476,53 +483,67 @@ class _RedisCaller(Caller):
def __enter__(self):
self.client = redis.Redis(self.host, self.port, **self.kwargs)
- def __call__(self, element, *args, **kwargs):
-if self.mode == _RedisMode.READ:
- cache_request = self.source_caller.get_cache_key(element)
- # check if the caller is a enrichment handler. EnrichmentHandler
- # provides the request format for cache.
- if cache_request:
-encoded_request = self.request_coder.encode(cache_request)
- else:
-encoded_request = self.request_coder.encode(element)
-
- encoded_response = self.client.get(encoded_request)
- if not encoded_response:
-# no cache entry present for this request.
+ def _read_cache(self, element):
+cache_request = self.source_caller.get_cache_key(element)
+# check if the caller is a enrichment handler. EnrichmentHandler
+# provides the request format for cache.
+if cache_request:
+ encoded_request = self.request_coder.encode(cache_request)
+else:
+ encoded_request = self.request_coder.encode(element)
+
+encoded_response = self.client.get(encoded_request)
+if not encoded_response:
+ # no cache entry present for this request.
+ return element, None
+
+if self.response_coder is None:
+ try:
+response_dict = json.loads(encoded_response.decode('utf-8'))
+response = beam.Row(**response_dict)
+ except Exception:
+_LOGGER.warning(
+'cannot decode
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a8758939de4 Updating config from bot
a8758939de4 is described below
commit a8758939de4bff3d9dec423d6a8b4d63cdb0db64
Author: github-actions
AuthorDate: Fri May 17 18:35:47 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 50cc6b55fee..83ebeeb621e 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -3,7 +3,7 @@
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
"kennknowles": 1715775257531,
-"robertwb": 1715704626620,
+"robertwb": 1715970945695,
"kileys": 1674428678843,
"apilloud": 1678822446183,
"Abacn": 1715965530690,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 4f917e1074a Updating config from bot 4f917e1074a is described below commit 4f917e1074aa18624de895c743a2d258d6240439 Author: github-actions AuthorDate: Fri May 17 18:35:49 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 44bfba4f50b..778a7a27243 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -4,7 +4,7 @@ "chamikaramj": 1715715354400, "johnjcasey": 1715704626620, "pabloem": 1691787951165, -"Abacn": 1715516043716, +"Abacn": 1715970945695, "ahmedabu98": 1715775249004, "bvolpato": 1712595969392, "manavgarg": 1690826779210,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f682e17ff40 Updating config from bot
f682e17ff40 is described below
commit f682e17ff40e2c02d9bfc61c7ac3863b7d202b67
Author: github-actions
AuthorDate: Fri May 17 18:35:46 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31323.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31323.json
b/scripts/ci/pr-bot/state/pr-state/pr-31323.json
new file mode 100644
index 000..a7c19975bb1
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31323.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "robertwb",
+"io": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch tvalentyn-gha updated (97991328554 -> 9d64f3c6a2a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 97991328554 Also install setuptools. add 82e4a6bb0c0 Upgrade cibuildwheel and also install setuptools. add 9d64f3c6a2a drop: restrict tests to py312 only. No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python.yml | 2 +- .github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +- .github/workflows/build_wheels.yml| 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new e782ecdd716 Updating config from bot e782ecdd716 is described below commit e782ecdd7160c8c132bc0bc4d821cd558af0defc Author: github-actions AuthorDate: Fri May 17 17:05:33 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index fd434c9f74c..50cc6b55fee 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -6,7 +6,7 @@ "robertwb": 1715704626620, "kileys": 1674428678843, "apilloud": 1678822446183, -"Abacn": 1715516043156, +"Abacn": 1715965530690, "bvolpato": 1712595969392, "m-trieu": 1715715354400, "damondouglas": 1715614479475
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new fef9aa1af15 Updating config from bot fef9aa1af15 is described below commit fef9aa1af15c95638a664d531f3f5cf49aabb597 Author: github-actions AuthorDate: Fri May 17 17:05:35 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 15256d1effc..44bfba4f50b 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -8,7 +8,7 @@ "ahmedabu98": 1715775249004, "bvolpato": 1712595969392, "manavgarg": 1690826779210, -"damondouglas": 1715362487046, +"damondouglas": 1715965530691, "shunping": 1715614479475 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2846ca9faf6 Updating config from bot
2846ca9faf6 is described below
commit 2846ca9faf657b93f56eacde8670de890e0a54a0
Author: github-actions
AuthorDate: Fri May 17 17:05:32 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31334.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31334.json
b/scripts/ci/pr-bot/state/pr-state/pr-31334.json
new file mode 100644
index 000..32a3e8f8f49
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31334.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "Abacn",
+"io": "damondouglas"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated: Upgrade cibuildwheel version. (#31321)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d7165f940cc Upgrade cibuildwheel version. (#31321)
d7165f940cc is described below
commit d7165f940cc700e369363cbd8ef3469f31448914
Author: tvalentyn
AuthorDate: Fri May 17 09:30:05 2024 -0700
Upgrade cibuildwheel version. (#31321)
---
sdks/python/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index e21f5db75ad..e10b2a90dfb 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -166,7 +166,7 @@ platform_identifiers_map.each { platform, idsuffix ->
args '-c', ". ${envdir}/bin/activate && " +
// note: sync cibuildwheel version with GitHub Action
// .github/workflows/build_wheel.yml:build_wheels "Install
cibuildwheel" step
- "pip install cibuildwheel==2.9.0 && " +
+ "pip install cibuildwheel==2.17.0 && " +
"cibuildwheel --print-build-identifiers --platform
${platform} --archs ${archs} && " +
"cibuildwheel --output-dir ${buildDir} --platform ${platform}
--archs ${archs} "
}
(beam) branch master updated: Update code-change-guide.md (#31333)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d15d98c0026 Update code-change-guide.md (#31333) d15d98c0026 is described below commit d15d98c0026aa56232c4bac3832ea3bd3fd1dd47 Author: liferoad AuthorDate: Fri May 17 12:11:35 2024 -0400 Update code-change-guide.md (#31333) --- contributor-docs/code-change-guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributor-docs/code-change-guide.md b/contributor-docs/code-change-guide.md index ee1944ccc65..935a2c6276c 100644 --- a/contributor-docs/code-change-guide.md +++ b/contributor-docs/code-change-guide.md @@ -505,7 +505,7 @@ To run an integration test on the Dataflow Runner, follow these steps: ``` cd sdks/python - pip install build && python -m build –sdist + pip install build && python -m build --sdist ``` The tarball file is generated in the `sdks/python/sdist/` directory. @@ -570,7 +570,7 @@ You can use the [official Beam SDK container image](https://gcr.io/apache-beam-t To run your pipeline with modified beam code, follow these steps: -1. Build the Beam SDK tarball. Under `sdks/python`, run `python -m build –sdist`. For more details, +1. Build the Beam SDK tarball. Under `sdks/python`, run `python -m build --sdist`. For more details, see [Run an integration test](#run-an-integration-test) on this page. 2. Install the Apache Beam Python SDK in your Python virtual environment with the necessary
(beam) branch master updated: Done with python scripts/tools/bomupgrader.py 26.39.0 based on https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md (#31318)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4f0c5fb Done with python scripts/tools/bomupgrader.py 26.39.0
based on
https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md
(#31318)
4f0c5fb is described below
commit 4f0c5fb966cbea17b9a8342a0ecbc4b82e72
Author: Tomo Suzuki
AuthorDate: Fri May 17 11:26:38 2024 -0400
Done with python scripts/tools/bomupgrader.py 26.39.0 based on
https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md
(#31318)
Latest bom is 26.39 -
https://github.com/googleapis/java-cloud-bom/releases/tag/v26.39.0
---
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 8
sdks/java/container/license_scripts/dep_urls_java.yaml| 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index bc56a13f19c..f7fcd8d916f 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -601,12 +601,12 @@ class BeamModulePlugin implements Plugin {
def dbcp2_version = "2.9.0"
def errorprone_version = "2.10.0"
// [bomupgrader] determined by: com.google.api:gax, consistent with:
google_cloud_platform_libraries_bom
-def gax_version = "2.47.0"
+def gax_version = "2.48.0"
def google_ads_version = "26.0.0"
def google_clients_version = "2.0.0"
def google_cloud_bigdataoss_version = "2.2.16"
// [bomupgrader] determined by: com.google.cloud:google-cloud-spanner,
consistent with: google_cloud_platform_libraries_bom
-def google_cloud_spanner_version = "6.65.1"
+def google_cloud_spanner_version = "6.66.0"
def google_code_gson_version = "2.10.1"
def google_oauth_clients_version = "1.34.1"
// [bomupgrader] determined by: io.grpc:grpc-netty, consistent with:
google_cloud_platform_libraries_bom
@@ -747,13 +747,13 @@ class BeamModulePlugin implements Plugin {
google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc", //
google_cloud_platform_libraries_bom sets version
google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog", //
google_cloud_platform_libraries_bom sets version
google_cloud_dataflow_java_proto_library_all:
"com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
-google_cloud_datastore_v1_proto_client :
"com.google.cloud.datastore:datastore-v1-proto-client:2.19.1", //
[bomupgrader] sets version
+google_cloud_datastore_v1_proto_client :
"com.google.cloud.datastore:datastore-v1-proto-client:2.19.2", //
[bomupgrader] sets version
google_cloud_firestore :
"com.google.cloud:google-cloud-firestore", //
google_cloud_platform_libraries_bom sets version
google_cloud_pubsub :
"com.google.cloud:google-cloud-pubsub", // google_cloud_platform_libraries_bom
sets version
google_cloud_pubsublite :
"com.google.cloud:google-cloud-pubsublite", //
google_cloud_platform_libraries_bom sets version
// [bomupgrader] the BOM version is set by
scripts/tools/bomupgrader.py. If update manually, also update
// libraries-bom version on
sdks/java/container/license_scripts/dep_urls_java.yaml
-google_cloud_platform_libraries_bom :
"com.google.cloud:libraries-bom:26.38.0",
+google_cloud_platform_libraries_bom :
"com.google.cloud:libraries-bom:26.39.0",
google_cloud_spanner:
"com.google.cloud:google-cloud-spanner", // google_cloud_platform_libraries_bom
sets version
google_cloud_spanner_test :
"com.google.cloud:google-cloud-spanner:$google_cloud_spanner_version:tests",
google_cloud_vertexai :
"com.google.cloud:google-cloud-vertexai", //
google_cloud_platform_libraries_bom sets version
diff --git a/sdks/java/container/license_scripts/dep_urls_java.yaml
b/sdks/java/container/license_scripts/dep_urls_java.yaml
index 837a9824d6e..c3a3fd2708f 100644
--- a/sdks/java/container/license_scripts/dep_urls_java.yaml
+++ b/sdks/java/container/license_scripts/dep_urls_java.yaml
@@ -46,7 +46,7 @@ jaxen:
'1.1.6':
type: "3-Clause BSD"
libraries-bom:
- '26.38.0':
+ '26.39.0':
license:
"https://raw.githubusercontent.com/GoogleCloudPlatform/cloud-opensource-java/master/LICENSE;
type: "Apache License 2.0"
paranamer:
(beam) branch master updated: [Java] Add job name to GCS custom audit info (#31316)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 28a7199e80b [Java] Add job name to GCS custom audit info (#31316)
28a7199e80b is described below
commit 28a7199e80bba9be8eb7f87ae7ab88c343492b40
Author: Shunping Huang
AuthorDate: Fri May 17 11:21:40 2024 -0400
[Java] Add job name to GCS custom audit info (#31316)
* Add job name to GCS custom audit info in Java
* Apply spotless
---
.../gcp/util/RetryHttpRequestInitializer.java | 11 +++
.../beam/sdk/extensions/gcp/util/Transport.java| 32 +++-
.../sdk/extensions/gcp/util/TransportTest.java | 35 ++
3 files changed, 65 insertions(+), 13 deletions(-)
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
index edda1abc25f..cfc925a515e 100644
---
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
@@ -32,6 +32,7 @@ import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
+import java.util.Map;
import java.util.Set;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
@@ -208,6 +209,8 @@ public class RetryHttpRequestInitializer implements
HttpRequestInitializer {
private Set ignoredResponseCodes = new
HashSet<>(DEFAULT_IGNORED_RESPONSE_CODES);
+ private Map httpHeaders = null;
+
public RetryHttpRequestInitializer() {
this(Collections.emptyList());
}
@@ -270,6 +273,10 @@ public class RetryHttpRequestInitializer implements
HttpRequestInitializer {
request.setUnsuccessfulResponseHandler(loggingHttpBackOffHandler);
request.setIOExceptionHandler(loggingHttpBackOffHandler);
+if (this.httpHeaders != null) {
+ request.getHeaders().putAll(this.httpHeaders);
+}
+
// Set response initializer
if (responseInterceptor != null) {
request.setResponseInterceptor(responseInterceptor);
@@ -284,4 +291,8 @@ public class RetryHttpRequestInitializer implements
HttpRequestInitializer {
public void setWriteTimeout(int writeTimeout) {
this.writeTimeout = writeTimeout;
}
+
+ public void setHttpHeaders(Map httpHeaders) {
+this.httpHeaders = httpHeaders;
+ }
}
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
index a789b594f46..f7ecbfeda77 100644
---
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.extensions.gcp.util;
+import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings.isNullOrEmpty;
+
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
@@ -31,9 +33,12 @@ import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Paths;
import java.security.GeneralSecurityException;
+import java.util.Optional;
import org.apache.beam.sdk.extensions.gcp.auth.NullCredentialInitializer;
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
+import org.apache.beam.sdk.util.ReleaseInfo;
import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
/** Helpers for cloud communication. */
public class Transport {
@@ -89,20 +94,31 @@ public class Transport {
/** Returns a Cloud Storage client builder using the specified {@link
GcsOptions}. */
public static Storage.Builder newStorageClient(GcsOptions options) {
-String applicationNameSuffix = " (GPN:Beam)";
+String jobName =
Optional.ofNullable(options.getJobName()).orElse("UNKNOWN");
+
+String applicationName =
+String.format(
+"%sapache-beam/%s (GPN:Beam)",
+isNullOrEmpty(options.getAppName()) ? "" : options.getAppName() +
" ",
+ReleaseInfo.getReleaseInfo().getSdkVersion());
+
String servicePath = options.getGcsEndpoint();
+
+
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new dc506816214 Updating config from bot
dc506816214 is described below
commit dc506816214ad4f2229fae7d54ac92943384282c
Author: github-actions
AuthorDate: Fri May 17 15:13:01 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31331.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31331.json
b/scripts/ci/pr-bot/state/pr-state/pr-31331.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31331.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 7f2ebf786ad Updating config from bot
7f2ebf786ad is described below
commit 7f2ebf786adb2b058b07370e6f689251e26a2312
Author: github-actions
AuthorDate: Fri May 17 08:07:19 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31328.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31328.json
b/scripts/ci/pr-bot/state/pr-state/pr-31328.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31328.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
