(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8b12c2d3e8e Updating config from bot 8b12c2d3e8e is described below commit 8b12c2d3e8e00aa4a088ada34599916733ff0197 Author: github-actions AuthorDate: Thu Mar 21 05:34:32 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index 383c9816fc2..1b6e3439deb 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -1,7 +1,7 @@ { "label": "go", "dateOfLastReviewAssignment": { -"lostluck": 171082643, +"lostluck": 1710999270077, "jrmccluskey": 1710911140654, "youngoli": 1657688896155, "damccorm": 1680501930289,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 41a93809b0e Updating config from bot 41a93809b0e is described below commit 41a93809b0e572611eb63b61de5c1a05f75b5e3b Author: github-actions AuthorDate: Thu Mar 21 05:34:34 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30698.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30698.json b/scripts/ci/pr-bot/state/pr-state/pr-30698.json new file mode 100644 index 000..aa3638a74ec --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30698.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"go": "riteshghorse" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new cb0ca4db22a Updating config from bot cb0ca4db22a is described below commit cb0ca4db22ac9169dbda2deb97b06d6265232746 Author: github-actions AuthorDate: Thu Mar 21 05:34:36 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index 1b6e3439deb..a61e5ccd085 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -5,6 +5,6 @@ "jrmccluskey": 1710911140654, "youngoli": 1657688896155, "damccorm": 1680501930289, -"riteshghorse": 1710911136987 +"riteshghorse": 1710999274126 } } \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d89d6498145 Updating config from bot d89d6498145 is described below commit d89d64981457d13d1b07459de5116c358438bae6 Author: github-actions AuthorDate: Thu Mar 21 05:34:31 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30699.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30699.json b/scripts/ci/pr-bot/state/pr-state/pr-30699.json new file mode 100644 index 000..8ea94810368 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30699.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"go": "lostluck" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch dependabot/go_modules/sdks/google.golang.org/api-0.170.0 created (now 41ff5ac4a62)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/api-0.170.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 41ff5ac4a62 Bump google.golang.org/api from 0.166.0 to 0.170.0 in /sdks No new revisions were added by this update.
(beam) branch dependabot/go_modules/sdks/github.com/golang/protobuf-1.5.4 created (now 048c69a48ce)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/golang/protobuf-1.5.4 in repository https://gitbox.apache.org/repos/asf/beam.git at 048c69a48ce Bump github.com/golang/protobuf from 1.5.3 to 1.5.4 in /sdks No new revisions were added by this update.
(beam) branch nightly-refs/heads/master updated (389e1067c9d -> 8955124219c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 389e1067c9d Disable unsupported custom window type test on samza and spark. (#30680) add 81cb194f8d1 Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks (#30685) add 8206d8976d5 Bump github.com/docker/docker in /sdks (#30684) add a9ce7700ade Send email when BQ early rollout workflow fails (#30694) add f4d03d49713 Minimize scope of expensive lock (#30679) add 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" add cdaae0a7fb6 Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually" add 4a3b6c55fd8 Use unique GCS object path (#30690) add 8955124219c Add BigTableIO Stress test (#30630) No new revisions were added by this update. Summary of changes: .../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 21 +- it/google-cloud-platform/build.gradle | 5 +- .../org/apache/beam/it/gcp/IOLoadTestBase.java | 34 ++ .../org/apache/beam/it/gcp/IOStressTestBase.java | 123 +++ .../apache/beam/it/gcp/bigquery/BigQueryIOST.java | 105 +- .../apache/beam/it/gcp/bigtable/BigTableIOST.java | 389 + it/kafka/build.gradle | 4 +- .../java/org/apache/beam/it/kafka/KafkaIOST.java | 156 ++--- .../src/main/groovy/MobileGamingCommands.groovy| 5 +- .../main/groovy/mobilegaming-java-dataflow.groovy | 5 +- .../main/groovy/mobilegaming-java-direct.groovy| 5 +- .../google-cloud-dataflow-java/worker/build.gradle | 12 - .../dataflow/worker/StreamingDataflowWorker.java | 11 +- .../worker/windmill/WindmillConnection.java| 5 - .../windmill/client/grpc/GrpcDispatcherClient.java | 4 +- .../windmill/client/grpc/GrpcWindmillServer.java | 29 +- .../client/grpc/StreamingEngineClient.java | 25 +- .../windmill/client/grpc/stubs/ChannelCache.java | 117 --- .../grpc/stubs/ChannelCachingStubFactory.java | 38 -- .../client/grpc/stubs/IsolationChannel.java| 2 +- ...Factory.java => RemoteWindmillStubFactory.java} | 38 +- .../client/grpc/stubs/WindmillChannelFactory.java | 2 +- .../client/grpc/GrpcWindmillServerTest.java| 3 +- .../client/grpc/StreamingEngineClientTest.java | 31 +- .../client/grpc/WindmillStreamSenderTest.java | 3 +- .../client/grpc/stubs/ChannelCacheTest.java| 150 .../windmill/testing/FakeWindmillStubFactory.java | 27 +- .../budget/EvenGetWorkBudgetDistributorTest.java | 7 +- sdks/go.mod| 6 +- sdks/go.sum| 12 +- .../resources/beam/checkstyle/suppressions.xml | 1 + .../apache_beam/runners/worker/sdk_worker.py | 14 +- 32 files changed, 706 insertions(+), 683 deletions(-) create mode 100644 it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOStressTestBase.java create mode 100644 it/google-cloud-platform/src/test/java/org/apache/beam/it/gcp/bigtable/BigTableIOST.java delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCachingStubFactory.java rename runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/{ChannelCachingRemoteStubFactory.java => RemoteWindmillStubFactory.java} (67%) delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
(beam) branch master updated: Add BigTableIO Stress test (#30630)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8955124219c Add BigTableIO Stress test (#30630) 8955124219c is described below commit 8955124219cf9bd6dcea74f22c70fed7940da2b8 Author: akashorabek <70029317+akashora...@users.noreply.github.com> AuthorDate: Thu Mar 21 06:07:57 2024 +0600 Add BigTableIO Stress test (#30630) * Add BigTableIO Stress test * refactor * update dependency tree * refactor * Add stress test files to grpc/protobuff exception ignore list * move exportMetrics method to IOLoadTestBase class * refactor * refactor --- it/google-cloud-platform/build.gradle | 5 +- .../org/apache/beam/it/gcp/IOLoadTestBase.java | 34 ++ .../org/apache/beam/it/gcp/IOStressTestBase.java | 123 +++ .../apache/beam/it/gcp/bigquery/BigQueryIOST.java | 105 +- .../apache/beam/it/gcp/bigtable/BigTableIOST.java | 389 + it/kafka/build.gradle | 4 +- .../java/org/apache/beam/it/kafka/KafkaIOST.java | 156 ++--- .../resources/beam/checkstyle/suppressions.xml | 1 + 8 files changed, 578 insertions(+), 239 deletions(-) diff --git a/it/google-cloud-platform/build.gradle b/it/google-cloud-platform/build.gradle index 9717b5f8c84..3353a9692cb 100644 --- a/it/google-cloud-platform/build.gradle +++ b/it/google-cloud-platform/build.gradle @@ -32,6 +32,7 @@ dependencies { implementation project(path: ":runners:google-cloud-dataflow-java") implementation project(path: ":it:conditions", configuration: "shadow") implementation project(path: ":it:truthmatchers", configuration: "shadow") +implementation project(path: ":sdks:java:testing:test-utils") implementation library.java.slf4j_api implementation library.java.vendored_guava_32_1_2_jre implementation library.java.jackson_core @@ -48,6 +49,7 @@ dependencies { implementation library.java.protobuf_java implementation library.java.threetenbp implementation 'org.awaitility:awaitility:4.2.0' +implementation 'joda-time:joda-time:2.10.10' // Google Cloud Dependencies implementation library.java.google_api_services_bigquery implementation library.java.google_cloud_core @@ -71,7 +73,6 @@ dependencies { implementation 'com.google.cloud:google-cloud-secretmanager' provided 'com.google.api.grpc:proto-google-cloud-secretmanager-v1' -testImplementation project(path: ":sdks:java:testing:test-utils") testImplementation project(path: ":sdks:java:io:google-cloud-platform") testImplementation project(path: ":sdks:java:extensions:protobuf", configuration: "testRuntimeMigration") testImplementation project(path: ":sdks:java:io:synthetic") @@ -83,6 +84,8 @@ dependencies { tasks.register("GCSPerformanceTest", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'FileBasedIOLT', ['configuration':'large','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) tasks.register("BigTablePerformanceTest", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigTableIOLT', ['configuration':'large','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) +tasks.register("BigTableStressTestMedium", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigTableIOST', ['configuration':'medium','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) +tasks.register("BigTableStressTestLarge", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigTableIOST', ['configuration':'large','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) tasks.register("BigQueryPerformanceTest", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigQueryIOLT', ['configuration':'medium','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) tasks.register("BigQueryStressTestMedium", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigQueryIOST', ['configuration':'medium','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) tasks.register("BigQueryStressTestLarge", IoPerformanceTestUtilities.IoPerformanceTest, project, 'google-cloud-platform', 'BigQueryIOST', ['configuration':'large','project':'apache-beam-testing', 'artifactBucket':'io-performance-temp']) diff --git a/it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOLoadTestBase.java b/it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOLoadTestBase.java index 6b728a6a60d..e5f20c07c01 100644 --- a/it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/IOLoadTestBase.java +++
(beam) branch asf-site updated: Publishing website 2024/03/20 23:39:10 at commit cdaae0a
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 85f275c7450 Publishing website 2024/03/20 23:39:10 at commit cdaae0a 85f275c7450 is described below commit 85f275c7450982c064e2071ba6ca05aa94a29823 Author: runner AuthorDate: Wed Mar 20 23:39:10 2024 + Publishing website 2024/03/20 23:39:10 at commit cdaae0a --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 49333a57b1e..f382cc53eb9 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.54.0/2024-03-20T09:57:22-07:00/categories/blog/2024-03-20T09:57:22-07:00/blog/2024-03-20T09:57:22-07:00/categories/2024-03-20T09:57:22-07:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.54.0/2024-03-20T18:28:21-04:00/categories/blog/2024-03-20T18:28:21-04:00/blog/2024-03-20T18:28:21-04:00/categories/2024-03-20T18:28:21-04:00/catego [...] \ No newline at end of file
(beam) branch master updated: Use unique GCS object path (#30690)
This is an automated email from the ASF dual-hosted git repository. damondouglas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 4a3b6c55fd8 Use unique GCS object path (#30690) 4a3b6c55fd8 is described below commit 4a3b6c55fd8a02911b99d7ef8b5fa6f006c23fb2 Author: Damon AuthorDate: Wed Mar 20 16:35:59 2024 -0700 Use unique GCS object path (#30690) --- release/src/main/groovy/MobileGamingCommands.groovy | 5 +++-- release/src/main/groovy/mobilegaming-java-dataflow.groovy | 5 - release/src/main/groovy/mobilegaming-java-direct.groovy | 5 - 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/release/src/main/groovy/MobileGamingCommands.groovy b/release/src/main/groovy/MobileGamingCommands.groovy index 2b38943067a..39f526d7da8 100644 --- a/release/src/main/groovy/MobileGamingCommands.groovy +++ b/release/src/main/groovy/MobileGamingCommands.groovy @@ -21,6 +21,7 @@ class MobileGamingCommands { private TestScripts testScripts + private String testRunId private static final INPUT_GAMING_DATA = "gs://dataflow-samples/game/5000_gaming_data.csv" @@ -58,11 +59,11 @@ class MobileGamingCommands { "BattleshipGrey")) public String getUserScoreOutputName(String runner){ -return "java-userscore-result-${RUNNERS[runner]}.txt" +return "java-userscore-result-${RUNNERS[runner]}/${testRunId}/output" } public String getHourlyTeamScoreOutputName(String runner){ -return "java-hourlyteamscore-result-${RUNNERS[runner]}.txt" +return "java-hourlyteamscore-result-${RUNNERS[runner]}/${testRunId}/output" } public String createPipelineCommand(String exampleName, String runner, String jobName='', String className=null){ diff --git a/release/src/main/groovy/mobilegaming-java-dataflow.groovy b/release/src/main/groovy/mobilegaming-java-dataflow.groovy index 462b3d2cea0..bb0b76bd675 100644 --- a/release/src/main/groovy/mobilegaming-java-dataflow.groovy +++ b/release/src/main/groovy/mobilegaming-java-dataflow.groovy @@ -18,7 +18,6 @@ */ t = new TestScripts(args) -mobileGamingCommands = new MobileGamingCommands(testScripts: t) /* * Run the mobile game examples on Dataflow. @@ -36,6 +35,8 @@ String command_output_text * Run the UserScore example on DataflowRunner * */ +mobileGamingCommands = new MobileGamingCommands(testScripts: t, testRunId: UUID.randomUUID().toString()) + t.intent("Running: UserScore example on DataflowRunner") t.run(mobileGamingCommands.createPipelineCommand("UserScore", runner)) command_output_text = t.run "gsutil cat gs://${t.gcsBucket()}/${mobileGamingCommands.getUserScoreOutputName(runner)}* | grep user19_BananaWallaby" @@ -48,6 +49,8 @@ t.run "gsutil rm gs://${t.gcsBucket()}/${mobileGamingCommands.getUserScoreOutput * Run the HourlyTeamScore example on DataflowRunner * */ +mobileGamingCommands = new MobileGamingCommands(testScripts: t, testRunId: UUID.randomUUID().toString()) + t.intent("Running: HourlyTeamScore example on DataflowRunner") t.run(mobileGamingCommands.createPipelineCommand("HourlyTeamScore", runner)) command_output_text = t.run "gsutil cat gs://${t.gcsBucket()}/${mobileGamingCommands.getHourlyTeamScoreOutputName(runner)}* | grep AzureBilby " diff --git a/release/src/main/groovy/mobilegaming-java-direct.groovy b/release/src/main/groovy/mobilegaming-java-direct.groovy index 9ff24cefaf9..3c6f4ca01a6 100644 --- a/release/src/main/groovy/mobilegaming-java-direct.groovy +++ b/release/src/main/groovy/mobilegaming-java-direct.groovy @@ -18,7 +18,6 @@ */ t = new TestScripts(args) -mobileGamingCommands = new MobileGamingCommands(testScripts: t) /* * Run the mobile game examples on DirectRunner. @@ -36,6 +35,8 @@ String command_output_text * Run the UserScore example with DirectRunner * */ +mobileGamingCommands = new MobileGamingCommands(testScripts: t, testRunId: UUID.randomUUID().toString()) + t.intent("Running: UserScore example on DirectRunner") t.run(mobileGamingCommands.createPipelineCommand("UserScore", runner)) command_output_text = t.run "grep user19_BananaWallaby ${mobileGamingCommands.getUserScoreOutputName(runner)}* " @@ -47,6 +48,8 @@ t.success("UserScore successfully run on DirectRunners.") * Run the HourlyTeamScore example with DirectRunner * */ +mobileGamingCommands = new MobileGamingCommands(testScripts: t, testRunId: UUID.randomUUID().toString()) + t.intent("Running: HourlyTeamScore example on DirectRunner") t.run(mobileGamingCommands.createPipelineCommand("HourlyTeamScore", runner)) command_output_text = t.run "grep AzureBilby ${mobileGamingCommands.getHourlyTeamScoreOutputName(runner)}* "
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8a080aa5038 Updating config from bot 8a080aa5038 is described below commit 8a080aa50381cf5ff2bd5981c754419e57138031 Author: github-actions AuthorDate: Wed Mar 20 23:34:41 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index cb711f1f431..bfbec6f766c 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -10,7 +10,7 @@ "y1chi": 1667002607045, "damccorm": 1709863822385, "jrmccluskey": 1710776076591, -"riteshghorse": 1709764491759, +"riteshghorse": 1710977678377, "liferoad": 1710819285271, "shunping": 1710265960889 }
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 100be857096 Updating config from bot 100be857096 is described below commit 100be857096cf5f96677112a79071d36d134498d Author: github-actions AuthorDate: Wed Mar 20 23:34:39 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30696.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30696.json b/scripts/ci/pr-bot/state/pr-state/pr-30696.json new file mode 100644 index 000..c43704aa814 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30696.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"python": "riteshghorse" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new e5176e8e631 Updating config from bot e5176e8e631 is described below commit e5176e8e6313cd998f3097815bd3cd121ddd64bd Author: github-actions AuthorDate: Wed Mar 20 23:06:53 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30690.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30690.json b/scripts/ci/pr-bot/state/pr-state/pr-30690.json new file mode 100644 index 000..9c51466b9ee --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30690.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"build": "Abacn" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new f27cbbfacea Updating config from bot f27cbbfacea is described below commit f27cbbfaceaa504fae40de70288f6f526f0a388c Author: github-actions AuthorDate: Wed Mar 20 23:06:54 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json b/scripts/ci/pr-bot/state/reviewers-for-label-build.json index d5eeca57d2f..a9d18e2bc9c 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json @@ -2,6 +2,6 @@ "label": "build", "dateOfLastReviewAssignment": { "damccorm": 1710819285271, -"Abacn": 1710630350352 +"Abacn": 1710976011952 } } \ No newline at end of file
(beam) branch revert-30425-mt-shutdown-channels deleted (was 8681ab9901d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch revert-30425-mt-shutdown-channels in repository https://gitbox.apache.org/repos/asf/beam.git was 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) 01/01: Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually"
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit cdaae0a7fb63bd679823488f901603e135d6db67 Merge: f4d03d49713 8681ab9901d Author: Kenneth Knowles AuthorDate: Wed Mar 20 18:28:21 2024 -0400 Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually" .../google-cloud-dataflow-java/worker/build.gradle | 12 -- .../dataflow/worker/StreamingDataflowWorker.java | 11 +- .../worker/windmill/WindmillConnection.java| 5 - .../windmill/client/grpc/GrpcDispatcherClient.java | 4 +- .../windmill/client/grpc/GrpcWindmillServer.java | 29 +--- .../client/grpc/StreamingEngineClient.java | 25 ++-- .../windmill/client/grpc/stubs/ChannelCache.java | 117 .../grpc/stubs/ChannelCachingStubFactory.java | 38 -- .../client/grpc/stubs/IsolationChannel.java| 2 +- ...Factory.java => RemoteWindmillStubFactory.java} | 38 +++--- .../client/grpc/stubs/WindmillChannelFactory.java | 2 +- .../client/grpc/GrpcWindmillServerTest.java| 3 +- .../client/grpc/StreamingEngineClientTest.java | 31 +++-- .../client/grpc/WindmillStreamSenderTest.java | 3 +- .../client/grpc/stubs/ChannelCacheTest.java| 150 - .../windmill/testing/FakeWindmillStubFactory.java | 27 ++-- .../budget/EvenGetWorkBudgetDistributorTest.java | 7 +- 17 files changed, 78 insertions(+), 426 deletions(-)
(beam) branch master updated (f4d03d49713 -> cdaae0a7fb6)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f4d03d49713 Minimize scope of expensive lock (#30679) add 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" new cdaae0a7fb6 Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../google-cloud-dataflow-java/worker/build.gradle | 12 -- .../dataflow/worker/StreamingDataflowWorker.java | 11 +- .../worker/windmill/WindmillConnection.java| 5 - .../windmill/client/grpc/GrpcDispatcherClient.java | 4 +- .../windmill/client/grpc/GrpcWindmillServer.java | 29 +--- .../client/grpc/StreamingEngineClient.java | 25 ++-- .../windmill/client/grpc/stubs/ChannelCache.java | 117 .../grpc/stubs/ChannelCachingStubFactory.java | 38 -- .../client/grpc/stubs/IsolationChannel.java| 2 +- ...Factory.java => RemoteWindmillStubFactory.java} | 38 +++--- .../client/grpc/stubs/WindmillChannelFactory.java | 2 +- .../client/grpc/GrpcWindmillServerTest.java| 3 +- .../client/grpc/StreamingEngineClientTest.java | 31 +++-- .../client/grpc/WindmillStreamSenderTest.java | 3 +- .../client/grpc/stubs/ChannelCacheTest.java| 150 - .../windmill/testing/FakeWindmillStubFactory.java | 27 ++-- .../budget/EvenGetWorkBudgetDistributorTest.java | 7 +- 17 files changed, 78 insertions(+), 426 deletions(-) delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCachingStubFactory.java rename runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/{ChannelCachingRemoteStubFactory.java => RemoteWindmillStubFactory.java} (67%) delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 5c5a033bc24 Updating config from bot 5c5a033bc24 is described below commit 5c5a033bc2420f30bcff63ca831681a4950855a4 Author: github-actions AuthorDate: Wed Mar 20 22:05:42 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30695.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30695.json b/scripts/ci/pr-bot/state/pr-state/pr-30695.json new file mode 100644 index 000..242a48d7d3b --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30695.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": true, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch users/damccorm/critSection deleted (was afe5bc1ebd7)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git was afe5bc1ebd7 fmt The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Minimize scope of expensive lock (#30679)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new f4d03d49713 Minimize scope of expensive lock (#30679) f4d03d49713 is described below commit f4d03d49713cf89260c141ee35b4dadb31ad4193 Author: Danny McCormick AuthorDate: Wed Mar 20 16:42:38 2024 -0400 Minimize scope of expensive lock (#30679) * Minimize scope of expensive lock * Build list in lock * Update sdks/python/apache_beam/runners/worker/sdk_worker.py Co-authored-by: tvalentyn * Move comment * Simplify * fmt - Co-authored-by: tvalentyn --- sdks/python/apache_beam/runners/worker/sdk_worker.py | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py b/sdks/python/apache_beam/runners/worker/sdk_worker.py index 04b58a6f198..65059ab054f 100644 --- a/sdks/python/apache_beam/runners/worker/sdk_worker.py +++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py @@ -607,12 +607,18 @@ class BundleProcessorCache(object): # type: () -> None def shutdown_inactive_bundle_processors(): # type: () -> None + inactive_descriptor_ids = [] + inactive_time = time.time( + ) - DEFAULT_BUNDLE_PROCESSOR_CACHE_SHUTDOWN_THRESHOLD_S with self._lock: for descriptor_id, last_access_time in self.last_access_times.items(): - if (time.time() - last_access_time > - DEFAULT_BUNDLE_PROCESSOR_CACHE_SHUTDOWN_THRESHOLD_S): -BundleProcessorCache._shutdown_cached_bundle_processors( -self.cached_bundle_processors[descriptor_id]) + if (inactive_time > last_access_time): +inactive_descriptor_ids.append(descriptor_id) + + # Shutdown can be expensive, keep out of lock + for descriptor_id in inactive_descriptor_ids: +BundleProcessorCache._shutdown_cached_bundle_processors( +self.cached_bundle_processors[descriptor_id]) self.periodic_shutdown = PeriodicThread( DEFAULT_BUNDLE_PROCESSOR_CACHE_SHUTDOWN_THRESHOLD_S,
(beam) branch master updated: Send email when BQ early rollout workflow fails (#30694)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a9ce7700ade Send email when BQ early rollout workflow fails (#30694) a9ce7700ade is described below commit a9ce7700adef058cf93a84906d725d4ecd646aad Author: Danny McCormick AuthorDate: Wed Mar 20 16:14:06 2024 -0400 Send email when BQ early rollout workflow fails (#30694) --- .../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml b/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml index f832c5fab56..d0b58dd77c8 100644 --- a/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml +++ b/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml @@ -85,6 +85,25 @@ jobs: uses: ./.github/actions/gradle-command-self-hosted-action with: gradle-command: :sdks:java:io:google-cloud-platform:bigQueryEarlyRolloutIntegrationTest + - name: Generate Date +if: failure() +run: | + date=$(date -u +"%Y-%m-%d") + echo "date=$date" >> $GITHUB_ENV + - name: Send email +uses: dawidd6/action-send-mail@v3 +if: failure() +with: + server_address: smtp.gmail.com + server_port: 465 + secure: true + username: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_ADDRESS }} + password: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_PASSWORD }} + subject: BigQueryEarlyRollout Beam Failure (${{ env.date }}) - investigate and escalate quickly + to: datapls-plat-t...@google.com # Team at Google responsible for escalating BQ failures + from: gacti...@beam.apache.org + body: | +PostCommit Java BigQueryEarlyRollout failed on ${{ env.date }}. This test monitors BigQuery rollouts impacting Beam and should be escalated immediately if a real issue is encountered to pause further rollouts. For further details refer to the following links:\n * Failing job: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml \n * Job configuration: https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_Bi [...] - name: Archive JUnit Test Results uses: actions/upload-artifact@v4 if: ${{ !success() }} @@ -97,4 +116,4 @@ jobs: with: commit: '${{ env.prsha || env.GITHUB_SHA }}' comment_mode: ${{ github.event_name == 'issue_comment' && 'always' || 'off' }} - files: '**/build/test-results/**/*.xml' \ No newline at end of file + files: '**/build/test-results/**/*.xml'
(beam) branch users/damccorm/bq-escalation deleted (was 4c4039b73d3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq-escalation in repository https://gitbox.apache.org/repos/asf/beam.git was 4c4039b73d3 Send email when BQ early rollout workflow fails The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 43c057a3b25 Updating config from bot 43c057a3b25 is described below commit 43c057a3b255d917c334003c1e3e8d6f7d1f6971 Author: github-actions AuthorDate: Wed Mar 20 20:06:30 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30693.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30693.json b/scripts/ci/pr-bot/state/pr-state/pr-30693.json new file mode 100644 index 000..242a48d7d3b --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30693.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": true, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a12de0dd2a7 Updating config from bot a12de0dd2a7 is described below commit a12de0dd2a76a4bcbae458820937d79c93e475cc Author: github-actions AuthorDate: Wed Mar 20 19:31:35 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30694.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30694.json b/scripts/ci/pr-bot/state/pr-state/pr-30694.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30694.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch users/damccorm/bq-escalation created (now 4c4039b73d3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq-escalation in repository https://gitbox.apache.org/repos/asf/beam.git at 4c4039b73d3 Send email when BQ early rollout workflow fails This branch includes the following new commits: new 4c4039b73d3 Send email when BQ early rollout workflow fails The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Send email when BQ early rollout workflow fails
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/bq-escalation in repository https://gitbox.apache.org/repos/asf/beam.git commit 4c4039b73d37a9946a69624e260767d21843f364 Author: Danny McCormick AuthorDate: Wed Mar 20 15:28:53 2024 -0400 Send email when BQ early rollout workflow fails --- .../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml b/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml index f832c5fab56..d0b58dd77c8 100644 --- a/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml +++ b/.github/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml @@ -85,6 +85,25 @@ jobs: uses: ./.github/actions/gradle-command-self-hosted-action with: gradle-command: :sdks:java:io:google-cloud-platform:bigQueryEarlyRolloutIntegrationTest + - name: Generate Date +if: failure() +run: | + date=$(date -u +"%Y-%m-%d") + echo "date=$date" >> $GITHUB_ENV + - name: Send email +uses: dawidd6/action-send-mail@v3 +if: failure() +with: + server_address: smtp.gmail.com + server_port: 465 + secure: true + username: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_ADDRESS }} + password: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_PASSWORD }} + subject: BigQueryEarlyRollout Beam Failure (${{ env.date }}) - investigate and escalate quickly + to: datapls-plat-t...@google.com # Team at Google responsible for escalating BQ failures + from: gacti...@beam.apache.org + body: | +PostCommit Java BigQueryEarlyRollout failed on ${{ env.date }}. This test monitors BigQuery rollouts impacting Beam and should be escalated immediately if a real issue is encountered to pause further rollouts. For further details refer to the following links:\n * Failing job: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml \n * Job configuration: https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_Bi [...] - name: Archive JUnit Test Results uses: actions/upload-artifact@v4 if: ${{ !success() }} @@ -97,4 +116,4 @@ jobs: with: commit: '${{ env.prsha || env.GITHUB_SHA }}' comment_mode: ${{ github.event_name == 'issue_comment' && 'always' || 'off' }} - files: '**/build/test-results/**/*.xml' \ No newline at end of file + files: '**/build/test-results/**/*.xml'
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 3452ab757e4 Updating config from bot 3452ab757e4 is described below commit 3452ab757e4ccf9461ec17c9319c8315c137e427 Author: github-actions AuthorDate: Wed Mar 20 18:31:23 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30691.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30691.json b/scripts/ci/pr-bot/state/pr-state/pr-30691.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30691.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch revert-30425-mt-shutdown-channels created (now 8681ab9901d)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch revert-30425-mt-shutdown-channels in repository https://gitbox.apache.org/repos/asf/beam.git at 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" This branch includes the following new commits: new 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Revert "Cache and close windmill grpc channels (#30425)"
This is an automated email from the ASF dual-hosted git repository. yhu pushed a commit to branch revert-30425-mt-shutdown-channels in repository https://gitbox.apache.org/repos/asf/beam.git commit 8681ab9901d38af3c826eb75bbf0816cc55b0627 Author: Yi Hu AuthorDate: Wed Mar 20 14:27:29 2024 -0400 Revert "Cache and close windmill grpc channels (#30425)" This reverts commit c1c255a0a433d7cdf1b5f0bc61986d395c7703ad. --- .../google-cloud-dataflow-java/worker/build.gradle | 12 -- .../dataflow/worker/StreamingDataflowWorker.java | 11 +- .../worker/windmill/WindmillConnection.java| 5 - .../windmill/client/grpc/GrpcDispatcherClient.java | 4 +- .../windmill/client/grpc/GrpcWindmillServer.java | 29 +--- .../client/grpc/StreamingEngineClient.java | 25 ++-- .../windmill/client/grpc/stubs/ChannelCache.java | 117 .../grpc/stubs/ChannelCachingStubFactory.java | 38 -- .../client/grpc/stubs/IsolationChannel.java| 2 +- ...Factory.java => RemoteWindmillStubFactory.java} | 38 +++--- .../client/grpc/stubs/WindmillChannelFactory.java | 2 +- .../client/grpc/GrpcWindmillServerTest.java| 3 +- .../client/grpc/StreamingEngineClientTest.java | 31 +++-- .../client/grpc/WindmillStreamSenderTest.java | 3 +- .../client/grpc/stubs/ChannelCacheTest.java| 150 - .../windmill/testing/FakeWindmillStubFactory.java | 27 ++-- .../budget/EvenGetWorkBudgetDistributorTest.java | 7 +- 17 files changed, 78 insertions(+), 426 deletions(-) diff --git a/runners/google-cloud-dataflow-java/worker/build.gradle b/runners/google-cloud-dataflow-java/worker/build.gradle index 4ddb3e2755e..124c11026df 100644 --- a/runners/google-cloud-dataflow-java/worker/build.gradle +++ b/runners/google-cloud-dataflow-java/worker/build.gradle @@ -71,10 +71,6 @@ def excluded_dependencies = [ library.java.truth // Test only ] -// For Java8+ and less than Java11, use versions 2.x.x. -// Java11+ can use versions 3.x.x per https://github.com/ben-manes/caffeine. -def caffeine_cache_version = "2.9.3" - applyJavaNature( automaticModuleName: 'org.apache.beam.runners.dataflow.worker', archivesBaseName: 'beam-runners-google-cloud-dataflow-java-legacy-worker', @@ -141,13 +137,6 @@ applyJavaNature( relocate("org.eclipse.jetty", getWorkerRelocatedPath("org.eclipse.jetty")) relocate("javax.servlet", getWorkerRelocatedPath("javax.servlet")) -// Use Caffeine cache instead of Guava cache. -// Context: https://guava.dev/releases/snapshot/api/docs/com/google/common/cache/CacheBuilder -dependencies { - include(dependency("com.github.ben-manes.caffeine:caffeine:${caffeine_cache_version}}")) -} -relocate("com.github.ben-manes.caffeine", getWorkerRelocatedPath("com.github.ben-manes.caffeine")) - // We don't relocate windmill since it is already underneath the org.apache.beam.runners.dataflow.worker namespace and never // expect a user pipeline to include it. There is also a JNI component that windmill server relies on which makes // arbitrary relocation more difficult. @@ -213,7 +202,6 @@ dependencies { implementation "javax.servlet:javax.servlet-api:3.1.0" implementation "org.eclipse.jetty:jetty-server:9.2.10.v20150310" implementation "org.eclipse.jetty:jetty-servlet:9.2.10.v20150310" -implementation "com.github.ben-manes.caffeine:caffeine:${caffeine_cache_version}" implementation library.java.avro implementation library.java.jackson_annotations implementation library.java.jackson_core diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java index cc5b890bc60..4c3ffd08a0b 100644 --- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java +++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java @@ -457,13 +457,12 @@ public class StreamingDataflowWorker { public static StreamingDataflowWorker fromOptions(DataflowWorkerHarnessOptions options) { ConcurrentMap computationMap = new ConcurrentHashMap<>(); long clientId = clientIdGenerator.nextLong(); - -Consumer> workHeartbeatResponseProcessor = -new WorkHeartbeatResponseProcessor( -computationId -> Optional.ofNullable(computationMap.get(computationId))); - return new StreamingDataflowWorker( -createWindmillServerStub(options, clientId, workHeartbeatResponseProcessor), +createWindmillServerStub( +options, +clientId, +
(beam) branch users/damccorm/critSection updated (d927d984843 -> afe5bc1ebd7)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git from d927d984843 Simplify add afe5bc1ebd7 fmt No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam-starter-go) branch dependabot/go_modules/github.com/docker/docker-24.0.9incompatible deleted (was c4ae0c9)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/github.com/docker/docker-24.0.9incompatible in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git was c4ae0c9 Bump github.com/docker/docker The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam-starter-go) branch main updated: Bump github.com/docker/docker (#32)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git The following commit(s) were added to refs/heads/main by this push: new 3d7e55c Bump github.com/docker/docker (#32) 3d7e55c is described below commit 3d7e55cee98574095b73eacf098bd55898305036 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Mar 20 13:25:34 2024 -0400 Bump github.com/docker/docker (#32) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.7+incompatible to 24.0.9+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](https://github.com/docker/docker/compare/v24.0.7...v24.0.9) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 2 +- go.sum | 6 -- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 7398ea0..68399c2 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( cloud.google.com/go/storage v1.36.0 // indirect github.com/Microsoft/go-winio v0.6.1 // indirect github.com/docker/distribution v2.8.2+incompatible // indirect - github.com/docker/docker v24.0.7+incompatible // indirect + github.com/docker/docker v24.0.9+incompatible // indirect github.com/docker/go-connections v0.4.0 // indirect github.com/docker/go-units v0.5.0 // indirect github.com/dustin/go-humanize v1.0.1 // indirect diff --git a/go.sum b/go.sum index 60f5980..21be0f1 100644 --- a/go.sum +++ b/go.sum @@ -30,8 +30,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/docker/distribution v2.8.2+incompatible h1:T3de5rq0dB1j30rp0sA2rER+m322EBzniBPB6ZIzuh8= github.com/docker/distribution v2.8.2+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= -github.com/docker/docker v24.0.7+incompatible h1:Wo6l37AuwP3JaMnZa226lzVXGA3F9Ig1seQen0cKYlM= -github.com/docker/docker v24.0.7+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v24.0.9+incompatible h1:HPGzNmwfLZWdxHqK9/II92pyi1EpYKsAqcl4G0Of9v0= +github.com/docker/docker v24.0.9+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ= github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= @@ -131,6 +131,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/rogpeppe/clock v0.0.0-20190514195947-2896927a307a h1:3QH7VyOaaiUHNrA9Se4YQIRkDTCw1EJls9xTUCaCeRM= github.com/rogpeppe/clock v0.0.0-20190514195947-2896927a307a/go.mod h1:4r5QyqhjIWCcK8DO4KMclc5Iknq5qVBAlbYYzAbUScQ= +github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= +github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
(beam-starter-go) branch dependabot/go_modules/github.com/docker/docker-24.0.9incompatible created (now c4ae0c9)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/github.com/docker/docker-24.0.9incompatible in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git at c4ae0c9 Bump github.com/docker/docker No new revisions were added by this update.
(beam) branch dependabot/go_modules/sdks/github.com/docker/docker-25.0.5incompatible deleted (was 38a0341ccf7)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/github.com/docker/docker-25.0.5incompatible in repository https://gitbox.apache.org/repos/asf/beam.git was 38a0341ccf7 Bump github.com/docker/docker in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump github.com/docker/docker in /sdks (#30684)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8206d8976d5 Bump github.com/docker/docker in /sdks (#30684) 8206d8976d5 is described below commit 8206d8976d524853613a9ab350a7061088eda50b Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Mar 20 09:57:22 2024 -0700 Bump github.com/docker/docker in /sdks (#30684) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 25.0.3+incompatible to 25.0.5+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](https://github.com/docker/docker/compare/v25.0.3...v25.0.5) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index c5887395a14..26e9bba48cd 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -143,7 +143,7 @@ require ( github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa // indirect github.com/containerd/containerd v1.7.11 // indirect github.com/cpuguy83/dockercfg v0.3.1 // indirect - github.com/docker/docker v25.0.3+incompatible // but required to resolve issue docker has with go1.20 + github.com/docker/docker v25.0.5+incompatible // but required to resolve issue docker has with go1.20 github.com/docker/go-units v0.5.0 // indirect github.com/envoyproxy/go-control-plane v0.12.0 // indirect github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index eaa713f7d06..d59d0ba2173 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -160,8 +160,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/distribution/reference v0.5.0 h1:/FUIFXtfc/x2gpa5/VGfiGLuOIdYa1t65IKK2OFGvA0= github.com/distribution/reference v0.5.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= -github.com/docker/docker v25.0.3+incompatible h1:D5fy/lYmY7bvZa0XTZ5/UJPljor41F+vdyJG5luQLfQ= -github.com/docker/docker v25.0.3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v25.0.5+incompatible h1:UmQydMduGkrD5nQde1mecF/YnSbTOaPeFIeP5C4W+DE= +github.com/docker/docker v25.0.5+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c= github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
(beam) branch users/damccorm/critSection updated (b5a45c1c632 -> d927d984843)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git from b5a45c1c632 Move comment add d927d984843 Simplify No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
(beam) branch users/damccorm/critSection updated (cccaeee8919 -> b5a45c1c632)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git from cccaeee8919 Update sdks/python/apache_beam/runners/worker/sdk_worker.py add b5a45c1c632 Move comment No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch users/damccorm/critSection updated (434a3334c3b -> cccaeee8919)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git from 434a3334c3b Build list in lock add cccaeee8919 Update sdks/python/apache_beam/runners/worker/sdk_worker.py No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch dependabot/go_modules/sdks/golang.org/x/oauth2-0.18.0 deleted (was d9ff2eadf20)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/golang.org/x/oauth2-0.18.0 in repository https://gitbox.apache.org/repos/asf/beam.git was d9ff2eadf20 Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks (#30685)
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 81cb194f8d1 Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks (#30685) 81cb194f8d1 is described below commit 81cb194f8d12e5aa1d39ec9a10c1f1404c4073f7 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Mar 20 10:07:48 2024 -0400 Bump golang.org/x/oauth2 from 0.17.0 to 0.18.0 in /sdks (#30685) Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.17.0 to 0.18.0. - [Commits](https://github.com/golang/oauth2/compare/v0.17.0...v0.18.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index a44a90adba7..c5887395a14 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -54,8 +54,8 @@ require ( github.com/xitongsys/parquet-go v1.6.2 github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.13.1 - golang.org/x/net v0.21.0 - golang.org/x/oauth2 v0.17.0 + golang.org/x/net v0.22.0 + golang.org/x/oauth2 v0.18.0 golang.org/x/sync v0.6.0 golang.org/x/sys v0.18.0 golang.org/x/text v0.14.0 diff --git a/sdks/go.sum b/sdks/go.sum index 72b1037f254..eaa713f7d06 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -554,15 +554,15 @@ golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-2022202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4= -golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= +golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc= +golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.17.0 h1:6m3ZPmLEFdVxKKWnKq4VqZ60gutO35zm+zrAHVmHyDQ= -golang.org/x/oauth2 v0.17.0/go.mod h1:OzPDGQiuQMguemayvdylqddI7qcD9lnSDb+1FiwQ5HA= +golang.org/x/oauth2 v0.18.0 h1:09qnuIAgzdx1XplqJvW6CQqMCtGZykZWcXzPMPUusvI= +golang.org/x/oauth2 v0.18.0/go.mod h1:Wf7knwG0MPoWIMMBgFlEaSUDaKskp0dCfrlJRJXbBi8= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new f635b639712 Updating config from bot f635b639712 is described below commit f635b6397126b7bcc2a0178d2e36dce76ecacceb Author: github-actions AuthorDate: Wed Mar 20 14:07:26 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30689.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30689.json b/scripts/ci/pr-bot/state/pr-state/pr-30689.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-30689.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
(beam) branch users/damccorm/critSection updated (eda0b519d49 -> 434a3334c3b)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/critSection in repository https://gitbox.apache.org/repos/asf/beam.git from eda0b519d49 Minimize scope of expensive lock add 434a3334c3b Build list in lock No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 18 ++ 1 file changed, 10 insertions(+), 8 deletions(-)