(beam) branch master updated: [#32601][prism] Initial Deep Dive Documentation (#32143)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 001ac59aa58 [#32601][prism] Initial Deep Dive Documentation (#32143) 001ac59aa58 is described below commit 001ac59aa58e7becf7412a429b11c4a0214a3e42 Author: Robert Burke AuthorDate: Thu Oct 3 10:41:44 2024 -0700 [#32601][prism] Initial Deep Dive Documentation (#32143) Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com> --- sdks/go/pkg/beam/runners/prism/internal/README.md | 492 +- 1 file changed, 484 insertions(+), 8 deletions(-) diff --git a/sdks/go/pkg/beam/runners/prism/internal/README.md b/sdks/go/pkg/beam/runners/prism/internal/README.md index d19f2ca5fa6..684d0a80c51 100644 --- a/sdks/go/pkg/beam/runners/prism/internal/README.md +++ b/sdks/go/pkg/beam/runners/prism/internal/README.md @@ -17,6 +17,8 @@ under the License. --> +The following refers to the current state of Prism, and is subject to change. + # Prism internal packages Go has a mechanism for ["internal" packages](https://go.dev/doc/go1.4#internalpackages) @@ -26,7 +28,7 @@ This mechanism is used thoroughly for Prism to ensure we can make changes to the runner's internals without worrying about the exposed surface changes breaking non-compliant users. -# Structure +## Structure Here's a loose description of the current structure of the runner. Leaf packages should not depend on other parts of the runner. Runner packages can and do depend on other @@ -37,8 +39,9 @@ Handler configurations are registered by dependant packages. `urns` contains beam URN strings pulled from the protos. Leaf package. -`engine` contains the core manager for handling elements, watermarks, and windowing strategies. -Determines bundle readiness, and stages to execute. Leaf package. +`engine` contains the core manager for handling elements, watermarks, windowing strategies, +state, and timers. Determines bundle readiness, and stages to execute. Leaf package. +Should never depend on the protocol buffers. `jobservices` contains GRPC service handlers for job management and submission. Should only depend on the `config` and `urns` packages. @@ -46,9 +49,10 @@ Should only depend on the `config` and `urns` packages. `worker` contains interactions with FnAPI services to communicate with worker SDKs. Leaf package except for dependency on `engine.TentativeData` which will likely be removed at some point. -`internal` AKA the package in this directory root. Contains fhe job execution -flow. Jobs are sent to it from `jobservices`, and those jobs are then executed by coordinating -with the `engine` and `worker` packages, and handlers urn. +`internal` AKA the package in this directory root. Contains the job execution +flow. Jobs are sent to it from `jobservices`, through the Job Executor function, +and those jobs are then executed by coordinating with the `engine` and `worker` packages. +Handles the pipeline protos directly. Most configurable behavior is determined here. `web` contains a web server to visualize aspects of the runner, and should never be @@ -56,12 +60,484 @@ depended on by the other packages for the runner. It will primarily abstract access to pipeline data through a JobManagement API client. If no other option exists, it may use direct access to exported data from the other packages. -# Testing +## Testing The sub packages should have reasonable Unit Test coverage in their own directories, but -most features will be exercised via executing pipelines in this package. +most features will be exercised via executing pipelines in the internal and engine +packages. To avoid overly re-implementing tests, use of the "_test" formalism to validate +coverage through re-using tests from sdks/go/tests is recommended. For the time being test DoFns should be added to standard build in order to validate execution coverage, in particular for Combine and Splittable DoFns. Eventually these behaviors should be covered by using Prism in the main SDK tests. + +# Deep Dive + +These sections cover how Prism operates just above the code level, and addresses the +questions of "how it works". + +## Job Execution + +```mermaid +sequenceDiagram +autonumber +participant X as Job Executor +Note over X: Pre-process pipelinegraph into stages + +create participant S as Stages +X->>S: create +create participant W as SDK Worker +X->>W: create +create participant EM as ElementManager +X->>EM: Configure Stages in ElementManager +activate EM +X->>+EM: Request Bundles +loop For each bundle from the ElementManager +EM-
(beam) branch master updated (e91882f010c -> 2829c140c63)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e91882f010c Fix pull_licenses_java script retry broken for tenacity 8.5 (#32626) add 2829c140c63 Update building steps to use go 1.23.2 (#32629) No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- sdks/go/run_with_go_version.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(beam) branch master updated: Bump google.golang.org/api from 0.197.0 to 0.199.0 in /sdks (#32605)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8bdea3df66e Bump google.golang.org/api from 0.197.0 to 0.199.0 in /sdks (#32605) 8bdea3df66e is described below commit 8bdea3df66eda7bad84de321b96acd8ad48f4045 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Oct 2 07:43:56 2024 -0700 Bump google.golang.org/api from 0.197.0 to 0.199.0 in /sdks (#32605) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.197.0 to 0.199.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.197.0...v0.199.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 9 - sdks/go.sum | 16 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index a414cd5d8fd..4210c4927cb 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -58,9 +58,9 @@ require ( golang.org/x/sync v0.8.0 golang.org/x/sys v0.25.0 golang.org/x/text v0.18.0 - google.golang.org/api v0.197.0 + google.golang.org/api v0.199.0 google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 - google.golang.org/grpc v1.66.2 + google.golang.org/grpc v1.67.0 google.golang.org/protobuf v1.34.2 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 @@ -69,13 +69,12 @@ require ( require ( github.com/avast/retry-go/v4 v4.6.0 github.com/fsouza/fake-gcs-server v1.49.2 - github.com/golang/protobuf v1.5.4 golang.org/x/exp v0.0.0-20231006140011-7918f672742d ) require ( cel.dev/expr v0.16.0 // indirect - cloud.google.com/go/auth v0.9.3 // indirect + cloud.google.com/go/auth v0.9.5 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect cloud.google.com/go/monitoring v1.21.1 // indirect dario.cat/mergo v1.0.0 // indirect @@ -118,7 +117,7 @@ require ( require ( cloud.google.com/go v0.115.1 // indirect - cloud.google.com/go/compute/metadata v0.5.0 // indirect + cloud.google.com/go/compute/metadata v0.5.2 // indirect cloud.google.com/go/iam v1.2.1 // indirect cloud.google.com/go/longrunning v0.6.1 // indirect github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index c34f6b1c92b..04a60cc6383 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -101,8 +101,8 @@ cloud.google.com/go/assuredworkloads v1.7.0/go.mod h1:z/736/oNmtGAyU47reJgGN+KVo cloud.google.com/go/assuredworkloads v1.8.0/go.mod h1:AsX2cqyNCOvEQC8RMPnoc0yEarXQk6WEKkxYfL6kGIo= cloud.google.com/go/assuredworkloads v1.9.0/go.mod h1:kFuI1P78bplYtT77Tb1hi0FMxM0vVpRC7VVoJC3ZoT0= cloud.google.com/go/assuredworkloads v1.10.0/go.mod h1:kwdUQuXcedVdsIaKgKTp9t0UJkE5+PAVNhdQm4ZVq2E= -cloud.google.com/go/auth v0.9.3 h1:VOEUIAADkkLtyfr3BLa3R8Ed/j6w1jTBmARx+wb5w5U= -cloud.google.com/go/auth v0.9.3/go.mod h1:7z6VY+7h3KUdRov5F1i8NDP5ZzWKYmEPO842BgCsmTk= +cloud.google.com/go/auth v0.9.5 h1:4CTn43Eynw40aFVr3GpPqsQponx2jv0BQpjvajsbbzw= +cloud.google.com/go/auth v0.9.5/go.mod h1:Xo0n7n66eHyOWWCnitop6870Ilwo3PiZyodVkkH1xWM= cloud.google.com/go/auth/oauth2adapt v0.2.4 h1:0GWE/FUsXhf6C+jAkWgYm7X9tK8cuEIfy19DBn6B6bY= cloud.google.com/go/auth/oauth2adapt v0.2.4/go.mod h1:jC/jOpwFP6JBxhB3P5Rr0a9HLMC/Pe3eaL4NmdvqPtc= cloud.google.com/go/automl v1.5.0/go.mod h1:34EjfoFGMZ5sgJ9EoLsRtdPSNZLcfflJR39VbVNS2M0= @@ -188,8 +188,8 @@ cloud.google.com/go/compute/metadata v0.1.0/go.mod h1:Z1VN+bulIf6bt4P/C37K4DyZYZ cloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k= cloud.google.com/go/compute/metadata v0.2.1/go.mod h1:jgHgmJd2RKBGzXqF5LR2EZMGxBkeanZ9wwa75XHJgOM= cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= -cloud.google.com/go/compute/metadata v0.5.0 h1:Zr0eK8JbFv6+Wi4ilXAR8FJ3wyNdpxHKJNPos6LTZOY= -cloud.google.com/go/compute/metadata v0.5.0/go.mod h1:aHnloV2TPI38yx4s9+wAZhHykWvVCfu7hQbF+9CWoiY= +cloud.google.com/go/compute/metadata v0.5.2 h1:UxK4uu/Tn+I3p2dYWTfiX4wva7aYlKixAHn3fyqngqo= +cloud.google.com/go/compute/metadata v0.5.2/go.mod h1:C66sj2AluDcIqakBq/M8lw8/ybHgOZqin2obFxa/E5k= cloud.google.com/go/contactcenterinsights v1.3.0/go.mod h1:E
(beam) branch master updated: Bump cloud.google.com/go/bigtable from 1.31.0 to 1.33.0 in /sdks (#32556)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6a09ca9f091 Bump cloud.google.com/go/bigtable from 1.31.0 to 1.33.0 in /sdks (#32556) 6a09ca9f091 is described below commit 6a09ca9f0919f7609252986a7ee04fb6f059ecf5 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Oct 2 06:41:20 2024 -0700 Bump cloud.google.com/go/bigtable from 1.31.0 to 1.33.0 in /sdks (#32556) Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.31.0 to 1.33.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.31.0...pubsub/v1.33.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigtable dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 10 +- sdks/go.sum | 20 ++-- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index bc7e43b9115..a414cd5d8fd 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -24,7 +24,7 @@ go 1.21 require ( cloud.google.com/go/bigquery v1.63.0 - cloud.google.com/go/bigtable v1.31.0 + cloud.google.com/go/bigtable v1.33.0 cloud.google.com/go/datastore v1.19.0 cloud.google.com/go/profiler v0.4.1 cloud.google.com/go/pubsub v1.43.0 @@ -60,7 +60,7 @@ require ( golang.org/x/text v0.18.0 google.golang.org/api v0.197.0 google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 - google.golang.org/grpc v1.66.1 + google.golang.org/grpc v1.66.2 google.golang.org/protobuf v1.34.2 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 @@ -77,7 +77,7 @@ require ( cel.dev/expr v0.16.0 // indirect cloud.google.com/go/auth v0.9.3 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect - cloud.google.com/go/monitoring v1.21.0 // indirect + cloud.google.com/go/monitoring v1.21.1 // indirect dario.cat/mergo v1.0.0 // indirect filippo.io/edwards25519 v1.1.0 // indirect github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect @@ -119,8 +119,8 @@ require ( require ( cloud.google.com/go v0.115.1 // indirect cloud.google.com/go/compute/metadata v0.5.0 // indirect - cloud.google.com/go/iam v1.2.0 // indirect - cloud.google.com/go/longrunning v0.6.0 // indirect + cloud.google.com/go/iam v1.2.1 // indirect + cloud.google.com/go/longrunning v0.6.1 // indirect github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/Microsoft/go-winio v0.6.2 // indirect github.com/apache/arrow/go/arrow v0.0.0-20200730104253-651201b0f516 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 978e2071655..c34f6b1c92b 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -135,8 +135,8 @@ cloud.google.com/go/bigquery v1.49.0/go.mod h1:Sv8hMmTFFYBlt/ftw2uN6dFdQPzBlREY9 cloud.google.com/go/bigquery v1.50.0/go.mod h1:YrleYEh2pSEbgTBZYMJ5SuSr0ML3ypjRB1zgf7pvQLU= cloud.google.com/go/bigquery v1.63.0 h1:yQFuJXdDukmBkiUUpjX0i1CtHLFU62HqPs/VDvSzaZo= cloud.google.com/go/bigquery v1.63.0/go.mod h1:TQto6OR4kw27bqjNTGkVk1Vo5PJlTgxvDJn6YEIZL/E= -cloud.google.com/go/bigtable v1.31.0 h1:/uVLxGVRbK4mxK/iO89VqXcL/zoTSmkltVfIDYVBluQ= -cloud.google.com/go/bigtable v1.31.0/go.mod h1:N/mwZO+4TSHOeyiE1JxO+sRPnW4bnR7WLn9AEaiJqew= +cloud.google.com/go/bigtable v1.33.0 h1:2BDaWLRAwXO14DJL/u8crbV2oUbMZkIa2eGq8Yao1bk= +cloud.google.com/go/bigtable v1.33.0/go.mod h1:HtpnH4g25VT1pejHRtInlFPnN5sjTxbQlsYBjh9t5l0= cloud.google.com/go/billing v1.4.0/go.mod h1:g9IdKBEFlItS8bTtlrZdVLWSSdSyFUZKXNS02zKMOZY= cloud.google.com/go/billing v1.5.0/go.mod h1:mztb1tBc3QekhjSgmpf/CV4LzWXLzCArwpLmP2Gm88s= cloud.google.com/go/billing v1.6.0/go.mod h1:WoXzguj+BeHXPbKfNWkqVtDdzORazmCjraY+vrxcyvI= @@ -327,8 +327,8 @@ cloud.google.com/go/iam v0.8.0/go.mod h1:lga0/y3iH6CX7sYqypWJ33hf7kkfXJag67naqGE cloud.google.com/go/iam v0.11.0/go.mod h1:9PiLDanza5D+oWFZiH1uG+RnRCfEGKoyl6yo4cgWZGY= cloud.google.com/go/iam v0.12.0/go.mod h1:knyHGviacl11zrtZUoDuYpDgLjvr28sLQaG0YB2GYAY= cloud.google.com/go/iam v0.13.0/go.mod h1:ljOg+rcNfzZ5d6f1nAUJ8ZIxOaZUVoS14bKCtaLZ/D0= -cloud.google.com/go/iam v1.2.0 h1:kZKMKVNk/IsSSc/udOb83K0hL/Yh/Gcqpz+oAkoIFN8= -cloud.google.com/go/iam v1.2.0/go.mod h1:zITGuWgsLZxd8OwAlX+eMFgZDXzBm7i
(beam) branch dependabot/go_modules/sdks/github.com/docker/docker-27.3.1incompatible deleted (was b8d07f1d0fb)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/github.com/docker/docker-27.3.1incompatible in repository https://gitbox.apache.org/repos/asf/beam.git was b8d07f1d0fb Bump github.com/docker/docker in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (36a8d39a3de -> a92eb38db67)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 36a8d39a3de [prism][java] Update Prism locator to match Python SDK semantics. (#32619) add a92eb38db67 Bump github.com/docker/docker in /sdks (#32554) No new revisions were added by this update. Summary of changes: sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
(beam) branch master updated: [prism][java] Update Prism locator to match Python SDK semantics. (#32619)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 36a8d39a3de [prism][java] Update Prism locator to match Python SDK
semantics. (#32619)
36a8d39a3de is described below
commit 36a8d39a3de42eafe94e0f2d080ec34e5161eda7
Author: Robert Burke
AuthorDate: Tue Oct 1 19:31:06 2024 -0700
[prism][java] Update Prism locator to match Python SDK semantics. (#32619)
* Update PrismLocator to match the python SDK semantics.
* Update validate beam release with prism RC validation instructions.
* rename resolveLocation to resolveSource
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
.../apache/beam/runners/prism/PrismLocator.java| 69 --
.../beam/runners/prism/PrismPipelineOptions.java | 7 +-
.../beam/runners/prism/PrismLocatorTest.java | 85 --
.../site/content/en/blog/validate-beam-release.md | 31
4 files changed, 147 insertions(+), 45 deletions(-)
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
index f69260344d1..27aea3f64df 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
+++
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
@@ -54,9 +54,9 @@ class PrismLocator {
private static final String PRISM_BIN_PATH = ".apache_beam/cache/prism/bin";
private static final Set PERMS =
PosixFilePermissions.fromString("rwxr-xr-x");
- private static final String GITHUB_DOWNLOAD_PREFIX =
- "https://github.com/apache/beam/releases/download";;
- private static final String GITHUB_TAG_PREFIX =
"https://github.com/apache/beam/releases/tag";;
+ private static final String GITHUB_COMMON_PREFIX =
"https://github.com/apache/beam/releases/";;
+ private static final String GITHUB_DOWNLOAD_PREFIX = GITHUB_COMMON_PREFIX +
"download";
+ private static final String GITHUB_TAG_PREFIX = GITHUB_COMMON_PREFIX + "tag";
private final PrismPipelineOptions options;
@@ -64,24 +64,51 @@ class PrismLocator {
this.options = options;
}
+ String resolveSource() {
+String from =
+String.format(
+"%s/v%s/%s.zip", GITHUB_DOWNLOAD_PREFIX,
RELEASE_INFO.getSdkVersion(), buildFileName());
+
+if (Strings.isNullOrEmpty(options.getPrismLocation())) {
+ return from;
+}
+from = options.getPrismLocation();
+
+// Likely a local file, return it directly.
+if (!from.startsWith("http")) {
+ return from;
+}
+
+// Validate that it's from a safe location: A Beam Github Release
+checkArgument(
+options.getPrismLocation().startsWith(GITHUB_COMMON_PREFIX),
+"Provided --prismLocation URL is not an Apache Beam Github "
++ "Release page URL or download URL: ",
+options.getPrismLocation());
+
+from = options.getPrismLocation();
+
+// If this is the tag prefix, then build the release download with the
version
+// from the given url.
+if (options.getPrismLocation().startsWith(GITHUB_TAG_PREFIX)) {
+ Path tagPath = Paths.get(options.getPrismLocation());
+ Path locVersion = tagPath.getName(tagPath.getNameCount() - 1);
+ // The "v" prefix is already included in the version name segment.
+ from = String.format("%s/%s/%s.zip", GITHUB_DOWNLOAD_PREFIX, locVersion,
buildFileName());
+}
+checkArgument(
+from.startsWith(GITHUB_DOWNLOAD_PREFIX),
+"Provided --prismLocation URL could not be resolved to a download URL.
",
+options.getPrismLocation());
+return from;
+ }
+
/**
* Downloads and prepares a Prism executable for use with the {@link
PrismRunner}. The returned
* {@link String} is the absolute path to the Prism executable.
*/
String resolve() throws IOException {
-
-String from =
-String.format("%s/v%s/%s.zip", GITHUB_DOWNLOAD_PREFIX,
getSDKVersion(), buildFileName());
-
-if (!Strings.isNullOrEmpty(options.getPrismLocation())) {
- checkArgument(
- !options.getPrismLocation().startsWith(GITHUB_TAG_PREFIX),
- "Provided --prismLocation URL is not an Apache Beam Github "
- + "Release page URL or download URL: ",
- from);
-
- from = options.getPrismLocation();
-}
+String from = resolveSource();
String fromFileName = getNameWithoutExtension(from);
Path to = Paths.get(userHome(), PRISM_BIN_PATH, fromFileName);
@@ -135,11 +162,6 @@ class PrismLocato
(beam) branch master updated: [Go] Update Go version used by Beam repo to go1.23.1 (#32575)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7da6c89592b [Go] Update Go version used by Beam repo to go1.23.1
(#32575)
7da6c89592b is described below
commit 7da6c89592b6a991ec81997c67638d222584f257
Author: Robert Burke
AuthorDate: Mon Sep 30 12:45:03 2024 -0700
[Go] Update Go version used by Beam repo to go1.23.1 (#32575)
* Update Go version used by Beam repo.
* update release docker in docker container
* update CHANGES.md with this note
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
.github/actions/setup-default-test-properties/test-properties.json| 2 +-
.github/actions/setup-environment-action/action.yml | 2 +-
.github/gh-actions-self-hosted-runners/arc/images/Dockerfile | 2 +-
.github/workflows/build_release_candidate.yml | 2 +-
CHANGES.md| 3 +++
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 2 +-
dev-support/docker/Dockerfile | 2 +-
release/src/main/Dockerfile | 4 ++--
sdks/go/run_with_go_version.sh| 2 +-
9 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/.github/actions/setup-default-test-properties/test-properties.json
b/.github/actions/setup-default-test-properties/test-properties.json
index a11af748efe..098e4ca1935 100644
--- a/.github/actions/setup-default-test-properties/test-properties.json
+++ b/.github/actions/setup-default-test-properties/test-properties.json
@@ -18,6 +18,6 @@
"SPARK_VERSIONS": ["2", "3"]
},
"GoTestProperties": {
-"SUPPORTED_VERSIONS": ["1.22"]
+"SUPPORTED_VERSIONS": ["1.23"]
}
}
diff --git a/.github/actions/setup-environment-action/action.yml
b/.github/actions/setup-environment-action/action.yml
index 2da2cf0becd..72246566290 100644
--- a/.github/actions/setup-environment-action/action.yml
+++ b/.github/actions/setup-environment-action/action.yml
@@ -57,5 +57,5 @@ runs:
if: ${{ inputs.go-version != '' }}
uses: actions/setup-go@v5
with:
-go-version: ${{ inputs.go-version == 'default' && '1.22' ||
inputs.go-version }} # never set patch, to get latest patch releases.
+go-version: ${{ inputs.go-version == 'default' && '1.23' ||
inputs.go-version }} # never set patch, to get latest patch releases.
cache-dependency-path: $${{ inputs.disable-cache && '' ||
'sdks/go.sum' }}
diff --git a/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile
b/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile
index c1c1955a217..84117c2c07e 100644
--- a/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile
+++ b/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile
@@ -31,7 +31,7 @@ RUN curl -OL
https://nodejs.org/dist/v18.16.0/node-v18.16.0-linux-x64.tar.xz &&
mv /usr/local/node-v18.16.0-linux-x64 /usr/local/node
ENV PATH="${PATH}:/usr/local/node/bin"
#Install Go
-ARG go_version=1.22.5
+ARG go_version=1.23.1
RUN curl -OL https://go.dev/dl/go${go_version}.linux-amd64.tar.gz && \
tar -C /usr/local -xzf go${go_version}.linux-amd64.tar.gz && \
rm go${go_version}.linux-amd64.tar.gz
diff --git a/.github/workflows/build_release_candidate.yml
b/.github/workflows/build_release_candidate.yml
index ec65ae99072..d2bd863483f 100644
--- a/.github/workflows/build_release_candidate.yml
+++ b/.github/workflows/build_release_candidate.yml
@@ -433,7 +433,7 @@ jobs:
fi
- uses: actions/setup-go@v5
with:
- go-version: '1.22'
+ go-version: '1.23'
- name: Import GPG key
id: import_gpg
uses:
crazy-max/ghaction-import-gpg@111c56156bcc6918c056dbef52164cfa583dc549
diff --git a/CHANGES.md b/CHANGES.md
index 7c377648c11..e7b26851f99 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -64,6 +64,9 @@
* Dataflow worker can install packages from Google Artifact Registry Python
repositories (Python) ([#32123](https://github.com/apache/beam/issues/32123)).
* Added support for Zstd codec in SerializableAvroCodecFactory (Java)
([#32349](https://github.com/apache/beam/issues/32349))
* Added support for using vLLM in the RunInference transform (Python)
([#32528](https://github.com/apache/beam/issues/32528))
+* Prism release binaries and container bootloaders are now being built with
the latest Go 1.23 patch. ([#32575](https://github.com/
(beam) branch master updated: [#32562] Incorporate Prism into the Beam Website. (#32563)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9487927adce [#32562] Incorporate Prism into the Beam Website. (#32563) 9487927adce is described below commit 9487927adce935db60caf4ee428c7d0aab654b1a Author: Robert Burke AuthorDate: Mon Sep 30 11:46:40 2024 -0700 [#32562] Incorporate Prism into the Beam Website. (#32563) --- .github/autolabeler.yml| 7 +- README.md | 1 + website/www/site/content/en/blog/beam-2.59.0.md| 2 +- .../documentation/pipelines/test-your-pipeline.md | 4 +- .../content/en/documentation/runners/direct.md | 1 + .../site/content/en/documentation/runners/prism.md | 145 + website/www/site/content/en/roadmap/_index.md | 4 +- .../www/site/content/en/roadmap/prism-runner.md| 35 + .../www/site/data/en/documentation_runners.yaml| 3 + .../layouts/partials/section-menu/en/roadmap.html | 1 + .../layouts/partials/section-menu/en/runners.html | 1 + 11 files changed, 196 insertions(+), 8 deletions(-) diff --git a/.github/autolabeler.yml b/.github/autolabeler.yml index a4ce41fce07..a1a4c3bf039 100644 --- a/.github/autolabeler.yml +++ b/.github/autolabeler.yml @@ -67,7 +67,7 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*", "sdks/python/apache_beam/ "mqtt": ["sdks/java/io/mqtt/**/*"] "parquet": ["sdks/java/io/parquet/**/*"] "rabbitmq": ["sdks/java/io/rabbitmq/**/*"] -"redis": ["sdks/java/io/redis/**/*"] +"redis": ["sdks/java/io/redis/**/*"] "solr": ["sdks/java/io/solr/**/*"] "spanner": ["sdks/go/pkg/beam/io/spannerio/**/*", "sdks/python/apache_beam/io/gcp/spanner.py", "sdks/python/apache_beam/io/gcp/experimental/spannerio.py", "sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/**/*"] "bigtable": ["sdks/go/pkg/beam/io/bigtableio/**/*", "sdks/go/pkg/beam/io/xlang/bigtableio/**/*", "sdks/python/apache_beam/io/gcp/bigtableio.py", "sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/**/*"] @@ -76,9 +76,9 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*", "sdks/python/apache_beam/ "thrift": ["sdks/java/io/thrift/**/*"] "tika": ["sdks/java/io/tika/**/*"] "xml": ["sdks/java/io/xml/**/*"] - + # Runners -"runners": ["runners/**/*", "sdks/go/pkg/beam/runners/**/*", "sdks/python/runners/**/*", "sdks/typescript/src/apache_beam/runners/**/*"] +"runners": ["runners/**/*", "sdks/go/pkg/beam/runners/**/*", "sdks/python/apache_beam/runners/**/*", "sdks/typescript/src/apache_beam/runners/**/*"] "core": ["runners/core-construction-java/**/*", "runners/core-java/**/*"] "dataflow": ["runners/google-cloud-dataflow-java/**/*", "sdks/go/pkg/beam/runners/dataflow/**/*", "sdks/python/runners/dataflow/**/*"] "direct": ["runners/direct-java/**/*", "sdks/go/pkg/beam/runners/direct/**/*", "sdks/python/runners/direct/**/*"] @@ -88,6 +88,7 @@ io: ["sdks/go/pkg/beam/io/**/*", "sdks/java/io/**/*", "sdks/python/apache_beam/ "jet": ["runners/jet/**/*"] "local": ["runners/local-java/**/*"] "portability": ["runners/portability/**/*"] +"prism": ["runners/prism/**/*", "sdks/go/pkg/beam/runners/prism/**/*", "sdks/go/cmd/prism/**/*", "sdks/python/apache_beam/runners/portability/prism_runner.py","sdks/python/apache_beam/runners/portability/prism_runner_test.py"] "samza": ["runners/samza/**/*"] "spark": ["runners/spark/**/*", "sdks/go/pkg/beam/runners/spark/**/*"] "twister2": ["runners/twister2/**/*"] diff --git a/README.md b/README.md index 8a6db162206..24ab4963701 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,7 @@ Have ideas for new SDKs or DSLs? See the [sdk-ideas label](https://github.com/ap Beam supports executing programs on multiple distributed processing backends through PipelineRunners. Currently, the following PipelineRunners are available: - The `DirectRunner` runs the pipeline on your local machine. +- The `PrismRunner` runs the pipeline on your local machine using Beam Portability. - The `DataflowRunn
(beam) branch master updated: Tour of Beam markdown touchups (#32536)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1eddbdca4c5 Tour of Beam markdown touchups (#32536)
1eddbdca4c5 is described below
commit 1eddbdca4c5efe4bfccc11af29624f9689ea2997
Author: Hai Joey Tran
AuthorDate: Thu Sep 26 11:49:24 2024 -0400
Tour of Beam markdown touchups (#32536)
* Touch up introduction md
* runners docs touchup
* add techincal clarificaiton to ptransform definition
---
.../overview-pipeline/description.md | 2 +-
.../runner-concepts/description.md | 6 ++--
.../introduction/introduction-guide/description.md | 35 --
3 files changed, 30 insertions(+), 13 deletions(-)
diff --git
a/learning/tour-of-beam/learning-content/introduction/introduction-concepts/pipeline-concepts/overview-pipeline/description.md
b/learning/tour-of-beam/learning-content/introduction/introduction-concepts/pipeline-concepts/overview-pipeline/description.md
index 5144f737524..50955741a9f 100644
---
a/learning/tour-of-beam/learning-content/introduction/introduction-concepts/pipeline-concepts/overview-pipeline/description.md
+++
b/learning/tour-of-beam/learning-content/introduction/introduction-concepts/pipeline-concepts/overview-pipeline/description.md
@@ -22,7 +22,7 @@ The Beam SDKs provide several abstractions that simplify the
mechanics of large-
→ `PCollection`: A PCollection represents a distributed data set that your
Beam pipeline operates on. The data set can be bounded, meaning it comes from a
fixed source like a file, or unbounded, meaning it comes from a continuously
updating source via a subscription or other mechanism. Your pipeline typically
creates an initial PCollection by reading data from an external data source,
but you can also create a PCollection from in-memory data within your driver
program. From there, PColle [...]
-→ `PTransform`: A PTransform represents a data processing operation, or a
step, in your pipeline. Every PTransform takes one or more PCollection objects
as the input, performs a processing function that you provide on the elements
of that PCollection, and then produces zero or more output PCollection objects.
+→ `PTransform`: A PTransform represents a data processing operation, or a
step, in your pipeline. Every PTransform takes zero or more PCollection objects
as the input, performs a processing function that you provide on the elements
of that PCollection, and then produces zero or more output PCollection objects.
{{if (eq .Sdk "go")}}
→ `Scope`: The Go SDK has an explicit scope variable used to build a
`Pipeline`. A Pipeline can return it’s root scope with the `Root()` method. The
scope variable is then passed to `PTransform` functions that place them in the
`Pipeline` that owns the `Scope`.
diff --git
a/learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md
b/learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md
index 3989f6de651..6eb1c04e966 100644
---
a/learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md
+++
b/learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md
@@ -15,7 +15,7 @@ limitations under the License.
Apache Beam provides a portable API layer for building sophisticated
data-parallel processing `pipelines` that may be executed across a diversity of
execution engines, or `runners`. The core concepts of this layer are based upon
the Beam Model (formerly referred to as the Dataflow Model), and implemented to
varying degrees in each Beam `runner`.
-### Direct runner
+### Direct Runner
The Direct Runner executes pipelines on your machine and is designed to
validate that pipelines adhere to the Apache Beam model as closely as possible.
Instead of focusing on efficient pipeline execution, the Direct Runner performs
additional checks to ensure that users do not rely on semantics that are not
guaranteed by the model. Some of these checks include:
* enforcing immutability of elements
@@ -61,9 +61,9 @@ In java, you need to set runner to `args` when you start the
program.
{{end}}
{{if (eq .Sdk "python")}}
-In the Python SDK , the default is runner **DirectRunner**.
+In the Python SDK , the **DirectRunner** is the default runner and is used if
no runner is specified.
-Additionally, you can read more about the Direct Runner
[here](https://beam.apache.org/documentation/runners/direct/)
+You can read more about the **DirectRunner**
[here](https://beam.apache.org/documentation/runners/direct/)
Run example
diff --git
a/learning/tour-of-beam/learning-content/introduction/introduction-guide/description.md
(beam) branch master updated: Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32543)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new c503ec4fda5 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32543) c503ec4fda5 is described below commit c503ec4fda5ab27b845d9e52d963a1daf3a46fab Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Sep 24 11:17:24 2024 -0700 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32543) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.17.12 to 1.17.23. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.12...credentials/v1.17.23) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 14 +++--- sdks/go.sum | 28 ++-- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index bc2f7bfc69c..5be3689dbcd 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -31,10 +31,10 @@ require ( cloud.google.com/go/spanner v1.67.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.31.0 - github.com/aws/aws-sdk-go-v2/config v1.27.34 + github.com/aws/aws-sdk-go-v2/config v1.27.37 github.com/aws/aws-sdk-go-v2/credentials v1.17.35 - github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.12 - github.com/aws/aws-sdk-go-v2/service/s3 v1.61.2 + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.23 + github.com/aws/aws-sdk-go-v2/service/s3 v1.63.1 github.com/aws/smithy-go v1.21.0 github.com/docker/go-connections v0.5.0 github.com/dustin/go-humanize v1.0.1 @@ -125,16 +125,16 @@ require ( github.com/apache/arrow/go/arrow v0.0.0-20200730104253-651201b0f516 // indirect github.com/apache/thrift v0.17.0 // indirect github.com/aws/aws-sdk-go v1.34.0 // indirect - github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 // indirect + github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.5 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.14 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.18 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.18 // indirect github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect - github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.17 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.18 // indirect github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.5 // indirect - github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.19 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.20 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.20 // indirect - github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.17 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.18 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.23.1 // indirect github.com/aws/aws-sdk-go-v2/service/ssooidc v1.27.1 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.31.1 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 950c6b10adf..3ec1ac5a055 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -679,11 +679,11 @@ github.com/aws/aws-sdk-go v1.34.0/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU github.com/aws/aws-sdk-go-v2 v1.7.1/go.mod h1:L5LuPC1ZgDr2xQS7AmIec/Jlc7O/Y1u2KxJyNVab250= github.com/aws/aws-sdk-go-v2 v1.31.0 h1:3V05LbxTSItI5kUqNwhJrrrY1BAXxXt0sN0l72QmG5U= github.com/aws/aws-sdk-go-v2 v1.31.0/go.mod h1:ztolYtaEUtdpf9Wftr31CJfLVjOnD/CVRkKOOYgF8hA= -github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 h1:70PVAiL15/aBMh5LThwgXdSQorVr91L127ttckI9QQU= -github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4/go.mod h1:/MQxMqci8tlqDH+pjmoLu1i0tbWCUP1hhyMRuFxpQCw= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.5 h1:xDAuZTn4IMm8o1LnBZvmrL8JA1io4o3YWNXgohbf20g= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.5/go.mod h1:wYSv6iDS621sEFLfKvpPE2ugjTuGlAG7iROg0hLOkfc= github.com/aws/aws-sdk-go-v2/config v1.5.0/go.mod h1:RWlPOAW3E3tbtNAqTwvSW54Of/yP3oiZXMI0xfUdjyA= -github.com/aws/aws-sdk-go-v2/config v1.27.34 h1:5sLceuETg/215nLtY/QIVB2O6cosS0iC/Tx5oyqUhbw= -github.com/aws/aws-sdk-go-v2/config v1.27.3
(beam) branch master updated: [Prism] Support BundleFinalization DoFn parameter (#32425)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 986ee964858 [Prism] Support BundleFinalization DoFn parameter (#32425)
986ee964858 is described below
commit 986ee9648582470c2ca96016698096b3edd51e42
Author: Damon
AuthorDate: Sat Sep 21 10:41:47 2024 -0600
[Prism] Support BundleFinalization DoFn parameter (#32425)
* Support BundleFinalization DoFn parameter
* Replace beam.Register with register.DoFn2x0
* Add TestParDoBundleFinalizer.* to filters
* Register test funcs
* Add filter to portable runner tests
* Temporarily skip test
* Simply tests; refactor per PR comments
* Skip tests for not lookback mode
* Clean up tests; add to coverage
* Fix import ordering
-
Co-authored-by: Robert Burke
---
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 1 +
sdks/go/pkg/beam/core/typex/special.go | 12 +++-
sdks/go/pkg/beam/forward.go| 1 +
.../pkg/beam/runners/prism/internal/handlepardo.go | 6 +-
.../beam/runners/prism/internal/jobservices/job.go | 1 +
.../pkg/beam/runners/prism/internal/preprocess.go | 1 +
sdks/go/pkg/beam/runners/prism/internal/stage.go | 12 +++-
.../runners/prism/internal/unimplemented_test.go | 1 +
.../beam/runners/prism/internal/worker/bundle.go | 11 +++
sdks/go/test/integration/integration.go| 15
sdks/go/test/integration/primitives/pardo.go | 80 ++
sdks/go/test/integration/primitives/pardo_test.go | 43
12 files changed, 177 insertions(+), 7 deletions(-)
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/translate.go
b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
index 65280ef6b93..1e30d425850 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/translate.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
@@ -494,6 +494,7 @@ func (m *marshaller) addMultiEdge(edge NamedEdge)
([]string, error) {
m.requirements[URNRequiresSplittableDoFn] = true
}
if _, ok :=
edge.Edge.DoFn.ProcessElementFn().BundleFinalization(); ok {
+ payload.RequestsFinalization = true
m.requirements[URNRequiresBundleFinalization] = true
}
if _, ok := edge.Edge.DoFn.ProcessElementFn().StateProvider();
ok {
diff --git a/sdks/go/pkg/beam/core/typex/special.go
b/sdks/go/pkg/beam/core/typex/special.go
index edc1249fe76..af36ba92d28 100644
--- a/sdks/go/pkg/beam/core/typex/special.go
+++ b/sdks/go/pkg/beam/core/typex/special.go
@@ -69,8 +69,18 @@ type Window interface {
Equals(o Window) bool
}
-// BundleFinalization allows registering callbacks to be performed after the
runner durably persists bundle results.
+// BundleFinalization allows registering callbacks for the runner to invoke
after the bundle completes and the runner
+// commits the output. Parameter is accessible during DoFn StartBundle,
ProcessElement, FinishBundle.
+// However, if your DoFn implementation requires BundleFinalization in
StartBundle or FinishBundle, it is needed in the
+// ProcessElement signature, even if not invoked,
+// Common use cases for BundleFinalization would be to perform work after
elements in a bundle have been processed.
+// See beam.ParDo for documentation on these DoFn lifecycle methods.
type BundleFinalization interface {
+
+ // RegisterCallback registers the runner to invoke func() after the
runner persists the bundle of processed elements.
+ // The time.Duration configures the callback expiration, after which
the runner will not invoke func().
+ // Returning error communicates to the runner that bundle finalization
failed and the runner may choose to attempt
+ // finalization again.
RegisterCallback(time.Duration, func() error)
}
diff --git a/sdks/go/pkg/beam/forward.go b/sdks/go/pkg/beam/forward.go
index 210c39ab4e4..b2f610b703e 100644
--- a/sdks/go/pkg/beam/forward.go
+++ b/sdks/go/pkg/beam/forward.go
@@ -204,6 +204,7 @@ type Window = typex.Window
// BundleFinalization represents the parameter used to register callbacks to
// be run once the runner has durably persisted output for a bundle.
+// See typex.BundleFinalization for more details.
type BundleFinalization = typex.BundleFinalization
// These are the reflect.Type instances of the universal types, which are used
diff --git a/sdks/go/pkg/beam/runners/prism/internal/handlepardo.go
b/sdks/go/pkg/beam/runners/prism/internal/handlepardo.go
index 2d3425af33c..13e9b6f1b79 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/handlepardo.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/handlepardo.go
@@ -78,11 +78,7 @@ func (h *pardo
(beam) branch master updated: [#32498][prism] Add split / progress back off + catch-up. (#32526)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 818966c6163 [#32498][prism] Add split / progress back off + catch-up.
(#32526)
818966c6163 is described below
commit 818966c6163dc85acbef7fc0670d86a939c914de
Author: Robert Burke
AuthorDate: Fri Sep 20 19:47:08 2024 -0700
[#32498][prism] Add split / progress back off + catch-up. (#32526)
* [#32498] Add split / progress back off.
* Use 100 milliseconds, and decrease "additively".
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
.../beam/runners/prism/internal/preprocess_test.go | 6 ++-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 46 +-
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git a/sdks/go/pkg/beam/runners/prism/internal/preprocess_test.go
b/sdks/go/pkg/beam/runners/prism/internal/preprocess_test.go
index 1c35794658d..56879a3455f 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/preprocess_test.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/preprocess_test.go
@@ -134,7 +134,11 @@ func Test_preprocessor_preProcessGraph(t *testing.T) {
}})
gotStages := pre.preProcessGraph(test.input, nil)
- if diff := cmp.Diff(test.wantStages, gotStages,
cmp.AllowUnexported(stage{}, link{}), cmpopts.EquateEmpty()); diff != "" {
+ if diff := cmp.Diff(test.wantStages, gotStages,
+ cmp.AllowUnexported(stage{}, link{}),
+ cmpopts.EquateEmpty(),
+ cmpopts.IgnoreFields(stage{}, "baseProgTick"),
+ ); diff != "" {
t.Errorf("preProcessGraph(%q) stages diff
(-want,+got)\n%v", test.name, diff)
}
diff --git a/sdks/go/pkg/beam/runners/prism/internal/stage.go
b/sdks/go/pkg/beam/runners/prism/internal/stage.go
index da23ca8ccce..24cfc750933 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/stage.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/stage.go
@@ -20,6 +20,7 @@ import (
"context"
"fmt"
"io"
+ "sync/atomic"
"time"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/mtime"
@@ -76,9 +77,32 @@ type stage struct {
SinkToPCollection map[string]string
OutputsToCoders map[string]engine.PColInfo
+
+ // Stage specific progress and splitting interval.
+ baseProgTick atomic.Value // time.Duration
+}
+
+// The minimum and maximum durations between each ProgressBundleRequest and
split evaluation.
+const (
+ minimumProgTick = 100 * time.Millisecond
+ maximumProgTick = 30 * time.Second
+)
+
+func clampTick(dur time.Duration) time.Duration {
+ switch {
+ case dur < minimumProgTick:
+ return minimumProgTick
+ case dur > maximumProgTick:
+ return maximumProgTick
+ default:
+ return dur
+ }
}
func (s *stage) Execute(ctx context.Context, j *jobservices.Job, wk *worker.W,
comps *pipepb.Components, em *engine.ElementManager, rb engine.RunBundle) (err
error) {
+ if s.baseProgTick.Load() == nil {
+ s.baseProgTick.Store(minimumProgTick)
+ }
defer func() {
// Convert execution panics to errors to fail the bundle.
if e := recover(); e != nil {
@@ -142,7 +166,9 @@ func (s *stage) Execute(ctx context.Context, j
*jobservices.Job, wk *worker.W, c
previousTotalCount := int64(-2) // Total count of all pcollection
elements.
unsplit := true
- progTick := time.NewTicker(100 * time.Millisecond)
+ baseTick := s.baseProgTick.Load().(time.Duration)
+ ticked := false
+ progTick := time.NewTicker(baseTick)
defer progTick.Stop()
var dataFinished, bundleFinished bool
// If we have no data outputs, we still need to have progress & splits
@@ -170,6 +196,7 @@ progress:
break progress // exit progress loop on close.
}
case <-progTick.C:
+ ticked = true
resp, err := b.Progress(ctx, wk)
if err != nil {
slog.Debug("SDK Error from progress, aborting
progress", "bundle", rb, "error", err.Error())
@@ -196,6 +223,7 @@ progress:
unsplit = false
continue progress
}
+
// T
(beam) branch dependabot/go_modules/sdks/github.com/docker/docker-27.2.1incompatible deleted (was baa8e5d0520)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/github.com/docker/docker-27.2.1incompatible in repository https://gitbox.apache.org/repos/asf/beam.git was baa8e5d0520 Bump github.com/docker/docker in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump github.com/docker/docker in /sdks (#32495)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 70dd6473abd Bump github.com/docker/docker in /sdks (#32495) 70dd6473abd is described below commit 70dd6473abd9601072cf23b0e6dd8697ee6b1186 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Sep 17 22:43:40 2024 -0700 Bump github.com/docker/docker in /sdks (#32495) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 27.1.2+incompatible to 27.2.1+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](https://github.com/docker/docker/compare/v27.1.2...v27.2.1) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index bbe01e42953..ffd16ceb8e5 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -143,7 +143,7 @@ require ( github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cncf/xds/go v0.0.0-20240822171458-6449f94b4d59 // indirect github.com/cpuguy83/dockercfg v0.3.1 // indirect - github.com/docker/docker v27.1.2+incompatible // but required to resolve issue docker has with go1.20 + github.com/docker/docker v27.2.1+incompatible // but required to resolve issue docker has with go1.20 github.com/docker/go-units v0.5.0 // indirect github.com/envoyproxy/go-control-plane v0.13.0 // indirect github.com/envoyproxy/protoc-gen-validate v1.1.0 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index e1523f9e1f9..fc8765815cf 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -775,8 +775,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= -github.com/docker/docker v27.1.2+incompatible h1:AhGzR1xaQIy53qCkxARaFluI00WPGtXn0AJuoQsVYTY= -github.com/docker/docker v27.1.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v27.2.1+incompatible h1:fQdiLfW7VLscyoeYEBz7/J8soYFDZV1u6VW6gJEjNMI= +github.com/docker/docker v27.2.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c= github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
(beam) branch dependabot/go_modules/sdks/golang.org/x/sys-0.25.0 deleted (was c6aed22584c)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/golang.org/x/sys-0.25.0 in repository https://gitbox.apache.org/repos/asf/beam.git was c6aed22584c Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks (#32479)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new b949ac2ca43 Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks (#32479) b949ac2ca43 is described below commit b949ac2ca43321649a80345169cb886e8e1d99d0 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Sep 16 22:11:59 2024 -0700 Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks (#32479) Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.24.0 to 0.25.0. - [Commits](https://github.com/golang/sys/compare/v0.24.0...v0.25.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 81f674c13b0..853f28e2570 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -56,7 +56,7 @@ require ( golang.org/x/net v0.28.0 golang.org/x/oauth2 v0.23.0 golang.org/x/sync v0.8.0 - golang.org/x/sys v0.24.0 + golang.org/x/sys v0.25.0 golang.org/x/text v0.18.0 google.golang.org/api v0.196.0 google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 diff --git a/sdks/go.sum b/sdks/go.sum index 9317d71eee9..d8897330cb3 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1507,8 +1507,8 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.24.0 h1:Twjiwq9dn6R1fQcyiK+wQyHWfaz/BJB+YIpzU/Cv3Xg= -golang.org/x/sys v0.24.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.25.0 h1:r+8e+loiHxRqhXVl6ML1nO3l1+oFoWbnlu2Ehimmi34= +golang.org/x/sys v0.25.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
(beam) branch master updated: Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks (#32466)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 12974595cd0 Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks (#32466) 12974595cd0 is described below commit 12974595cd0405942ab8b90ba6475c7eae0d4769 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Sun Sep 15 22:25:51 2024 -0700 Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks (#32466) Bumps [golang.org/x/text](https://github.com/golang/text) from 0.17.0 to 0.18.0. - [Release notes](https://github.com/golang/text/releases) - [Commits](https://github.com/golang/text/compare/v0.17.0...v0.18.0) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 8ead1b2d964..81f674c13b0 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -57,7 +57,7 @@ require ( golang.org/x/oauth2 v0.23.0 golang.org/x/sync v0.8.0 golang.org/x/sys v0.24.0 - golang.org/x/text v0.17.0 + golang.org/x/text v0.18.0 google.golang.org/api v0.196.0 google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 google.golang.org/grpc v1.66.0 diff --git a/sdks/go.sum b/sdks/go.sum index c946a5a656c..9317d71eee9 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1535,8 +1535,8 @@ golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= -golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc= -golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= +golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224= +golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
(beam) branch master updated: fix docker compose command (#32459)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 7a6121a3231 fix docker compose command (#32459) 7a6121a3231 is described below commit 7a6121a3231e96b604de890292a8f81db804a081 Author: Vlado Djerek AuthorDate: Sat Sep 14 23:44:19 2024 +0200 fix docker compose command (#32459) * fix docker compose command * typo fix for runner label --- .github/workflows/tour_of_beam_backend_integration.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/tour_of_beam_backend_integration.yml b/.github/workflows/tour_of_beam_backend_integration.yml index 0ba2711d0d8..11cd2e2c878 100644 --- a/.github/workflows/tour_of_beam_backend_integration.yml +++ b/.github/workflows/tour_of_beam_backend_integration.yml @@ -69,7 +69,7 @@ env: jobs: integration: -runs-on: ubuntu-latest +runs-on: ubuntu-22.04 defaults: run: working-directory: ./learning/tour-of-beam/backend @@ -88,7 +88,7 @@ jobs: # 1. Start emulators - name: Start emulators -run: docker-compose up -d +run: docker compose up -d # 2. start function-framework processes in BG - name: Compile CF @@ -118,7 +118,7 @@ jobs: - name: Stop emulators if: always() -run: docker-compose down +run: docker compose down # 5. Compare storage/datastore/index.yml VS generated - name: Check index.yaml
(beam) branch master updated: Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in /sdks (#32415)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new fb5424e5888 Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in /sdks (#32415) fb5424e5888 is described below commit fb5424e58887b9c4739ecfa4eaded3819ea1eb0d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Sep 13 15:22:18 2024 -0700 Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in /sdks (#32415) Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.41.0 to 1.43.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.41.0...pubsub/v1.43.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 58 ++-- sdks/go.sum | 126 ++-- 2 files changed, 93 insertions(+), 91 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index a4d7b69af70..8ead1b2d964 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -24,10 +24,10 @@ go 1.21 require ( cloud.google.com/go/bigquery v1.62.0 - cloud.google.com/go/bigtable v1.29.0 - cloud.google.com/go/datastore v1.17.1 + cloud.google.com/go/bigtable v1.31.0 + cloud.google.com/go/datastore v1.19.0 cloud.google.com/go/profiler v0.4.1 - cloud.google.com/go/pubsub v1.41.0 + cloud.google.com/go/pubsub v1.43.0 cloud.google.com/go/spanner v1.67.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.30.4 @@ -54,13 +54,13 @@ require ( github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.16.1 golang.org/x/net v0.28.0 - golang.org/x/oauth2 v0.22.0 + golang.org/x/oauth2 v0.23.0 golang.org/x/sync v0.8.0 golang.org/x/sys v0.24.0 golang.org/x/text v0.17.0 - google.golang.org/api v0.192.0 - google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf - google.golang.org/grpc v1.65.0 + google.golang.org/api v0.196.0 + google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 + google.golang.org/grpc v1.66.0 google.golang.org/protobuf v1.34.2 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 @@ -73,10 +73,10 @@ require ( ) require ( - cel.dev/expr v0.15.0 // indirect - cloud.google.com/go/auth v0.8.1 // indirect - cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect - cloud.google.com/go/monitoring v1.20.3 // indirect + cel.dev/expr v0.16.0 // indirect + cloud.google.com/go/auth v0.9.3 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect + cloud.google.com/go/monitoring v1.21.0 // indirect dario.cat/mergo v1.0.0 // indirect filippo.io/edwards25519 v1.1.0 // indirect github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect @@ -88,7 +88,6 @@ require ( github.com/go-logr/logr v1.4.2 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-ole/go-ole v1.2.6 // indirect - github.com/golang/protobuf v1.5.4 // indirect github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect github.com/minio/highwayhash v1.0.3 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect @@ -97,29 +96,30 @@ require ( github.com/nats-io/jwt/v2 v2.5.8 // indirect github.com/nats-io/nkeys v0.4.7 // indirect github.com/nats-io/nuid v1.0.1 // indirect + github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect github.com/shirou/gopsutil/v3 v3.23.12 // indirect github.com/shoenig/go-m1cpu v0.1.6 // indirect github.com/tklauser/go-sysconf v0.3.12 // indirect github.com/tklauser/numcpus v0.6.1 // indirect github.com/yusufpapurcu/wmi v1.2.3 // indirect - go.einride.tech/aip v0.67.1 // indirect - go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect - go.opentelemetry.io/otel v1.24.0 // indirect + go.einride.tech
(beam) branch lostluck-go-play deleted (was 4e3a3f68604)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git was 4e3a3f68604 Update playground router build.gradle to use latest Go version. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Update Playground and other location usage of Go to use latest released version by default. (#32444)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 83ae551cea1 Update Playground and other location usage of Go to use
latest released version by default. (#32444)
83ae551cea1 is described below
commit 83ae551cea12c6386123186f4c2770933aad14d3
Author: Robert Burke
AuthorDate: Fri Sep 13 11:20:21 2024 -0700
Update Playground and other location usage of Go to use latest released
version by default. (#32444)
---
playground/README.md | 2 +-
playground/backend/containers/go/build.gradle | 2 +-
playground/backend/containers/java/Dockerfile | 2 +-
playground/backend/containers/python/Dockerfile| 2 +-
playground/backend/containers/python/build.gradle | 2 +-
playground/backend/containers/router/Dockerfile| 2 +-
playground/backend/containers/router/build.gradle | 2 +-
playground/backend/containers/scio/Dockerfile | 2 +-
playground/backend/playground_functions/Dockerfile | 2 +-
playground/infrastructure/cloudbuild/playground_ci_examples.sh | 4 ++--
release/go-licenses/Dockerfile | 2 +-
11 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/playground/README.md b/playground/README.md
index eb6ed3619bc..8f4aff1663f 100644
--- a/playground/README.md
+++ b/playground/README.md
@@ -41,7 +41,7 @@ build, test, and deploy the frontend and backend services.
> - buf
> - sbt
-1. Install Go 1.20+
+1. Install Go 1.23+
**Ubuntu 22.04 and newer:**
```shell
diff --git a/playground/backend/containers/go/build.gradle
b/playground/backend/containers/go/build.gradle
index 39d79103d63..04e86eb53d3 100644
--- a/playground/backend/containers/go/build.gradle
+++ b/playground/backend/containers/go/build.gradle
@@ -88,7 +88,7 @@ docker {
buildArgs(
['BASE_IMAGE' : project.rootProject.hasProperty(["base-image"]) ?
project.rootProject["base-image"] :
- "golang:1.20-bullseye",
+ "golang:1-bullseye",
'SDK_TAG' : project.rootProject.hasProperty(["sdk-tag"]) ?
project.rootProject["sdk-tag"] :
project.rootProject.sdk_version,
'SDK_TAG_LOCAL': project.rootProject.sdk_version,
diff --git a/playground/backend/containers/java/Dockerfile
b/playground/backend/containers/java/Dockerfile
index 18a37f6f016..161fd3283f7 100644
--- a/playground/backend/containers/java/Dockerfile
+++ b/playground/backend/containers/java/Dockerfile
@@ -16,7 +16,7 @@
# limitations under the License.
###
ARG BEAM_VERSION=2.44.0
-FROM golang:1.20-bullseye AS build
+FROM golang:1-bullseye AS build
ARG BEAM_VERSION
ARG GIT_COMMIT=""
ARG GIT_TIMESTAMP="0"
diff --git a/playground/backend/containers/python/Dockerfile
b/playground/backend/containers/python/Dockerfile
index fd7d8b7f895..dc734fe34d2 100644
--- a/playground/backend/containers/python/Dockerfile
+++ b/playground/backend/containers/python/Dockerfile
@@ -15,7 +15,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###
-ARG GO_BASE_IMAGE=golang:1.20-bullseye
+ARG GO_BASE_IMAGE=golang:1-bullseye
ARG SDK_TAG
ARG BASE_IMAGE=apache/beam_python3.10_sdk:$SDK_TAG
FROM $GO_BASE_IMAGE AS build
diff --git a/playground/backend/containers/python/build.gradle
b/playground/backend/containers/python/build.gradle
index 4a845b516b8..cce805f393b 100644
--- a/playground/backend/containers/python/build.gradle
+++ b/playground/backend/containers/python/build.gradle
@@ -75,7 +75,7 @@ docker {
buildArgs(
['GO_BASE_IMAGE': project.rootProject.hasProperty(["go-base-image"]) ?
project.rootProject["go-base-image"] :
- "golang:1.20-bullseye",
+ "golang:1-bullseye",
'SDK_TAG' : project.rootProject.hasProperty(["sdk-tag"]) ?
project.rootProject["sdk-tag"] : default_beam_version,
'GIT_COMMIT' : getGitCommitHash(),
diff --git a/playground/backend/containers/router/Dockerfile
b/playground/backend/containers/router/Dockerfile
index 57717d09f8c..863461013a4 100644
--- a/playground/backend/containers/router/Dockerfile
+++ b/playground/backend/containers/router/Dockerfile
@@ -16,7 +16,7 @@
# limitations under the License.
#
(beam) branch lostluck-go-play updated: Update playground router build.gradle to use latest Go version.
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch lostluck-go-play
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/lostluck-go-play by this push:
new 4e3a3f68604 Update playground router build.gradle to use latest Go
version.
4e3a3f68604 is described below
commit 4e3a3f68604c74eb9097204fc0257aa41f1c9e20
Author: Robert Burke
AuthorDate: Thu Sep 12 11:19:48 2024 -0700
Update playground router build.gradle to use latest Go version.
---
playground/backend/containers/router/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/playground/backend/containers/router/build.gradle
b/playground/backend/containers/router/build.gradle
index 49ae2dc5d40..48dcda7a689 100644
--- a/playground/backend/containers/router/build.gradle
+++ b/playground/backend/containers/router/build.gradle
@@ -70,7 +70,7 @@ docker {
tags containerImageTags()
buildArgs(['BASE_IMAGE' : project.rootProject.hasProperty(["base-image"])
?
project.rootProject["base-image"] :
- "golang:1.20-bullseye",
+ "golang:1-bullseye",
'GIT_COMMIT' : getGitCommitHash(),
'GIT_TIMESTAMP': getGitCommitTimestamp()])
}
(beam) branch lostluck-go-play updated: Update playground Python Dockerfile to use latest Go version.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new 88df03c385a Update playground Python Dockerfile to use latest Go version. 88df03c385a is described below commit 88df03c385ac533157ef7b7a8cccaeeebf81725c Author: Robert Burke AuthorDate: Thu Sep 12 11:18:39 2024 -0700 Update playground Python Dockerfile to use latest Go version. --- playground/backend/containers/python/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/backend/containers/python/Dockerfile b/playground/backend/containers/python/Dockerfile index fd7d8b7f895..dc734fe34d2 100644 --- a/playground/backend/containers/python/Dockerfile +++ b/playground/backend/containers/python/Dockerfile @@ -15,7 +15,7 @@ # See the License for the specific language governing permissions and # limitations under the License. ### -ARG GO_BASE_IMAGE=golang:1.20-bullseye +ARG GO_BASE_IMAGE=golang:1-bullseye ARG SDK_TAG ARG BASE_IMAGE=apache/beam_python3.10_sdk:$SDK_TAG FROM $GO_BASE_IMAGE AS build
(beam) branch lostluck-go-play updated: Update playground Java Dockerfile to use latest Go version.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new f3472122c2e Update playground Java Dockerfile to use latest Go version. f3472122c2e is described below commit f3472122c2e71946d3ceb6809959c1b70127752b Author: Robert Burke AuthorDate: Thu Sep 12 11:17:45 2024 -0700 Update playground Java Dockerfile to use latest Go version. --- playground/backend/containers/java/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/backend/containers/java/Dockerfile b/playground/backend/containers/java/Dockerfile index 18a37f6f016..161fd3283f7 100644 --- a/playground/backend/containers/java/Dockerfile +++ b/playground/backend/containers/java/Dockerfile @@ -16,7 +16,7 @@ # limitations under the License. ### ARG BEAM_VERSION=2.44.0 -FROM golang:1.20-bullseye AS build +FROM golang:1-bullseye AS build ARG BEAM_VERSION ARG GIT_COMMIT="" ARG GIT_TIMESTAMP="0"
(beam) branch lostluck-go-play updated: Update playground scio Dockerfile to use latest Go version.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new 4aa3cc81cb6 Update playground scio Dockerfile to use latest Go version. 4aa3cc81cb6 is described below commit 4aa3cc81cb63e3a8d2f70dc0849d43f4d114949f Author: Robert Burke AuthorDate: Thu Sep 12 11:17:11 2024 -0700 Update playground scio Dockerfile to use latest Go version. --- playground/backend/containers/scio/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/backend/containers/scio/Dockerfile b/playground/backend/containers/scio/Dockerfile index 8e68404360d..9c9e0ffa32e 100644 --- a/playground/backend/containers/scio/Dockerfile +++ b/playground/backend/containers/scio/Dockerfile @@ -16,7 +16,7 @@ # limitations under the License. ### ARG BASE_IMAGE=openjdk:11 -FROM golang:1.20-bullseye AS build +FROM golang:1-bullseye AS build ARG GIT_COMMIT="" ARG GIT_TIMESTAMP="0"
(beam) branch lostluck-go-play updated: Update playground router Dockerfile to use latest Go version.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new 0f93dd75fdc Update playground router Dockerfile to use latest Go version. 0f93dd75fdc is described below commit 0f93dd75fdcf163aee4cae45b3c8296c845203e9 Author: Robert Burke AuthorDate: Thu Sep 12 11:15:51 2024 -0700 Update playground router Dockerfile to use latest Go version. --- playground/backend/containers/router/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/backend/containers/router/Dockerfile b/playground/backend/containers/router/Dockerfile index 57717d09f8c..863461013a4 100644 --- a/playground/backend/containers/router/Dockerfile +++ b/playground/backend/containers/router/Dockerfile @@ -16,7 +16,7 @@ # limitations under the License. ### #Dokerfile to set up the Beam Go SDK -ARG BASE_IMAGE=golang:1.20-bullseye +ARG BASE_IMAGE=golang:1-bullseye #Two-stage assembly FROM $BASE_IMAGE AS build ARG GIT_COMMIT=""
(beam) branch lostluck-go-play updated: Update playground Go build.gradle to use latest Go version.
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch lostluck-go-play
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/lostluck-go-play by this push:
new 36f27e027c4 Update playground Go build.gradle to use latest Go version.
36f27e027c4 is described below
commit 36f27e027c48ac99266308b53af5de6ad8a450e5
Author: Robert Burke
AuthorDate: Thu Sep 12 11:13:49 2024 -0700
Update playground Go build.gradle to use latest Go version.
---
playground/backend/containers/go/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/playground/backend/containers/go/build.gradle
b/playground/backend/containers/go/build.gradle
index 39d79103d63..04e86eb53d3 100644
--- a/playground/backend/containers/go/build.gradle
+++ b/playground/backend/containers/go/build.gradle
@@ -88,7 +88,7 @@ docker {
buildArgs(
['BASE_IMAGE' : project.rootProject.hasProperty(["base-image"]) ?
project.rootProject["base-image"] :
- "golang:1.20-bullseye",
+ "golang:1-bullseye",
'SDK_TAG' : project.rootProject.hasProperty(["sdk-tag"]) ?
project.rootProject["sdk-tag"] :
project.rootProject.sdk_version,
'SDK_TAG_LOCAL': project.rootProject.sdk_version,
(beam) branch lostluck-go-play updated: Update playground_functions Dockerfile to use latest go.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new 57a43ea897d Update playground_functions Dockerfile to use latest go. 57a43ea897d is described below commit 57a43ea897df3efeafbb7a3df50341beda52225e Author: Robert Burke AuthorDate: Thu Sep 12 11:09:55 2024 -0700 Update playground_functions Dockerfile to use latest go. --- playground/backend/playground_functions/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/backend/playground_functions/Dockerfile b/playground/backend/playground_functions/Dockerfile index a4045c02131..3ee2c2bdc04 100644 --- a/playground/backend/playground_functions/Dockerfile +++ b/playground/backend/playground_functions/Dockerfile @@ -18,7 +18,7 @@ # This Dockerfile is only for local testing -FROM golang:1.20-alpine as build +FROM golang:1-alpine as build COPY . /app WORKDIR /app/playground_functions
(beam) branch lostluck-go-play updated: Update go-licenses Dockerfile to always use latest Go.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new da147887ec1 Update go-licenses Dockerfile to always use latest Go. da147887ec1 is described below commit da147887ec1f4fb38ae1e4b5f5b08831946e853d Author: Robert Burke AuthorDate: Thu Sep 12 11:08:44 2024 -0700 Update go-licenses Dockerfile to always use latest Go. --- release/go-licenses/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release/go-licenses/Dockerfile b/release/go-licenses/Dockerfile index 1dfddc6e374..f37b161ab3e 100644 --- a/release/go-licenses/Dockerfile +++ b/release/go-licenses/Dockerfile @@ -16,7 +16,7 @@ # limitations under the License. ### -FROM golang:1.20-bookworm +FROM golang:1-bookworm WORKDIR /usr/src/app COPY go.mod ./
(beam) branch lostluck-go-play updated: Update playground/README.md to refer to Go 1.23
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/lostluck-go-play by this push: new 05bc5fe5e74 Update playground/README.md to refer to Go 1.23 05bc5fe5e74 is described below commit 05bc5fe5e74b9037ccf2d5a61f3a3c7d896a190f Author: Robert Burke AuthorDate: Thu Sep 12 11:07:43 2024 -0700 Update playground/README.md to refer to Go 1.23 --- playground/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/playground/README.md b/playground/README.md index eb6ed3619bc..8f4aff1663f 100644 --- a/playground/README.md +++ b/playground/README.md @@ -41,7 +41,7 @@ build, test, and deploy the frontend and backend services. > - buf > - sbt -1. Install Go 1.20+ +1. Install Go 1.23+ **Ubuntu 22.04 and newer:** ```shell
(beam) branch lostluck-go-play updated: Update playground Python build.gradle to use latest go all the time
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch lostluck-go-play
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/lostluck-go-play by this push:
new 88643d2c01c Update playground Python build.gradle to use latest go all
the time
88643d2c01c is described below
commit 88643d2c01cdb1bb73b37ec237a6679f7f3e482b
Author: Robert Burke
AuthorDate: Thu Sep 12 11:05:40 2024 -0700
Update playground Python build.gradle to use latest go all the time
---
playground/backend/containers/python/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/playground/backend/containers/python/build.gradle
b/playground/backend/containers/python/build.gradle
index 4a845b516b8..cce805f393b 100644
--- a/playground/backend/containers/python/build.gradle
+++ b/playground/backend/containers/python/build.gradle
@@ -75,7 +75,7 @@ docker {
buildArgs(
['GO_BASE_IMAGE': project.rootProject.hasProperty(["go-base-image"]) ?
project.rootProject["go-base-image"] :
- "golang:1.20-bullseye",
+ "golang:1-bullseye",
'SDK_TAG' : project.rootProject.hasProperty(["sdk-tag"]) ?
project.rootProject["sdk-tag"] : default_beam_version,
'GIT_COMMIT' : getGitCommitHash(),
(beam) branch lostluck-go-play created (now 2e1d37a3fc2)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git at 2e1d37a3fc2 Update playground_ci_examples.sh to use Go 1.23 This branch includes the following new commits: new 2e1d37a3fc2 Update playground_ci_examples.sh to use Go 1.23 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Update playground_ci_examples.sh to use Go 1.23
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-go-play in repository https://gitbox.apache.org/repos/asf/beam.git commit 2e1d37a3fc277294389337762c72c977b6b6ee87 Author: Robert Burke AuthorDate: Thu Sep 12 11:03:14 2024 -0700 Update playground_ci_examples.sh to use Go 1.23 --- playground/infrastructure/cloudbuild/playground_ci_examples.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/playground/infrastructure/cloudbuild/playground_ci_examples.sh b/playground/infrastructure/cloudbuild/playground_ci_examples.sh index 962d18a0f47..437cc337faf 100755 --- a/playground/infrastructure/cloudbuild/playground_ci_examples.sh +++ b/playground/infrastructure/cloudbuild/playground_ci_examples.sh @@ -84,7 +84,7 @@ export STEP=CI export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml" export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml" export GRADLE_VERSION=7.5.1 -export GO_VERSION=1.20 +export GO_VERSION=1.23 LogOutput "Installing python java8 and dependencies" apt-get update > /dev/null @@ -244,4 +244,4 @@ do exit 1 fi done -exit 0 \ No newline at end of file +exit 0
(beam-starter-java) branch lostluck-2590 deleted (was ed8dd87)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-2590 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git was ed8dd87 Update pom.xml for 2.59.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam-starter-java) 01/01: Merge pull request #115 from apache/lostluck-2590
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git commit 6447705bfb76be733e2a8c9ca123e75157787364 Merge: ca7907d ed8dd87 Author: Robert Burke AuthorDate: Thu Sep 12 10:45:55 2024 -0700 Merge pull request #115 from apache/lostluck-2590 Update beam-starter-java to use Beam 2.59.0 build.gradle | 4 ++-- build.sbt| 2 +- pom.xml | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
(beam-starter-java) branch main updated (ca7907d -> 6447705)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git from ca7907d Merge pull request #99 from jrmccluskey/twoFiveEight add dffa114 Update build.gradle for 2.59.0 add f2437b4 Update build.sbt for 2.59.0 add ed8dd87 Update pom.xml for 2.59.0 new 6447705 Merge pull request #115 from apache/lostluck-2590 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle | 4 ++-- build.sbt| 2 +- pom.xml | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
(beam-starter-java) branch lostluck-2590 updated: Update build.sbt for 2.59.0
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch lostluck-2590
in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git
The following commit(s) were added to refs/heads/lostluck-2590 by this push:
new f2437b4 Update build.sbt for 2.59.0
f2437b4 is described below
commit f2437b431a073ec6e4687d6c04a21746752b8390
Author: Robert Burke
AuthorDate: Thu Sep 12 10:20:14 2024 -0700
Update build.sbt for 2.59.0
---
build.sbt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build.sbt b/build.sbt
index ef0c6e5..8545867 100644
--- a/build.sbt
+++ b/build.sbt
@@ -8,7 +8,7 @@
mainClass := Some("com.example.App")
-val beamVersion = "2.58.0"
+val beamVersion = "2.59.0"
libraryDependencies ++= Seq(
// App dependencies.
"org.apache.beam" % "beam-sdks-java-core" % beamVersion,
(beam-starter-java) branch lostluck-2590 updated: Update pom.xml for 2.59.0
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-2590 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git The following commit(s) were added to refs/heads/lostluck-2590 by this push: new ed8dd87 Update pom.xml for 2.59.0 ed8dd87 is described below commit ed8dd87ebf3d28dcd033375fbb57e46e525526cf Author: Robert Burke AuthorDate: Thu Sep 12 10:21:01 2024 -0700 Update pom.xml for 2.59.0 --- pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index 924011b..d60ce10 100644 --- a/pom.xml +++ b/pom.xml @@ -80,13 +80,13 @@ org.apache.beam beam-sdks-java-core - 2.58.0 + 2.59.0 org.apache.beam beam-runners-direct-java - 2.58.0 + 2.59.0 runtime
(beam-starter-java) 01/01: Update build.gradle for 2.59.0
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch lostluck-2590
in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git
commit dffa1142b2da362ab04105b6987e517110e1ab9e
Author: Robert Burke
AuthorDate: Thu Sep 12 10:19:26 2024 -0700
Update build.gradle for 2.59.0
---
build.gradle | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/build.gradle b/build.gradle
index 0125357..ebcc42e 100644
--- a/build.gradle
+++ b/build.gradle
@@ -25,8 +25,8 @@ test {
dependencies {
// App dependencies.
-implementation "org.apache.beam:beam-sdks-java-core:2.58.0"
-implementation "org.apache.beam:beam-runners-direct-java:2.58.0"
+implementation "org.apache.beam:beam-sdks-java-core:2.59.0"
+implementation "org.apache.beam:beam-runners-direct-java:2.59.0"
implementation "org.slf4j:slf4j-jdk14:1.7.32"
// Tests dependencies.
(beam-starter-java) branch lostluck-2590 created (now dffa114)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-2590 in repository https://gitbox.apache.org/repos/asf/beam-starter-java.git at dffa114 Update build.gradle for 2.59.0 This branch includes the following new commits: new dffa114 Update build.gradle for 2.59.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch lostluck-patch-3 deleted (was 47a7169dd56)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-patch-3 in repository https://gitbox.apache.org/repos/asf/beam.git was 47a7169dd56 Fixes link typo in 2.59, and fixes archive links. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (9c31e45d595 -> ef1bb52d3d3)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9c31e45d595 2.59.0 Beam site updates. (#32308) add ef1bb52d3d3 Fixes link typo in 2.59, and fixes archive links. (#32437) No new revisions were added by this update. Summary of changes: website/www/site/content/en/get-started/downloads.md | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-)
(beam) 01/01: Fixes link typo in 2.59, and fixes archive links.
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch lostluck-patch-3 in repository https://gitbox.apache.org/repos/asf/beam.git commit 47a7169dd569793cee03bfe984f4b208cc2ad816 Author: Robert Burke AuthorDate: Wed Sep 11 15:54:48 2024 -0700 Fixes link typo in 2.59, and fixes archive links. --- website/www/site/content/en/get-started/downloads.md | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/website/www/site/content/en/get-started/downloads.md b/website/www/site/content/en/get-started/downloads.md index 280b2d488d0..08614b8835c 100644 --- a/website/www/site/content/en/get-started/downloads.md +++ b/website/www/site/content/en/get-started/downloads.md @@ -97,30 +97,30 @@ versions denoted `0.x.y`. ## Releases ### 2.59.0 (2024-09-11) -Official [source code download](https://downloads.apache.org/beam/2.5901/apache-beam-2.59.0-source-release.zip). +Official [source code download](https://downloads.apache.org/beam/2.59.0/apache-beam-2.59.0-source-release.zip). [SHA-512](https://downloads.apache.org/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512). [signature](https://downloads.apache.org/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc). [Release notes](https://github.com/apache/beam/releases/tag/v2.59.0) ### 2.58.1 (2024-08-15) -Official [source code download](https://archive.apache.org/beam/2.58.1/apache-beam-2.58.1-source-release.zip). -[SHA-512](https://archive.apache.org/beam/2.58.1/apache-beam-2.58.1-source-release.zip.sha512). -[signature](https://archive.apache.org/beam/2.58.1/apache-beam-2.58.1-source-release.zip.asc). +Official [source code download](https://archive.apache.org/dist/beam/2.58.1/apache-beam-2.58.1-source-release.zip). +[SHA-512](https://archive.apache.org/dist/beam/2.58.1/apache-beam-2.58.1-source-release.zip.sha512). +[signature](https://archive.apache.org/dist/beam/2.58.1/apache-beam-2.58.1-source-release.zip.asc). [Release notes](https://github.com/apache/beam/releases/tag/v2.58.1) ### 2.58.0 (2024-08-06) -Official [source code download](https://archive.apache.org/beam/2.58.0/apache-beam-2.58.0-source-release.zip). -[SHA-512](https://archive.apache.org/beam/2.58.0/apache-beam-2.58.0-source-release.zip.sha512). -[signature](https://archive.apache.org/beam/2.58.0/apache-beam-2.58.0-source-release.zip.asc). +Official [source code download](https://archive.apache.org/dist/beam/2.58.0/apache-beam-2.58.0-source-release.zip). +[SHA-512](https://archive.apache.org/dist/beam/2.58.0/apache-beam-2.58.0-source-release.zip.sha512). +[signature](https://archive.apache.org/dist/beam/2.58.0/apache-beam-2.58.0-source-release.zip.asc). [Release notes](https://github.com/apache/beam/releases/tag/v2.58.0) ### 2.57.0 (2024-06-26) -Official [source code download](https://archive.apache.org/beam/2.57.0/apache-beam-2.57.0-source-release.zip). -[SHA-512](https://archive.apache.org/beam/2.57.0/apache-beam-2.57.0-source-release.zip.sha512). -[signature](https://archive.apache.org/beam/2.57.0/apache-beam-2.57.0-source-release.zip.asc). +Official [source code download](https://archive.apache.org/dist/beam/2.57.0/apache-beam-2.57.0-source-release.zip). +[SHA-512](https://archive.apache.org/dist/beam/2.57.0/apache-beam-2.57.0-source-release.zip.sha512). +[signature](https://archive.apache.org/dist/beam/2.57.0/apache-beam-2.57.0-source-release.zip.asc). [Release notes](https://github.com/apache/beam/releases/tag/v2.57.0)
(beam) branch lostluck-patch-3 created (now 47a7169dd56)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-patch-3 in repository https://gitbox.apache.org/repos/asf/beam.git at 47a7169dd56 Fixes link typo in 2.59, and fixes archive links. This branch includes the following new commits: new 47a7169dd56 Fixes link typo in 2.59, and fixes archive links. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
svn commit: r71469 [2/2] - in /release/beam: 2.58.1/ 2.59.0/ 2.59.0/prism/ 2.59.0/prism/darwin/ 2.59.0/prism/darwin/amd64/ 2.59.0/prism/darwin/arm64/ 2.59.0/prism/linux/ 2.59.0/prism/linux/amd64/ 2.59
Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc == --- release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc (added) +++ release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc Wed Sep 11 22:53:28 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmbKBLQYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQi460QAMQW/ISiFX1h5YNNllA78HSp +T77d5fOTPtD4tOP75pxZt9dSanQ9YxJZBvi+pTGURU0yqdJyec0Vndp2mmAqR4so +1IuFJM3rpQogyJ6Jd66fpjlfVimgRHZJ7XJQ79HW0uDO9fRJirRnpQ86GdilhYvJ +w35LdOr9/32bfcJKtCzJxlNkIjX6kXZK+NWOnbL+CgyjScTyvLBxaBdBuHjbms0B +G84pRQoV3uEhE2n7Q46bUwWrcqlSPSW/bYd60ZDTUScj7NsfZnIKX2O5huUvLoJU +Nv/iZctmq9lCsvgV5TQMpvEoBk0UTvxfKZYmpM+vP+0HjgTfAbJSCV9skrJrT5Df +tXYh21EjCOQc810gZkc1AlGW/3AoZb+ZTj8510dCa4fIVLb+gr2rnidVIG4GJkMp +wbQaWIqY76TUeTkMJF0paS7BoKWa6/m4QSwaFg8Cl7ffM4LnUXCtc7Z8lNBt6Lv6 +iVP0ccpoFRU2+/TIAqBHa3EheYlP6hDSxQq9PtLyYzqnj82uC+9kTuAGbyeCqjpq +dQfTIVT65EPDG6k+YGvyu2RMuXTczjF91DFaP3hNezimlfGvA2BC9uTw4P4BUDQ9 +TAWr1i+qOKgfH6BUKPX/0siurTRJQIN7BCHfm6eg/q0az83QxotbCRPhy364j4QN +7U/3Xm8lDYEslIWn0hek +=0QcK +-END PGP SIGNATURE- Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 == --- release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 (added) +++ release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 Wed Sep 11 22:53:28 2024 @@ -0,0 +1 @@ +f735d33c100ebe166ca869c20e36a9385935b4073b6689e7742e5b068b3e4afccf5751decc93301700ccb6e54e35a83be84abb6646ba0a714a4b6cee44564451 apache_beam-2.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl == Binary file - no diff available. Propchange: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc == --- release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc (added) +++ release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc Wed Sep 11 22:53:28 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmbKBLQYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQijK4QAKq5njMqmzeJHL3B3dDK9mnz +MX4IAb1xcnX2xlF4gBpMUZqX0CKcV02eNs22T8DtJhPPximMOMEN0KPHoQIR96GF +varCqz2e5jBD9HnIPm3TsVc9km34zZcu0VVb9KCtxGHP0ZB0o6jBrMOBMsRm6s/S +5dR1WJ/wFyWXUKJjv5LyStsYBJ+M5xsXTS+HOK7JJ9med+HTuSPtJU/XSKFfXJJd +KKFAXqhy6VSPI22SjQlXLzCTglcEiPUsVoGrv4iw+204iTvo1gs9XzmRFqE3nHIN +wkRvpSgnfzZ2HXKQ/hd401TFG3qFA/Qe4fcnvEdoXYiectq2L9q4acdd4pSNRokL +SKAjDij4L3D/d70Z0N+5IANy/JEl3nlBIy8HGAzxSHKP32IEmHF5L3vGBbFRRXxT +Trc/w7LxcF01n03hfE2slnoAWRpWBwsph9m3ozvluBse6mgCGRxT8AobEKB/01q3 +P287M4q2sZeAz6lGx+6SKy5I0pivB0H24HVhCv0IncF1H8u/GzhSsnxgbSKpzyW9 +I6XIQvcjiylBzsqDok651Lvdd82A+QaOLB+8yFKs0TZkJvzSmwNOdE5OUjxLBHZm +ObQCoDuu0WY4n3FVC8kdgoKgqw9CPTWQT7iBus04ZL61arhG/71ZR+uRo13v3LWn +tyWt8Cp/z59GKM1Vtyj8 +=VQ5o +-END PGP SIGNATURE- Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 == --- release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 (added) +++ release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 Wed Sep 11 22:53:28 2024 @@ -0,0 +1 @@ +6fb11b52dce7900018736be0385619d72d11df727ff3f0ba57d3d72b54a5e44d3fbf493cf149cea926bffa2d60ca1b8d16ce371478f1791b133dae1e804ffd1f apache_beam-2.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-win32.whl == Binary file - no diff available. Propchange: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-win32.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.59.0/python/apache_beam-2.59.0-cp39-cp39-win32.whl.asc ==
svn commit: r71469 [1/2] - in /release/beam: 2.58.1/ 2.59.0/ 2.59.0/prism/ 2.59.0/prism/darwin/ 2.59.0/prism/darwin/amd64/ 2.59.0/prism/darwin/arm64/ 2.59.0/prism/linux/ 2.59.0/prism/linux/amd64/ 2.59
Author: lostluck Date: Wed Sep 11 22:53:28 2024 New Revision: 71469 Log: Adding artifacts for the 2.59.0 release and removing old artifacts Added: release/beam/2.59.0/ release/beam/2.59.0/apache-beam-2.59.0-source-release.zip (with props) release/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc release/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512 release/beam/2.59.0/prism/ release/beam/2.59.0/prism/darwin/ release/beam/2.59.0/prism/darwin/amd64/ release/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip (with props) release/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.asc release/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.sha512 release/beam/2.59.0/prism/darwin/arm64/ release/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip (with props) release/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip.asc release/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip.sha512 release/beam/2.59.0/prism/linux/ release/beam/2.59.0/prism/linux/amd64/ release/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip (with props) release/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip.asc release/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip.sha512 release/beam/2.59.0/prism/linux/arm64/ release/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip (with props) release/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip.asc release/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip.sha512 release/beam/2.59.0/prism/windows/ release/beam/2.59.0/prism/windows/amd64/ release/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip (with props) release/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip.asc release/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip.sha512 release/beam/2.59.0/prism/windows/arm64/ release/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip (with props) release/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip.asc release/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip.sha512 release/beam/2.59.0/python/ release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl.sha512 release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311
(beam) branch master updated: 2.59.0 Beam site updates. (#32308)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9c31e45d595 2.59.0 Beam site updates. (#32308) 9c31e45d595 is described below commit 9c31e45d59595ebac03d2bc05a8ac91ddbbdebbe Author: Robert Burke AuthorDate: Wed Sep 11 15:23:32 2024 -0700 2.59.0 Beam site updates. (#32308) * 2.59.0 Beam site updates. * Correct release data. - Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com> --- CHANGES.md | 16 - website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.59.0.md| 76 ++ .../www/site/content/en/get-started/downloads.md | 13 +++- 4 files changed, 101 insertions(+), 6 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index 47f1831b446..b936ba6dda3 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -94,11 +94,16 @@ * ([#X](https://github.com/apache/beam/issues/X)). -# [2.59.0] - Cut, 2024-08-22 +# [2.59.0] - 2024-09-11 ## Highlights * Added support for setting a configureable timeout when loading a model and performing inference in the [RunInference](https://beam.apache.org/documentation/ml/inference-overview/) transform using [with_exception_handling](https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.base.html#apache_beam.ml.inference.base.RunInference.with_exception_handling) ([#32137](https://github.com/apache/beam/issues/32137)) +* Initial experimental support for using Prism with the Java and Python SDKs + * Prism is presently targeting local testing usage, or other small scale execution. + * For Java, use 'PrismRunner', or 'TestPrismRunner' as an argument to the `--runner` flag. + * For Python, use 'PrismRunner' as an argument to the `--runner` flag. + * Go already uses Prism as the default local runner. ## I/Os @@ -115,7 +120,7 @@ * Updated Go protobuf package to new version (Go) ([#21515](https://github.com/apache/beam/issues/21515)). * Added support for setting a configureable timeout when loading a model and performing inference in the [RunInference](https://beam.apache.org/documentation/ml/inference-overview/) transform using [with_exception_handling](https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.base.html#apache_beam.ml.inference.base.RunInference.with_exception_handling) ([#32137](https://github.com/apache/beam/issues/32137)) * Adds OrderedListState support for Java SDK via FnApi. -* Initial support for using Prism from the Python SDK. +* Initial support for using Prism from the Python and Java SDKs. ## Bugfixes @@ -124,6 +129,13 @@ * (Python) Upgraded google-cloud-storage to version 2.18.2 to fix a data corruption issue ([#32135](https://github.com/apache/beam/pull/32135)). * (Go) Fix corruption on State API writes. ([#32245](https://github.com/apache/beam/issues/32245)). +## Known Issues + +* Prism is under active development and does not yet support all pipelines. See [#29650](https://github.com/apache/beam/issues/29650) for progress. + * In the 2.59.0 release, Prism passes most runner validations tests with the exceptions of pipelines using the following features: + OrderedListState, OnWindowExpiry (eg. GroupIntoBatches), CustomWindows, MergingWindowFns, Trigger and WindowingStrategy associated features, Bundle Finalization, Looping Timers, and some Coder related issues such as with Python combiner packing, and Java Schema transforms, and heterogenous flatten coders. Processing Time timers do not yet have real time support. + * If your pipeline is having difficulty with the Python or Java direct runners, but runs well on Prism, please let us know. + # [2.58.1] - 2024-08-15 ## New Features / Improvements diff --git a/website/www/site/config.toml b/website/www/site/config.toml index 23d88f752c2..e937289fbde 100644 --- a/website/www/site/config.toml +++ b/website/www/site/config.toml @@ -104,7 +104,7 @@ github_project_repo = "https://github.com/apache/beam"; [params] description = "Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Be [...] -release_latest = "2.58.1" +release_latest = "2.59.0" # The repository and branch where the files live in Github or Colab. This is used # to serve and stage from your local
(beam) branch release-2.59.0 updated: Cherrypick 32363 (#32368)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch release-2.59.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.59.0 by this push:
new 29a16a013e0 Cherrypick 32363 (#32368)
29a16a013e0 is described below
commit 29a16a013e00df13832d56f206aaab9de0de821b
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Tue Sep 3 20:47:39 2024 -0400
Cherrypick 32363 (#32368)
---
sdks/java/io/iceberg/hive/exec/build.gradle | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/sdks/java/io/iceberg/hive/exec/build.gradle
b/sdks/java/io/iceberg/hive/exec/build.gradle
index 581f71ddedd..bb0b147c5a8 100644
--- a/sdks/java/io/iceberg/hive/exec/build.gradle
+++ b/sdks/java/io/iceberg/hive/exec/build.gradle
@@ -39,10 +39,10 @@ artifacts {
shadowJar {
zip64 true
-// need to shade "com.google.guava" to avoid Guava conflict
-relocate 'com.google.protobuf', getJavaRelocatedPath('com.google.protobuf')
-relocate 'shaded.parquet', getJavaRelocatedPath('shaded.parquet')
-relocate 'org.apache.parquet', getJavaRelocatedPath('org.apache.parquet')
+relocate 'com.google.common',
getJavaRelocatedPath('iceberg.hive.com.google.common')
+relocate 'com.google.protobuf',
getJavaRelocatedPath('iceberg.hive.com.google.protobuf')
+relocate 'shaded.parquet',
getJavaRelocatedPath('iceberg.hive.shaded.parquet')
+relocate 'org.apache.parquet',
getJavaRelocatedPath('iceberg.hive.org.apache.parquet')
version "3.1.3"
mergeServiceFiles()
(beam) branch release-2.59.0 updated: Cherry Pick #32178: Loose BigQuery GCP project ID regex restrictions (#32314)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch release-2.59.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.59.0 by this push:
new 37d41fdb70d Cherry Pick #32178: Loose BigQuery GCP project ID regex
restrictions (#32314)
37d41fdb70d is described below
commit 37d41fdb70de9789c82897c0d214e82ee88536b6
Author: Kirill Berezin <126603518+kberezin-n...@users.noreply.github.com>
AuthorDate: Mon Aug 26 08:15:31 2024 -0700
Cherry Pick #32178: Loose BigQuery GCP project ID regex restrictions
(#32314)
---
.../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java | 9 +
.../org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 1 +
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index aa094ace756..71302339af1 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -574,11 +574,12 @@ public class BigQueryIO {
static final JsonFactory JSON_FACTORY = Transport.getJsonFactory();
/**
- * Project IDs must contain 6-63 lowercase letters, digits, or dashes. IDs
must start with a
- * letter and may not end with a dash. This regex isn't exact - this allows
for patterns that
- * would be rejected by the service, but this is sufficient for basic
parsing of table references.
+ * Formally, project IDs must contain 6-63 lowercase letters, digits, or
dashes, must start with a
+ * letter and may not end with a dash. This regex is used for basic parsing
of table references
+ * rather than validation purpose, e.g. it allows looser restriction for
testing on mock
+ * resources. It may allow for patterns that would be rejected by the service
*/
- private static final String PROJECT_ID_REGEXP =
"[a-z][-a-z0-9:.]{4,61}[a-z0-9]";
+ private static final String PROJECT_ID_REGEXP =
"[a-z][-a-z0-9:.]{0,61}[a-z0-9]";
/** Regular expression that matches Dataset IDs. */
private static final String DATASET_REGEXP = "[-\\w.]{1,1024}";
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
index 6779920702d..df96f3bc4c7 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
@@ -88,6 +88,7 @@ public class BigQueryHelpersTest {
public void testTableParsing_validPatterns() {
BigQueryHelpers.parseTableSpec("a123-456:foo_bar.d");
BigQueryHelpers.parseTableSpec("a12345:b.c");
+BigQueryHelpers.parseTableSpec("a1:b.c");
BigQueryHelpers.parseTableSpec("b12345.c");
}
svn commit: r71078 - /dev/beam/2.59.0/python/
Author: lostluck Date: Sat Aug 24 16:05:47 2024 New Revision: 71078 Log: Staging Python artifacts for Apache Beam 2.59.0 RC Added: dev/beam/2.59.0/python/ dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-macosx_10_9_x86_64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win32.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp310-cp310-win_amd64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-macosx_10_9_x86_64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win32.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win32.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win32.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win_amd64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win_amd64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp311-cp311-win_amd64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-macosx_10_9_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-macosx_10_9_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-macosx_10_9_x86_64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc dev/beam/2.59.0/python/apache_beam-2.59.0-cp312-cp312-manylinux_2_17_x86_64
svn commit: r71077 - in /dev/beam/2.59.0/prism: ./ darwin/ darwin/amd64/ darwin/arm64/ linux/ linux/amd64/ linux/arm64/ windows/ windows/amd64/ windows/arm64/
Author: lostluck Date: Sat Aug 24 16:00:35 2024 New Revision: 71077 Log: Staging Prism artifacts for Apache Beam 2.59.0 RC Added: dev/beam/2.59.0/prism/ dev/beam/2.59.0/prism/darwin/ dev/beam/2.59.0/prism/darwin/amd64/ dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip (with props) dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.asc dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.sha512 dev/beam/2.59.0/prism/darwin/arm64/ dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip (with props) dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip.asc dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip.sha512 dev/beam/2.59.0/prism/linux/ dev/beam/2.59.0/prism/linux/amd64/ dev/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip (with props) dev/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip.asc dev/beam/2.59.0/prism/linux/amd64/apache_beam-v2.59.0-prism-linux-amd64.zip.sha512 dev/beam/2.59.0/prism/linux/arm64/ dev/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip (with props) dev/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip.asc dev/beam/2.59.0/prism/linux/arm64/apache_beam-v2.59.0-prism-linux-arm64.zip.sha512 dev/beam/2.59.0/prism/windows/ dev/beam/2.59.0/prism/windows/amd64/ dev/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip (with props) dev/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip.asc dev/beam/2.59.0/prism/windows/amd64/apache_beam-v2.59.0-prism-windows-amd64.zip.sha512 dev/beam/2.59.0/prism/windows/arm64/ dev/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip (with props) dev/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip.asc dev/beam/2.59.0/prism/windows/arm64/apache_beam-v2.59.0-prism-windows-arm64.zip.sha512 Added: dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip == Binary file - no diff available. Propchange: dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.asc == --- dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.asc (added) +++ dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.asc Sat Aug 24 16:00:35 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmbKA3kYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQiiT4QAKF4cWeSJPCMwcYt4d28Uv7x +dxoOpew5Z0HDtwKDEN8FmvLfj8278aS4qaVQfk7jduvNSEh19nFwEqUNaD0iS5sC +mNfdWnSt683P7O0FD3a06ZmQo5ZuH7pfxLfH6t17EuLd8uxrb2kwvr29Vb8YGlAy +kfX3s0ykQXxPqhHZGuKmPMyy4Ou06PsdDyjoLLEsY+0aJloNsjjBrJ23U2rvoD61 +qvWKBr2E2y7DK+aTKjajOt6f93SMtIiIEd8FdJV6yXVE3mOSrhVXWctJmIDao9CT +fH0/i4G/u82NGVPWMPzU7jGmT27bbFP+659E8MNDyxWWBL6LCPJYXZgmLdr7cLoc +PI4b+B1QduuCoPfC1dgjoq5S7c8WEqJtJqKwJueEecbRkvSwjpiaQcogMyKpIg7r +FWYF/XugcgPcSrhnco80NMpVMM5nAt31Rlkw6c/nz3aE/DBk/Kn+P/oTQeyJ9v/S +HEEWyp20wrdHNNq/OTbu9j/7tMbvoNq86KOzU4pip+8MP2bruo3B7QwEZceRQtNP +wio9KjmjdEMOm1v1VBq9Z3/vPp2Q7IGlf0YQKxCtUoBZ9WwMRAkaPv3UsMAyi368 +qv/ye4QAdBuCSWlJSYXmYSNihbByUUW5bpRrZgd49VuPKGbaf8au+N5GCQDjmsxO +167fjAMM3EgktNnqr40V +=9pnQ +-END PGP SIGNATURE- Added: dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.sha512 == --- dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.sha512 (added) +++ dev/beam/2.59.0/prism/darwin/amd64/apache_beam-v2.59.0-prism-darwin-amd64.zip.sha512 Sat Aug 24 16:00:35 2024 @@ -0,0 +1 @@ +b8e42c521c9667ed90d7ca1031152d1b79ede166d36e50758377323262d05c196572b36bb3d6f4518b681b80bfa8b410e7b6924fb5558d7cedbd8b30878c097b apache_beam-v2.59.0-prism-darwin-amd64.zip Added: dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip == Binary file - no diff available. Propchange: dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.59.0/prism/darwin/arm64/apache_beam-v2.59.0-prism-darwin-arm64.zip.asc
(beam-site) branch updates_release_2.59.0 created (now f82136b679)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch updates_release_2.59.0 in repository https://gitbox.apache.org/repos/asf/beam-site.git at f82136b679 Update beam-site for release 2.59.0. No new revisions were added by this update.
svn commit: r71072 - in /dev/beam/2.59.0: ./ apache-beam-2.59.0-source-release.zip apache-beam-2.59.0-source-release.zip.asc apache-beam-2.59.0-source-release.zip.sha512
Author: lostluck Date: Sat Aug 24 05:25:54 2024 New Revision: 71072 Log: Staging Java artifacts for Apache Beam 2.59.0 RC1 Added: dev/beam/2.59.0/ dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip (with props) dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512 Added: dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip == Binary file - no diff available. Propchange: dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc == --- dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc (added) +++ dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.asc Sat Aug 24 05:25:54 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmbJbt8YHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQinnwQALNAMIeJeC9g/nF0VcJRLyIL +iFmcFeRooWAJMdUrbXk0B7iuCe4nO5pGvGe0oFVwMqxx4gDRJ38HFEMC8LIlbV98 +Y5PFgeTIY8HbgSesDuO+9HqeyfGe/SmUYqrVd01D8YP4U4wrZFJLp7kMh73aPhHv +jkL4jCgGNsqVu6YM80iSYWIXd+kUYyTaQR6Xoz/rlHlxp8wJGuX6qG/lMvTibxlq +i2vwoUdXM7dqJaHD/ADLYhVs9udbssyUHHKKpoAwcEeqyS8UDwWrGE2pfk/+B3iz +p/oao6wLiYZydHmNEmXD9heLbjOw/p+vUHcW4HOGJcuivh/Jl3XdhYlYSxAdFfru +lEe8L0ZWkePjKDjMSNt2bDhrwaW3/pwXa1mQ8oeJmf8kMj41YWxS9Uwaw+zw/2QA +pQKa3kvnjTGkhHyRXXZRzuPuUSRwQBoyq0WCjIpCNhjOFoDzozDyZrlgzCrTeHfG +db5B8WUw41lYmZLa4L2ztuWA3Mr+mhv8tU4r4ZIInoLoDd+7mBKrkvsPRiJtxv9p +OycNL36l05Q0j98tSaMk+h1sVksBf0a/UYxTj/EfxOdIdDytj+FtHTqwuXzqADK6 +OCgFlBpsy61q/IAqcgDghBGaRkXv1OCKtAovfD/D7vlKoVX9+Ii9aMvhxM/wU4dJ +AwsKd9cTbJtfbBNqtGnS +=q9tY +-END PGP SIGNATURE- Added: dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512 == --- dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512 (added) +++ dev/beam/2.59.0/apache-beam-2.59.0-source-release.zip.sha512 Sat Aug 24 05:25:54 2024 @@ -0,0 +1 @@ +c1b1c951006b6760d6ce3ddbb720e6674f5280fd932fa6a583f6bbd7991299c920c6aca4e80aad744bd22a2e255a02b511fd76da8380a019dd9a774639e90138 apache-beam-2.59.0-source-release.zip
(beam) annotated tag sdks/v2.59.0-RC1 updated (c5600759505 -> 8145c6acbd3)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to annotated tag sdks/v2.59.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag sdks/v2.59.0-RC1 was modified! *** from c5600759505 (commit) to 8145c6acbd3 (tag) tagging c56007595053fd69dcf5490563e1e963afb52343 (commit) replaces jupyterlab-sidepanel-v3.0.0 by lostluck on Fri Aug 23 22:17:03 2024 -0700 - Log - Go SDK v2.59.0-RC1 --- No new revisions were added by this update. Summary of changes:
(beam) annotated tag v2.59.0-RC1 updated (c5600759505 -> 8fa43368faf)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to annotated tag v2.59.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag v2.59.0-RC1 was modified! *** from c5600759505 (commit) to 8fa43368faf (tag) tagging c56007595053fd69dcf5490563e1e963afb52343 (commit) replaces jupyterlab-sidepanel-v3.0.0 by lostluck on Fri Aug 23 22:17:03 2024 -0700 - Log - v2.59.0-RC1 --- No new revisions were added by this update. Summary of changes:
(beam) branch release-2.59.0 updated: Cherry Pick #32294 (Java PrismRunner and TestPrismRunner) to the release. (#32304)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch release-2.59.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.59.0 by this push:
new 57d5ab5a144 Cherry Pick #32294 (Java PrismRunner and TestPrismRunner)
to the release. (#32304)
57d5ab5a144 is described below
commit 57d5ab5a144c03cba9d309fa2beed4baa984a474
Author: Robert Burke
AuthorDate: Fri Aug 23 16:32:51 2024 -0700
Cherry Pick #32294 (Java PrismRunner and TestPrismRunner) to the release.
(#32304)
---
runners/prism/java/build.gradle| 1 +
.../apache/beam/runners/prism/PrismExecutor.java | 5 +-
.../apache/beam/runners/prism/PrismLocator.java| 12 +-
.../beam/runners/prism/PrismPipelineOptions.java | 17 +++
.../beam/runners/prism/PrismPipelineResult.java| 33 ++
.../org/apache/beam/runners/prism/PrismRunner.java | 72 +---
.../beam/runners/prism/PrismRunnerRegistrar.java | 35 ++
.../runners/prism/TestPrismPipelineOptions.java| 23
.../apache/beam/runners/prism/TestPrismRunner.java | 84 +
.../beam/runners/prism/PrismExecutorTest.java | 8 +-
.../runners/prism/PrismPipelineResultTest.java | 130 -
.../apache/beam/runners/prism/PrismRunnerTest.java | 117 +++
12 files changed, 336 insertions(+), 201 deletions(-)
diff --git a/runners/prism/java/build.gradle b/runners/prism/java/build.gradle
index 96ab4e70a57..deee8876af6 100644
--- a/runners/prism/java/build.gradle
+++ b/runners/prism/java/build.gradle
@@ -37,6 +37,7 @@ dependencies {
implementation library.java.slf4j_api
implementation library.java.vendored_grpc_1_60_1
implementation library.java.vendored_guava_32_1_2_jre
+compileOnly library.java.hamcrest
testImplementation library.java.junit
testImplementation library.java.mockito_core
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
index 620d5508f22..fda5db923a7 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
+++
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
@@ -45,6 +45,9 @@ import org.slf4j.LoggerFactory;
abstract class PrismExecutor {
private static final Logger LOG =
LoggerFactory.getLogger(PrismExecutor.class);
+ static final String IDLE_SHUTDOWN_TIMEOUT = "-idle_shutdown_timeout=%s";
+ static final String JOB_PORT_FLAG_TEMPLATE = "-job_port=%s";
+ static final String SERVE_HTTP_FLAG_TEMPLATE = "-serve_http=%s";
protected @MonotonicNonNull Process process;
protected ExecutorService executorService =
Executors.newSingleThreadExecutor();
@@ -71,7 +74,7 @@ abstract class PrismExecutor {
}
executorService.shutdown();
try {
- boolean ignored = executorService.awaitTermination(1000L,
TimeUnit.MILLISECONDS);
+ boolean ignored = executorService.awaitTermination(5000L,
TimeUnit.MILLISECONDS);
} catch (InterruptedException ignored) {
}
if (process == null) {
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
index f32e4d88f42..f69260344d1 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
+++
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
@@ -127,7 +127,9 @@ class PrismLocator {
}
copyFn.accept(from.toUri().toURL().openStream(), to);
-ByteStreams.copy(from.toUri().toURL().openStream(),
Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from.toUri().toURL().openStream(), out);
+}
Files.setPosixFilePermissions(to, PERMS);
return to.toString();
@@ -159,16 +161,16 @@ class PrismLocator {
}
private static void copy(InputStream from, Path to) {
-try {
- ByteStreams.copy(from, Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from, out);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private static void download(URL from, Path to) {
-try {
- ByteStreams.copy(from.openStream(), Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from.openStream(), out);
} catch (IOException e) {
throw new RuntimeException(e);
}
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismPipelineOptions.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismPipelineOptions.java
index ec0f8beb620..6a6ca4e615d 100644
---
a/runners/prism/java
(beam) branch master updated: Implement Java TestPrismRunner and PrismRunner (#32294)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b8bbf593abf Implement Java TestPrismRunner and PrismRunner (#32294)
b8bbf593abf is described below
commit b8bbf593abf3fbae7df8343d9b052fe0aef1434a
Author: Damon
AuthorDate: Fri Aug 23 23:16:59 2024 +
Implement Java TestPrismRunner and PrismRunner (#32294)
---
runners/prism/java/build.gradle| 1 +
.../apache/beam/runners/prism/PrismExecutor.java | 5 +-
.../apache/beam/runners/prism/PrismLocator.java| 12 +-
.../beam/runners/prism/PrismPipelineOptions.java | 17 +++
.../beam/runners/prism/PrismPipelineResult.java| 33 ++
.../org/apache/beam/runners/prism/PrismRunner.java | 72 +---
.../beam/runners/prism/PrismRunnerRegistrar.java | 35 ++
.../runners/prism/TestPrismPipelineOptions.java| 23
.../apache/beam/runners/prism/TestPrismRunner.java | 84 +
.../beam/runners/prism/PrismExecutorTest.java | 8 +-
.../runners/prism/PrismPipelineResultTest.java | 130 -
.../apache/beam/runners/prism/PrismRunnerTest.java | 117 +++
12 files changed, 336 insertions(+), 201 deletions(-)
diff --git a/runners/prism/java/build.gradle b/runners/prism/java/build.gradle
index 96ab4e70a57..deee8876af6 100644
--- a/runners/prism/java/build.gradle
+++ b/runners/prism/java/build.gradle
@@ -37,6 +37,7 @@ dependencies {
implementation library.java.slf4j_api
implementation library.java.vendored_grpc_1_60_1
implementation library.java.vendored_guava_32_1_2_jre
+compileOnly library.java.hamcrest
testImplementation library.java.junit
testImplementation library.java.mockito_core
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
index 620d5508f22..fda5db923a7 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
+++
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismExecutor.java
@@ -45,6 +45,9 @@ import org.slf4j.LoggerFactory;
abstract class PrismExecutor {
private static final Logger LOG =
LoggerFactory.getLogger(PrismExecutor.class);
+ static final String IDLE_SHUTDOWN_TIMEOUT = "-idle_shutdown_timeout=%s";
+ static final String JOB_PORT_FLAG_TEMPLATE = "-job_port=%s";
+ static final String SERVE_HTTP_FLAG_TEMPLATE = "-serve_http=%s";
protected @MonotonicNonNull Process process;
protected ExecutorService executorService =
Executors.newSingleThreadExecutor();
@@ -71,7 +74,7 @@ abstract class PrismExecutor {
}
executorService.shutdown();
try {
- boolean ignored = executorService.awaitTermination(1000L,
TimeUnit.MILLISECONDS);
+ boolean ignored = executorService.awaitTermination(5000L,
TimeUnit.MILLISECONDS);
} catch (InterruptedException ignored) {
}
if (process == null) {
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
index f32e4d88f42..f69260344d1 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
+++
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismLocator.java
@@ -127,7 +127,9 @@ class PrismLocator {
}
copyFn.accept(from.toUri().toURL().openStream(), to);
-ByteStreams.copy(from.toUri().toURL().openStream(),
Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from.toUri().toURL().openStream(), out);
+}
Files.setPosixFilePermissions(to, PERMS);
return to.toString();
@@ -159,16 +161,16 @@ class PrismLocator {
}
private static void copy(InputStream from, Path to) {
-try {
- ByteStreams.copy(from, Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from, out);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private static void download(URL from, Path to) {
-try {
- ByteStreams.copy(from.openStream(), Files.newOutputStream(to));
+try (OutputStream out = Files.newOutputStream(to)) {
+ ByteStreams.copy(from.openStream(), out);
} catch (IOException e) {
throw new RuntimeException(e);
}
diff --git
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismPipelineOptions.java
b/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismPipelineOptions.java
index ec0f8beb620..6a6ca4e615d 100644
---
a/runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismPipelineOptions.java
+++
b/ru
(beam) branch master updated: Parallelize building wheels per language version (#32297)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1e80815d12d Parallelize building wheels per language version (#32297)
1e80815d12d is described below
commit 1e80815d12d08351ef67cafa0db25a7287e45ead
Author: Danny McCormick
AuthorDate: Fri Aug 23 18:23:26 2024 +0100
Parallelize building wheels per language version (#32297)
* Parallelize building wheels per language version
* Add comment to keep in sync
---
.github/workflows/build_wheels.yml | 29 -
1 file changed, 16 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 1275b38b9d2..69ffec573fc 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -48,6 +48,7 @@ jobs:
runs-on: [self-hosted, ubuntu-20.04, main]
env:
EVENT_NAME: ${{ github.event_name }}
+ # Keep in sync with py_version matrix value below - if changed, change
that as well.
PY_VERSIONS_FULL: "cp38-* cp39-* cp310-* cp311-* cp312-*"
outputs:
gcp-variables-set: ${{
steps.check_gcp_variables.outputs.gcp-variables-set }}
@@ -219,22 +220,21 @@ jobs:
- check_env_variables
- build_source
env:
- CIBW_ARCHS_LINUX: ${{matrix.arch}}
+ CIBW_ARCHS_LINUX: ${{matrix.os_python.arch}}
runs-on: ${{ matrix.os_python.runner }}
timeout-minutes: 480
strategy:
matrix:
os_python: [
- {"os": "ubuntu-20.04", "runner": [self-hosted, ubuntu-20.04, main],
"python": "${{ needs.check_env_variables.outputs.py-versions-full }}" },
+ {"os": "ubuntu-20.04", "runner": [self-hosted, ubuntu-20.04, main],
"python": "${{ needs.check_env_variables.outputs.py-versions-full }}", arch:
"auto" },
# Temporarily pin to macos-13 because macos-latest breaks this build
# TODO(https://github.com/apache/beam/issues/31114)
- {"os": "macos-13", "runner": "macos-13", "python": "${{
needs.check_env_variables.outputs.py-versions-test }}" },
- {"os": "windows-latest", "runner": "windows-latest", "python": "${{
needs.check_env_variables.outputs.py-versions-test }}" },
+ {"os": "macos-13", "runner": "macos-13", "python": "${{
needs.check_env_variables.outputs.py-versions-test }}", arch: "auto" },
+ {"os": "windows-latest", "runner": "windows-latest", "python": "${{
needs.check_env_variables.outputs.py-versions-test }}", arch: "auto" },
+ {"os": "ubuntu-20.04", "runner": [self-hosted, ubuntu-20.04, main],
"python": "${{ needs.check_env_variables.outputs.py-versions-test }}", arch:
"aarch64" }
]
-arch: [auto]
-include:
- - os_python: {"os": "ubuntu-20.04", "runner": [self-hosted,
ubuntu-20.04, main], "python": "${{
needs.check_env_variables.outputs.py-versions-test }}" }
-arch: aarch64
+# Keep in sync with PY_VERSIONS_FULL env var abvove - if changed,
change that as well.
+py_version: ["cp38-*", "cp39-*", "cp310-*", "cp311-*", "cp312-*"]
steps:
- name: Download python source distribution from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
@@ -254,15 +254,17 @@ jobs:
with:
python-version: 3.8
- uses: docker/setup-qemu-action@v1
- if: ${{matrix.arch == 'aarch64'}}
+ if: ${{matrix.os_python.arch == 'aarch64'}}
name: Set up QEMU
- name: Install cibuildwheel
# note: sync cibuildwheel version with gradle task sdks:python:bdistPy*
steps
run: pip install cibuildwheel==2.17.0 setuptools
- name: Build wheel
+ # Only build wheel if it is one of the target versions for this
platform, otherwise no-op
+ if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
working-directory: apache-beam-source
env:
-CIBW_BUILD: ${{ matrix.os_python.python }}
+CIBW_BUILD: ${{ matrix.py_version }}
# TODO: https://github.com/apache/beam/issues/23048
CIBW_SKIP: "*-musllinux_*"
CIBW_BEFORE_BUILD: pip install cython==0.29.36 numpy
--config-settings=setup-args="-Dallow-noblas=true" && pip install --upg
(beam) branch master updated: Remove some setup steps we don't need anymore (#32300)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1a8ef79eb35 Remove some setup steps we don't need anymore (#32300)
1a8ef79eb35 is described below
commit 1a8ef79eb3511e58c9fa7940946e3f0381cc4134
Author: Danny McCormick
AuthorDate: Fri Aug 23 18:21:25 2024 +0100
Remove some setup steps we don't need anymore (#32300)
---
contributor-docs/release-guide.md | 84 +--
1 file changed, 1 insertion(+), 83 deletions(-)
diff --git a/contributor-docs/release-guide.md
b/contributor-docs/release-guide.md
index b3d3c77d25d..de7be907a63 100644
--- a/contributor-docs/release-guide.md
+++ b/contributor-docs/release-guide.md
@@ -96,7 +96,7 @@ to release.
-### Prepare accounts, keys, etc
+### Prepare accounts, etc
Before your first release, you need to make sure you have all the necessary
accounts, keys, and access for publishing the release. The release process also
@@ -108,7 +108,6 @@ These are the credentials you will need:
- Apache ID and Password
- GitHub ID, Password, and Personal Access Token
- PyPi account with beam maintainer access and API Token
- - GPG pass phrase & 16-digit key ID
- Access to Beam's Apache Nexus repository
- Account to access to apache-beam-testing Google Cloud Platform project. The
account must have permissions to start Cloud Build triggers. Required for
@@ -142,64 +141,6 @@ you became an Apache Beam Committer.
- [ ] Generate a [PyPI APIToken](https://pypi.org/help/#apitoken) for use
during the release.
- GPG Key
-
-You need to have a GPG key to sign the release artifacts. Please be aware of
-the ASF-wide [release signing
-guidelines](https://www.apache.org/dev/release-signing.html). If you don’t
-have a GPG key associated with your Apache account, you must now create one
-according to the guidelines.
-
-Run the following helper script, or you can open it and run the commands
-individually (helpful if it doesn't work as intended or if you already are
-partially set up)
-
-./release/src/main/scripts/preparation_before_release.sh
-
-> **__NOTE__**:
-> When generating the key, please make sure you choose the key type as
-> __RSA and RSA (default)__ and key size as __4096 bit__.
-
-Now you should have:
-
- - [ ] A GPG key meeting ASF guidelines
- - [ ] The key added to
- [dev KEYS](https://dist.apache.org/repos/dist/dev/beam/KEYS) and
[release KEYS](https://dist.apache.org/repos/dist/release/beam/KEYS)
- **NOTE**: Only PMC can write into [release
repo](https://dist.apache.org/repos/dist/release/beam/).
- - [ ] The `user.signingkey` set in your `.gitconfig`
- - [ ] `gpg-agent` with the key loaded
-
-# Key ID
-
-You may need your Key ID for future steps. Determine your Apache GPG Key and
-Key ID as follows:
-
-gpg --list-sigs --keyid-format LONG
-
-This will list your GPG keys. One of these should reflect your Apache account,
-for example:
-
---
-pub rsa4096/845E6689845E6689 2016-02-23
-uid Nomen Nescio
-sub rsa4096/BA4D50BEBA4D50BE 2016-02-23
-
-Here, the key ID is the 16-digit hex string in the `pub` line:
`845E6689845E6689`.
-
-# Submit your GPG public key into Ubuntu OpenPGP Key Server
-
-In order to make yourself have right permission to stage java artifacts in
-Apache Nexus staging repository, please submit your GPG public key into the
-[Ubuntu OpenPGP Key Server](https://keyserver.ubuntu.com/).
-
-You will need to use an ascii-armored version of your key. This can be
-obtained by running:
-
-gpg --export --armor
-
-Copying the whole block including `-START PGP PUBLIC KEY BLOCK-` and
-`-END PGP PUBLIC KEY BLOCK-`
-
Access to Apache Nexus repository
Configure access to the [Apache Nexus
@@ -209,29 +150,6 @@ releases to the Maven Central Repository.
1. Log in with your Apache account.
2. Confirm you have appropriate access by finding `org.apache.beam` under
`Staging Profiles`.
-3. Navigate to your `Profile` (top right dropdown menu of the page).
-4. Choose `User Token` from the dropdown, then click `Access User Token`. Copy
- a snippet of the Maven XML configuration block.
-5. Insert this snippet
- twice into your global Maven `settings.xml` file, typically
- `${HOME}/.m2/settings.xml`. The end result should look like this, where
- `TOKEN_NAME` and `TOKEN_PASSWORD` are your secret tokens:
-
-
-
-
-
- apache.releases.https
- TOKEN_NAME
- TOKEN_PASSWORD
-
-
- apache.snapshots.https
- TOKEN_NAME
- TOKEN_PASSWORD
-
-
-
(beam) branch users/damccorm/setup deleted (was eb853de811c)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch users/damccorm/setup in repository https://gitbox.apache.org/repos/asf/beam.git was eb853de811c Remove some setup steps we don't need anymore The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (ebf80f326b2 -> e62daf70077)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ebf80f326b2 download BQMS catalog jar at build time (#32282) add e62daf70077 [CHANGES.md] Add 2.60.0 section. (#32287) No new revisions were added by this update. Summary of changes: CHANGES.md | 37 - 1 file changed, 36 insertions(+), 1 deletion(-)
(beam) branch users/damccorm/hot_key_cp deleted (was b106c72cb68)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch users/damccorm/hot_key_cp in repository https://gitbox.apache.org/repos/asf/beam.git was b106c72cb68 Check experiment for hotkey logging (#32285) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch release-2.59.0 updated: Cherrypick #32276 to 2.59.0 release branch. (#32280)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch release-2.59.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.59.0 by this push:
new 4173f54504f Cherrypick #32276 to 2.59.0 release branch. (#32280)
4173f54504f is described below
commit 4173f54504fbc95dd1e85c897f99b4333e7590ec
Author: Robert Burke
AuthorDate: Thu Aug 22 10:08:51 2024 -0700
Cherrypick #32276 to 2.59.0 release branch. (#32280)
* Add an idle shutdown timout to prism binary.
* Correct flag text.
Co-authored-by: Jack McCluskey
<34928439+jrmcclus...@users.noreply.github.com>
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
Co-authored-by: Jack McCluskey
<34928439+jrmcclus...@users.noreply.github.com>
---
sdks/go/cmd/prism/prism.go | 25 ++
.../prism/internal/jobservices/management.go | 22 +++--
.../runners/prism/internal/jobservices/server.go | 53 +-
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 8 ++--
sdks/go/pkg/beam/runners/prism/prism.go| 12 +
5 files changed, 102 insertions(+), 18 deletions(-)
diff --git a/sdks/go/cmd/prism/prism.go b/sdks/go/cmd/prism/prism.go
index 804ae0c2ab2..39c19df00dc 100644
--- a/sdks/go/cmd/prism/prism.go
+++ b/sdks/go/cmd/prism/prism.go
@@ -30,16 +30,24 @@ import (
)
var (
- jobPort= flag.Int("job_port", 8073, "specify the job
management service port")
- webPort= flag.Int("web_port", 8074, "specify the web ui
port")
- jobManagerEndpoint = flag.String("jm_override", "", "set to only stand
up a web ui that refers to a seperate JobManagement endpoint")
- serveHTTP = flag.Bool("serve_http", true, "enable or disable
the web ui")
+ jobPort = flag.Int("job_port", 8073, "specify the job
management service port")
+ webPort = flag.Int("web_port", 8074, "specify the web ui
port")
+ jobManagerEndpoint = flag.String("jm_override", "", "set to only stand
up a web ui that refers to a seperate JobManagement endpoint")
+ serveHTTP = flag.Bool("serve_http", true, "enable or disable
the web ui")
+ idleShutdownTimeout = flag.Duration("idle_shutdown_timeout", -1,
"duration that prism will wait for a new job before shutting itself down.
Negative durations disable auto shutdown. Defaults to never shutting down.")
)
func main() {
flag.Parse()
- ctx := context.Background()
- cli, err := makeJobClient(ctx, prism.Options{Port: *jobPort},
*jobManagerEndpoint)
+ ctx, cancel := context.WithCancelCause(context.Background())
+
+ cli, err := makeJobClient(ctx,
+ prism.Options{
+ Port:*jobPort,
+ IdleShutdownTimeout: *idleShutdownTimeout,
+ CancelFn:cancel,
+ },
+ *jobManagerEndpoint)
if err != nil {
log.Fatalf("error creating job server: %v", err)
}
@@ -47,10 +55,9 @@ func main() {
if err := prism.CreateWebServer(ctx, cli, prism.Options{Port:
*webPort}); err != nil {
log.Fatalf("error creating web server: %v", err)
}
- } else {
- // Block main thread forever to keep main from exiting.
- <-(chan struct{})(nil) // receives on nil channels block.
}
+ // Block main thread forever to keep main from exiting.
+ <-ctx.Done()
}
func makeJobClient(ctx context.Context, opts prism.Options, endpoint string)
(jobpb.JobServiceClient, error) {
diff --git a/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
b/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
index 2b03eddff05..b957b99ca63 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
@@ -73,12 +73,14 @@ func (e *joinError) Error() string {
return string(b)
}
-func (s *Server) Prepare(ctx context.Context, req *jobpb.PrepareJobRequest)
(*jobpb.PrepareJobResponse, error) {
+func (s *Server) Prepare(ctx context.Context, req *jobpb.PrepareJobRequest) (_
*jobpb.PrepareJobResponse, err error) {
s.mu.Lock()
defer s.mu.Unlock()
// Since jobs execute in the background, they should not be tied to a
request's context.
rootCtx, cancelFn := context.WithCancelCause(context.Background())
+ // Wrap in a Once so it will only be i
(beam) branch master updated: [prism] Add an idle shutdown timout to prism binary. (#32276)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 995724d651a [prism] Add an idle shutdown timout to prism binary.
(#32276)
995724d651a is described below
commit 995724d651ac8e65dbcfff05fbb932878b004cbf
Author: Robert Burke
AuthorDate: Thu Aug 22 07:31:30 2024 -0700
[prism] Add an idle shutdown timout to prism binary. (#32276)
---
sdks/go/cmd/prism/prism.go | 25 ++
.../prism/internal/jobservices/management.go | 22 +++--
.../runners/prism/internal/jobservices/server.go | 53 +-
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 8 ++--
sdks/go/pkg/beam/runners/prism/prism.go| 12 +
5 files changed, 102 insertions(+), 18 deletions(-)
diff --git a/sdks/go/cmd/prism/prism.go b/sdks/go/cmd/prism/prism.go
index 804ae0c2ab2..39c19df00dc 100644
--- a/sdks/go/cmd/prism/prism.go
+++ b/sdks/go/cmd/prism/prism.go
@@ -30,16 +30,24 @@ import (
)
var (
- jobPort= flag.Int("job_port", 8073, "specify the job
management service port")
- webPort= flag.Int("web_port", 8074, "specify the web ui
port")
- jobManagerEndpoint = flag.String("jm_override", "", "set to only stand
up a web ui that refers to a seperate JobManagement endpoint")
- serveHTTP = flag.Bool("serve_http", true, "enable or disable
the web ui")
+ jobPort = flag.Int("job_port", 8073, "specify the job
management service port")
+ webPort = flag.Int("web_port", 8074, "specify the web ui
port")
+ jobManagerEndpoint = flag.String("jm_override", "", "set to only stand
up a web ui that refers to a seperate JobManagement endpoint")
+ serveHTTP = flag.Bool("serve_http", true, "enable or disable
the web ui")
+ idleShutdownTimeout = flag.Duration("idle_shutdown_timeout", -1,
"duration that prism will wait for a new job before shutting itself down.
Negative durations disable auto shutdown. Defaults to never shutting down.")
)
func main() {
flag.Parse()
- ctx := context.Background()
- cli, err := makeJobClient(ctx, prism.Options{Port: *jobPort},
*jobManagerEndpoint)
+ ctx, cancel := context.WithCancelCause(context.Background())
+
+ cli, err := makeJobClient(ctx,
+ prism.Options{
+ Port:*jobPort,
+ IdleShutdownTimeout: *idleShutdownTimeout,
+ CancelFn:cancel,
+ },
+ *jobManagerEndpoint)
if err != nil {
log.Fatalf("error creating job server: %v", err)
}
@@ -47,10 +55,9 @@ func main() {
if err := prism.CreateWebServer(ctx, cli, prism.Options{Port:
*webPort}); err != nil {
log.Fatalf("error creating web server: %v", err)
}
- } else {
- // Block main thread forever to keep main from exiting.
- <-(chan struct{})(nil) // receives on nil channels block.
}
+ // Block main thread forever to keep main from exiting.
+ <-ctx.Done()
}
func makeJobClient(ctx context.Context, opts prism.Options, endpoint string)
(jobpb.JobServiceClient, error) {
diff --git a/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
b/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
index 2b03eddff05..b957b99ca63 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
@@ -73,12 +73,14 @@ func (e *joinError) Error() string {
return string(b)
}
-func (s *Server) Prepare(ctx context.Context, req *jobpb.PrepareJobRequest)
(*jobpb.PrepareJobResponse, error) {
+func (s *Server) Prepare(ctx context.Context, req *jobpb.PrepareJobRequest) (_
*jobpb.PrepareJobResponse, err error) {
s.mu.Lock()
defer s.mu.Unlock()
// Since jobs execute in the background, they should not be tied to a
request's context.
rootCtx, cancelFn := context.WithCancelCause(context.Background())
+ // Wrap in a Once so it will only be invoked a single time for the job.
+ terminalOnceWrap := sync.OnceFunc(s.jobTerminated)
job := &Job{
key:s.nextId(),
Pipeline: req.GetPipeline(),
@@ -86,10 +88,16 @@ func (s *Server) Prepare(ctx context.Context, req
*jobpb.PrepareJobRequest) (*jo
options:req.GetPipelineOptions(),
strea
(beam) branch master updated (c5bf5fc7395 -> 65f556d861e)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c5bf5fc7395 Revert "Portable runner fixes (#32247)" (#32271) add 65f556d861e Update requests package to 2.32.0 (#32270) No new revisions were added by this update. Summary of changes: sdks/python/container/py310/base_image_requirements.txt | 2 +- sdks/python/container/py311/base_image_requirements.txt | 2 +- sdks/python/container/py312/base_image_requirements.txt | 2 +- sdks/python/container/py38/base_image_requirements.txt | 2 +- sdks/python/container/py39/base_image_requirements.txt | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
(beam) branch revert-32247-portable-runner-fixes deleted (was dbff26e8839)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch revert-32247-portable-runner-fixes in repository https://gitbox.apache.org/repos/asf/beam.git was dbff26e8839 Revert "Portable runner fixes (#32247)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Revert "Portable runner fixes (#32247)" (#32271)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c5bf5fc7395 Revert "Portable runner fixes (#32247)" (#32271)
c5bf5fc7395 is described below
commit c5bf5fc7395f8b8c7cf65c0f8fd36b9b98a4ea01
Author: Robert Burke
AuthorDate: Wed Aug 21 11:47:06 2024 -0700
Revert "Portable runner fixes (#32247)" (#32271)
This reverts commit bd65ee9c5bfe63170849d1e2f11eb0ce3c49c0a5.
---
.../portability/JobServicePipelineResult.java | 252 +++--
.../beam/runners/portability/PortableRunner.java | 88 +++
.../runners/portability/PortableRunnerTest.java| 2 -
3 files changed, 183 insertions(+), 159 deletions(-)
diff --git
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
index 261017c83d6..9b286273a36 100644
---
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
+++
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
@@ -17,182 +17,196 @@
*/
package org.apache.beam.runners.portability;
-import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
-
+import java.util.Iterator;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
-import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.model.jobmanagement.v1.JobApi;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent;
import
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.ListenableFuture;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.ListeningScheduledExecutorService;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.checkerframework.checker.nullness.qual.Nullable;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+@SuppressWarnings({"keyfor", "nullness"}) //
TODO(https://github.com/apache/beam/issues/20497)
class JobServicePipelineResult implements PipelineResult, AutoCloseable {
private static final long POLL_INTERVAL_MS = 3_000;
private static final Logger LOG =
LoggerFactory.getLogger(JobServicePipelineResult.class);
- private final ListeningScheduledExecutorService executorService =
-
MoreExecutors.listeningDecorator(Executors.newSingleThreadScheduledExecutor());
-
- private final String jobId;
- private final JobServiceBlockingStub jobService;
- private final AtomicReference latestState = new
AtomicReference<>(State.UNKNOWN);
- private final Runnable cleanup;
- private final AtomicReference jobMetrics =
- new
AtomicReference<>(PortableMetrics.of(JobApi.MetricResults.getDefaultInstance()));
- private CompletableFuture terminalStateFuture = new
CompletableFuture<>();
- private CompletableFuture metricResultsCompletableFuture =
- new CompletableFuture<>();
- JobServicePipelineResult(String jobId, JobServiceBlockingStub jobService,
Runnable cleanup) {
+ private final ByteString jobId;
+ private final int jobServerTimeout;
+ private final CloseableResource jobService;
+ private @Nullable State terminalState;
+ private final @Nullable Runnable cleanup;
+ private org.apache.beam.model.jobmanagement.v1.JobApi.MetricResults
jobMetrics;
+
+ JobServicePipelineResult(
+ ByteString jobId,
+ int jobServerTimeout,
+ CloseableResource jobService,
+ Runnable cleanup) {
this.jobId = jobId;
+this.jobServerTimeout = jobServerTimeout;
this.jobService = jobService;
+this.terminalState = null;
this.cleanup = cleanup;
}
@Override
public State getState() {
-if (latestState.get().isTerminal()) {
(beam) branch revert-32247-portable-runner-fixes created (now dbff26e8839)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch revert-32247-portable-runner-fixes in repository https://gitbox.apache.org/repos/asf/beam.git at dbff26e8839 Revert "Portable runner fixes (#32247)" This branch includes the following new commits: new dbff26e8839 Revert "Portable runner fixes (#32247)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Revert "Portable runner fixes (#32247)"
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch revert-32247-portable-runner-fixes
in repository https://gitbox.apache.org/repos/asf/beam.git
commit dbff26e8839be8d136abfadc48d0dc096b963a08
Author: Robert Burke
AuthorDate: Wed Aug 21 10:03:34 2024 -0700
Revert "Portable runner fixes (#32247)"
This reverts commit bd65ee9c5bfe63170849d1e2f11eb0ce3c49c0a5.
---
.../portability/JobServicePipelineResult.java | 252 +++--
.../beam/runners/portability/PortableRunner.java | 88 +++
.../runners/portability/PortableRunnerTest.java| 2 -
3 files changed, 183 insertions(+), 159 deletions(-)
diff --git
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
index 261017c83d6..9b286273a36 100644
---
a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
+++
b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
@@ -17,182 +17,196 @@
*/
package org.apache.beam.runners.portability;
-import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
-
+import java.util.Iterator;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
-import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.model.jobmanagement.v1.JobApi;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent;
import
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.ListenableFuture;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.ListeningScheduledExecutorService;
-import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.checkerframework.checker.nullness.qual.Nullable;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+@SuppressWarnings({"keyfor", "nullness"}) //
TODO(https://github.com/apache/beam/issues/20497)
class JobServicePipelineResult implements PipelineResult, AutoCloseable {
private static final long POLL_INTERVAL_MS = 3_000;
private static final Logger LOG =
LoggerFactory.getLogger(JobServicePipelineResult.class);
- private final ListeningScheduledExecutorService executorService =
-
MoreExecutors.listeningDecorator(Executors.newSingleThreadScheduledExecutor());
-
- private final String jobId;
- private final JobServiceBlockingStub jobService;
- private final AtomicReference latestState = new
AtomicReference<>(State.UNKNOWN);
- private final Runnable cleanup;
- private final AtomicReference jobMetrics =
- new
AtomicReference<>(PortableMetrics.of(JobApi.MetricResults.getDefaultInstance()));
- private CompletableFuture terminalStateFuture = new
CompletableFuture<>();
- private CompletableFuture metricResultsCompletableFuture =
- new CompletableFuture<>();
- JobServicePipelineResult(String jobId, JobServiceBlockingStub jobService,
Runnable cleanup) {
+ private final ByteString jobId;
+ private final int jobServerTimeout;
+ private final CloseableResource jobService;
+ private @Nullable State terminalState;
+ private final @Nullable Runnable cleanup;
+ private org.apache.beam.model.jobmanagement.v1.JobApi.MetricResults
jobMetrics;
+
+ JobServicePipelineResult(
+ ByteString jobId,
+ int jobServerTimeout,
+ CloseableResource jobService,
+ Runnable cleanup) {
this.jobId = jobId;
+this.jobServerTimeout = jobServerTimeout;
this.jobService = jobService;
+this.terminalState = null;
this.cleanup = cleanup;
}
@Override
public State getState() {
-if (latestState.get().isTerminal()) {
- return latestState.get();
+if (terminalState != null) {
+ return terminalState;
}
+JobServiceBlockingStub stub =
+jobService.get(
(beam) branch master updated: Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32266)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 3fb4fd0f6ea Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32266) 3fb4fd0f6ea is described below commit 3fb4fd0f6ea5df9fc51548336812c1a58e54e758 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 20 22:08:13 2024 -0700 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32266) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.17.11 to 1.17.12. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.11...credentials/v1.17.12) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 9715a4eb654..a89b527dd02 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -33,8 +33,8 @@ require ( github.com/aws/aws-sdk-go-v2 v1.30.4 github.com/aws/aws-sdk-go-v2/config v1.27.28 github.com/aws/aws-sdk-go-v2/credentials v1.17.28 - github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.11 - github.com/aws/aws-sdk-go-v2/service/s3 v1.59.0 + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.12 + github.com/aws/aws-sdk-go-v2/service/s3 v1.60.0 github.com/aws/smithy-go v1.20.4 github.com/docker/go-connections v0.5.0 github.com/dustin/go-humanize v1.0.1 diff --git a/sdks/go.sum b/sdks/go.sum index 8d1f7c1131f..e1f164294e3 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -693,8 +693,8 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.3.0/go.mod h1:2LAuqPx1I6jNfaGDu github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.12 h1:yjwoSyDZF8Jth+mUk5lSPJCkMC0lMy6FaCD51jm6ayE= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.12/go.mod h1:fuR57fAgMk7ot3WcNQfb6rSEn+SUffl7ri+aa8uKysI= github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.3.2/go.mod h1:qaqQiHSrOUVOfKe6fhgQ6UzhxjwqVW8aHNegd6Ws4w4= -github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.11 h1:FEDZD/Axt5tKSkPAs967KZ++MkvYdBqr0a+cetRbjLM= -github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.11/go.mod h1:dvlsbA32KfvCzqwTiX7maABgFek2RyUuYEJ3kyn/PmQ= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.12 h1:i7cJ1izNlox4ka6cvbHPTztYGtbpW4Je/jyQIKOIU4A= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.12/go.mod h1:lHnam/4CTEVHaANZD54IrpE80VLK+lUU84WEeJ1FJ8M= github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 h1:TNyt/+X43KJ9IJJMjKfa3bNTiZbUP7DeCxfbTROESwY= github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16/go.mod h1:2DwJF39FlNAUiX5pAc0UNeiz16lK2t7IaFcm0LFHEgc= github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 h1:jYfy8UPmd+6kJW5YhY0L1/KftReOGxI/4NtVSTh9O/I= @@ -716,8 +716,8 @@ github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.5.1/go.mod h1:6EQZIwNN github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.16 h1:jg16PhLPUiHIj8zYIW6bqzeQSuHVEiWnGA0Brz5Xv2I= github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.16/go.mod h1:Uyk1zE1VVdsHSU7096h/rwnXDzOzYQVl+FNPhPw7ShY= github.com/aws/aws-sdk-go-v2/service/s3 v1.11.1/go.mod h1:XLAGFrEjbvMCLvAtWLLP32yTv8GpBquCApZEycDLunI= -github.com/aws/aws-sdk-go-v2/service/s3 v1.59.0 h1:Cso4Ev/XauMVsbwdhYEoxg8rxZWw43CFqqaPB5w3W2c= -github.com/aws/aws-sdk-go-v2/service/s3 v1.59.0/go.mod h1:BSPI0EfnYUuNHPS0uqIo5VrRwzie+Fp+YhQOUs16sKI= +github.com/aws/aws-sdk-go-v2/service/s3 v1.60.0 h1:2QXGJvG19QwqXUvgcdoCOZPyLuvZf8LiXPCN4P53TdI= +github.com/aws/aws-sdk-go-v2/service/s3 v1.60.0/go.mod h1:BSPI0EfnYUuNHPS0uqIo5VrRwzie+Fp+YhQOUs16sKI= github.com/aws/aws-sdk-go-v2/service/sso v1.3.1/go.mod h1:J3A3RGUvuCZjvSuZEcOpHDnzZP/sKbhDWV2T1EOzFIM= github.com/aws/aws-sdk-go-v2/service/sso v1.22.5 h1:zCsFCKvbj25i7p1u94imVoO447I/sFv8qq+lGJhRN0c= github.com/aws/aws-sdk-go-v2/service/sso v1.22.5/go.mod h1:ZeDX1SnKsVlejeuz41GiajjZpRSWR7/42q/EyA/QEiM=
(beam) branch dependabot/go_modules/sdks/github.com/tetratelabs/wazero-1.8.0 deleted (was f4deeef0036)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/github.com/tetratelabs/wazero-1.8.0 in repository https://gitbox.apache.org/repos/asf/beam.git was f4deeef0036 Bump github.com/tetratelabs/wazero from 1.7.3 to 1.8.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump github.com/tetratelabs/wazero from 1.7.3 to 1.8.0 in /sdks (#32193)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8601bbaed8d Bump github.com/tetratelabs/wazero from 1.7.3 to 1.8.0 in /sdks (#32193) 8601bbaed8d is described below commit 8601bbaed8d1cd385b86c8cd688dcace53d300bb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 20 17:36:00 2024 -0700 Bump github.com/tetratelabs/wazero from 1.7.3 to 1.8.0 in /sdks (#32193) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.7.3 to 1.8.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](https://github.com/tetratelabs/wazero/compare/v1.7.3...v1.8.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index a65eab9cc84..9715a4eb654 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -49,7 +49,7 @@ require ( github.com/proullon/ramsql v0.1.4 github.com/spf13/cobra v1.8.1 github.com/testcontainers/testcontainers-go v0.32.0 - github.com/tetratelabs/wazero v1.7.3 + github.com/tetratelabs/wazero v1.8.0 github.com/xitongsys/parquet-go v1.6.2 github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.16.1 diff --git a/sdks/go.sum b/sdks/go.sum index db6a94edee0..8d1f7c1131f 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1166,8 +1166,8 @@ github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsT github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/testcontainers/testcontainers-go v0.32.0 h1:ug1aK08L3gCHdhknlTTwWjPHPS+/alvLJU/DRxTD/ME= github.com/testcontainers/testcontainers-go v0.32.0/go.mod h1:CRHrzHLQhlXUsa5gXjTOfqIEJcrK5+xMDmBr/WMI88E= -github.com/tetratelabs/wazero v1.7.3 h1:PBH5KVahrt3S2AHgEjKu4u+LlDbbk+nsGE3KLucy6Rw= -github.com/tetratelabs/wazero v1.7.3/go.mod h1:ytl6Zuh20R/eROuyDaGPkp82O9C/DJfXAwJfQ3X6/7Y= +github.com/tetratelabs/wazero v1.8.0 h1:iEKu0d4c2Pd+QSRieYbnQC9yiFlMS9D+Jr0LsRmcF4g= +github.com/tetratelabs/wazero v1.8.0/go.mod h1:yAI0XTsMBhREkM/YDAK/zNou3GoiAce1P6+rp/wQhjs= github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU= github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI= github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
(beam) branch dependabot/go_modules/sdks/google.golang.org/api-0.192.0 deleted (was 231fd6a51e2)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/google.golang.org/api-0.192.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 231fd6a51e2 Bump google.golang.org/api from 0.191.0 to 0.192.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump google.golang.org/api from 0.191.0 to 0.192.0 in /sdks (#32175)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 228554b7319 Bump google.golang.org/api from 0.191.0 to 0.192.0 in /sdks (#32175) 228554b7319 is described below commit 228554b7319b561713bcc065414d133d0fe61572 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 20 17:26:47 2024 -0700 Bump google.golang.org/api from 0.191.0 to 0.192.0 in /sdks (#32175) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.191.0 to 0.192.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.191.0...v0.192.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 9da162fd69f..a65eab9cc84 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -58,7 +58,7 @@ require ( golang.org/x/sync v0.8.0 golang.org/x/sys v0.24.0 golang.org/x/text v0.17.0 - google.golang.org/api v0.191.0 + google.golang.org/api v0.192.0 google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf google.golang.org/grpc v1.65.0 google.golang.org/protobuf v1.34.2 @@ -74,7 +74,7 @@ require ( require ( cel.dev/expr v0.15.0 // indirect - cloud.google.com/go/auth v0.8.0 // indirect + cloud.google.com/go/auth v0.8.1 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect cloud.google.com/go/monitoring v1.20.3 // indirect dario.cat/mergo v1.0.0 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 26b61f81681..db6a94edee0 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -101,8 +101,8 @@ cloud.google.com/go/assuredworkloads v1.7.0/go.mod h1:z/736/oNmtGAyU47reJgGN+KVo cloud.google.com/go/assuredworkloads v1.8.0/go.mod h1:AsX2cqyNCOvEQC8RMPnoc0yEarXQk6WEKkxYfL6kGIo= cloud.google.com/go/assuredworkloads v1.9.0/go.mod h1:kFuI1P78bplYtT77Tb1hi0FMxM0vVpRC7VVoJC3ZoT0= cloud.google.com/go/assuredworkloads v1.10.0/go.mod h1:kwdUQuXcedVdsIaKgKTp9t0UJkE5+PAVNhdQm4ZVq2E= -cloud.google.com/go/auth v0.8.0 h1:y8jUJLl/Fg+qNBWxP/Hox2ezJvjkrPb952PC1p0G6A4= -cloud.google.com/go/auth v0.8.0/go.mod h1:qGVp/Y3kDRSDZ5gFD/XPUfYQ9xW1iI7q8RIRoCyBbJc= +cloud.google.com/go/auth v0.8.1 h1:QZW9FjC5lZzN864p13YxvAtGUlQ+KgRL+8Sg45Z6vxo= +cloud.google.com/go/auth v0.8.1/go.mod h1:qGVp/Y3kDRSDZ5gFD/XPUfYQ9xW1iI7q8RIRoCyBbJc= cloud.google.com/go/auth/oauth2adapt v0.2.3 h1:MlxF+Pd3OmSudg/b1yZ5lJwoXCEaeedAguodky1PcKI= cloud.google.com/go/auth/oauth2adapt v0.2.3/go.mod h1:tMQXOfZzFuNuUxOypHlQEXgdfX5cuhwU+ffUuXRJE8I= cloud.google.com/go/automl v1.5.0/go.mod h1:34EjfoFGMZ5sgJ9EoLsRtdPSNZLcfflJR39VbVNS2M0= @@ -1690,8 +1690,8 @@ google.golang.org/api v0.108.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/ google.golang.org/api v0.110.0/go.mod h1:7FC4Vvx1Mooxh8C5HWjzZHcavuS2f6pmJpZx60ca7iI= google.golang.org/api v0.111.0/go.mod h1:qtFHvU9mhgTJegR31csQ+rwxyUTHOKFqCKWp1J0fdw0= google.golang.org/api v0.114.0/go.mod h1:ifYI2ZsFK6/uGddGfAD5BMxlnkBqCmqHSDUVi45N5Yg= -google.golang.org/api v0.191.0 h1:cJcF09Z+4HAB2t5qTQM1ZtfL/PemsLFkcFG67qq2afk= -google.golang.org/api v0.191.0/go.mod h1:tD5dsFGxFza0hnQveGfVk9QQYKcfp+VzgRqyXFxE0+E= +google.golang.org/api v0.192.0 h1:PljqpNAfZaaSpS+TnANfnNAXKdzHM/B9bKhwRlo7JP0= +google.golang.org/api v0.192.0/go.mod h1:9VcphjvAxPKLmSxVSzPlSRXy/5ARMEw5bf58WoVXafQ= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
(beam) branch master updated: Bump cloud.google.com/go/spanner from 1.66.0 to 1.67.0 in /sdks (#32234)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 65427efe9d0 Bump cloud.google.com/go/spanner from 1.66.0 to 1.67.0 in /sdks (#32234) 65427efe9d0 is described below commit 65427efe9d0b79614926401143a2e07afd6cc92d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 20 16:59:37 2024 -0700 Bump cloud.google.com/go/spanner from 1.66.0 to 1.67.0 in /sdks (#32234) Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.66.0 to 1.67.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.66.0...spanner/v1.67.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/spanner dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 7c29492be8a..9da162fd69f 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -28,7 +28,7 @@ require ( cloud.google.com/go/datastore v1.17.1 cloud.google.com/go/profiler v0.4.1 cloud.google.com/go/pubsub v1.41.0 - cloud.google.com/go/spanner v1.66.0 + cloud.google.com/go/spanner v1.67.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.30.4 github.com/aws/aws-sdk-go-v2/config v1.27.28 @@ -187,7 +187,7 @@ require ( golang.org/x/crypto v0.26.0 // indirect golang.org/x/mod v0.18.0 // indirect golang.org/x/tools v0.22.0 // indirect - golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect + golang.org/x/xerrors v0.0.0-20240716161551-93cc26a95ae9 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20240725223205-93522f1f2a9f // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20240730163845-b1a4ccb954bf // indirect ) diff --git a/sdks/go.sum b/sdks/go.sum index 18bd39ba47b..26b61f81681 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -540,8 +540,8 @@ cloud.google.com/go/shell v1.6.0/go.mod h1:oHO8QACS90luWgxP3N9iZVuEiSF84zNyLytb+ cloud.google.com/go/spanner v1.41.0/go.mod h1:MLYDBJR/dY4Wt7ZaMIQ7rXOTLjYrmxLE/5ve9vFfWos= cloud.google.com/go/spanner v1.44.0/go.mod h1:G8XIgYdOK+Fbcpbs7p2fiprDw4CaZX63whnSMLVBxjk= cloud.google.com/go/spanner v1.45.0/go.mod h1:FIws5LowYz8YAE1J8fOS7DJup8ff7xJeetWEo5REA2M= -cloud.google.com/go/spanner v1.66.0 h1:PF1upR8n+DVUO9mUpCc1j5kyHn1Xfq0A53ZrnM0AmeU= -cloud.google.com/go/spanner v1.66.0/go.mod h1:gu+weqqrnoBsVlxOmMG5pzDZ2nkpqqJx4MsnmIacH5w= +cloud.google.com/go/spanner v1.67.0 h1:h8xfobxh5lQu4qJVMPH+wSiyU+ZM6ZTxRNqGeu9iIVA= +cloud.google.com/go/spanner v1.67.0/go.mod h1:Um+TNmxfcCHqNCKid4rmAMvoe/Iu1vdz6UfxJ9GPxRQ= cloud.google.com/go/speech v1.6.0/go.mod h1:79tcr4FHCimOp56lwC01xnt/WPJZc4v3gzyT7FoBkCM= cloud.google.com/go/speech v1.7.0/go.mod h1:KptqL+BAQIhMsj1kOP2la5DSEEerPDuOP/2mmkhHhZQ= cloud.google.com/go/speech v1.8.0/go.mod h1:9bYIl1/tjsAnMgKGHKmBZzXKEkGgtU+MpdDPTE9f7y0= @@ -1621,8 +1621,8 @@ golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= -golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 h1:+cNy6SZtPcJQH3LJVLOSmiC7MMxXNOb3PU/VUEz+EhU= -golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= +golang.org/x/xerrors v0.0.0-20240716161551-93cc26a95ae9 h1:LLhsEBxRTBLuKlQxFBYUOU8xyFgXv6cOTp2HASDlsDk= +golang.org/x/xerrors v0.0.0-20240716161551-93cc26a95ae9/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= gonum.org/v1/gonum v0.0.0-20180816165407-929014505bf4/go.mod h1:Y+Yx5eoAFn32cQvJDxZx5Dpnq+c3wtXuadVZAcxbbBo= gonum.org/v1/gonum v0.8.2/go.mod h1:oe/vMfY3deqTw+1EZJhuvEW2iwGF1bW9wwu7XCu0+v0= gonum.org/v1/gonum v0.9.3/go.mod h1:TZumC3NeyVQskjXqmyWt4S3bINhy7B4eYwW69EbyX+0=
(beam) branch master updated: [#32221] [prism] Terminate streams for each timerfamily+transform pair. (#32223)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 05b1781c6ea [#32221] [prism] Terminate streams for each
timerfamily+transform pair. (#32223)
05b1781c6ea is described below
commit 05b1781c6eae2df48bc69572ded7d0fcc6ed3757
Author: Robert Burke
AuthorDate: Tue Aug 20 15:27:45 2024 -0700
[#32221] [prism] Terminate streams for each timerfamily+transform pair.
(#32223)
* [#32221] Mark is-last for each timer stream correctly.
* Remove test override for clear, since it now passes unmodified.
* delint
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
sdks/go/pkg/beam/runners/prism/internal/stage.go | 22 ---
.../beam/runners/prism/internal/worker/bundle.go | 5 ++--
.../runners/portability/prism_runner_test.py | 32 --
3 files changed, 15 insertions(+), 44 deletions(-)
diff --git a/sdks/go/pkg/beam/runners/prism/internal/stage.go
b/sdks/go/pkg/beam/runners/prism/internal/stage.go
index 3d1e506f5e3..da23ca8ccce 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/stage.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/stage.go
@@ -55,15 +55,17 @@ type link struct {
// account, but all serialization boundaries remain since the pcollections
// would continue to get serialized.
type stage struct {
- ID string
- transforms []string
- primaryInput string // PCollection used as the
parallel input.
- outputs []link // PCollections that must escape
this stage.
- sideInputs []engine.LinkID // Non-parallel input PCollections
and their consumers
- internalCols []string// PCollections that escape. Used
for precise coder sending.
- envIDstring
- stateful bool
- hasTimers[]string
+ ID string
+ transforms []string
+ primaryInput string // PCollection used as the parallel input.
+ outputs []link // PCollections that must escape this
stage.
+ sideInputs []engine.LinkID // Non-parallel input PCollections and
their consumers
+ internalCols []string// PCollections that escape. Used for
precise coder sending.
+ envIDstring
+ stateful bool
+ // hasTimers indicates the transform+timerfamily pairs that need to be
waited on for
+ // the stage to be considered complete.
+ hasTimers[]struct{ Transform, TimerFamily string }
processingTimeTimers map[string]bool
exe transformExecuter
@@ -396,7 +398,7 @@ func buildDescriptor(stg *stage, comps *pipepb.Components,
wk *worker.W, em *eng
}
}
for timerID, v := range pardo.GetTimerFamilySpecs() {
- stg.hasTimers = append(stg.hasTimers, tid)
+ stg.hasTimers = append(stg.hasTimers, struct{
Transform, TimerFamily string }{Transform: tid, TimerFamily: timerID})
if v.TimeDomain == pipepb.TimeDomain_PROCESSING_TIME {
if stg.processingTimeTimers == nil {
stg.processingTimeTimers =
map[string]bool{}
diff --git a/sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go
b/sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go
index 842c5fdfc19..50e427ca36f 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go
@@ -42,7 +42,7 @@ type B struct {
InputTransformID string
Input []*engine.Block // Data and Timers for this
bundle.
EstimatedInputElements int
- HasTimers []string
+ HasTimers []struct{ Transform, TimerFamily string } //
Timer streams to terminate.
// IterableSideInputData is a map from transformID + inputID, to
window, to data.
IterableSideInputData map[SideInputKey]map[typex.Window][][]byte
@@ -175,7 +175,8 @@ func (b *B) ProcessOn(ctx context.Context, wk *W) <-chan
struct{} {
for _, tid := range b.HasTimers {
timers = append(timers, &fnpb.Elements_Timers{
InstructionId: b.InstID,
- TransformId: tid,
+ TransformId: tid.Transform,
+ TimerFamilyId: tid.TimerFamily,
IsLast:true,
})
}
diff --git a/sdks/python/apache_beam/runners/portability/prism_runner_test.py
b/sdks/python/apache_beam/runners/portability/prism_runner_test.py
index 387b7ba2bec..04a2dbd4fae
(beam) branch master updated (6582e7ae538 -> bd65ee9c5bf)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6582e7ae538 Fix it-mongo compile fail on Java11 (#32243) add bd65ee9c5bf Portable runner fixes (#32247) No new revisions were added by this update. Summary of changes: .../portability/JobServicePipelineResult.java | 252 ++--- .../beam/runners/portability/PortableRunner.java | 88 --- .../runners/portability/PortableRunnerTest.java| 2 + 3 files changed, 159 insertions(+), 183 deletions(-)
(beam) branch master updated (ff93c48a4c5 -> f1e214712c8)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ff93c48a4c5 Update Gemma 2 Notebook (#32200) add f1e214712c8 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32214) No new revisions were added by this update. Summary of changes: sdks/go.mod | 38 +++ sdks/go.sum | 76 ++--- 2 files changed, 57 insertions(+), 57 deletions(-)
(beam) branch master updated: [Java11 Migration] Migrate Go testing to use Java11 container image for xlang (#32212)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ced67ec1100 [Java11 Migration] Migrate Go testing to use Java11
container image for xlang (#32212)
ced67ec1100 is described below
commit ced67ec11007b7345ba1f5e68cfd39eac1af1b28
Author: Robert Burke
AuthorDate: Thu Aug 15 16:08:09 2024 -0700
[Java11 Migration] Migrate Go testing to use Java11 container image for
xlang (#32212)
---
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 ++
sdks/go/examples/kafka/taxi.go | 6 +++---
sdks/go/examples/xlang/bigquery/wordcount.go | 6 +++---
sdks/go/test/build.gradle | 10 +-
4 files changed, 13 insertions(+), 11 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index d36432c1d4d..a86ae4fa142 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -543,6 +543,8 @@ class BeamModulePlugin implements Plugin {
project.apply plugin: "com.dorongold.task-tree"
project.taskTree { noRepeat = true }
+project.ext.currentJavaVersion = getSupportedJavaVersion()
+
project.ext.allFlinkVersions = project.flink_versions.split(',')
project.ext.latestFlinkVersion = project.ext.allFlinkVersions.last()
diff --git a/sdks/go/examples/kafka/taxi.go b/sdks/go/examples/kafka/taxi.go
index 6f4532dad93..fe36dd4abde 100644
--- a/sdks/go/examples/kafka/taxi.go
+++ b/sdks/go/examples/kafka/taxi.go
@@ -91,8 +91,8 @@
// Docker Hub.
//
// export DOCKER_ROOT="Your Docker Repository Root"
-// ./gradlew :sdks:java:container:java8:docker
-Pdocker-repository-root=$DOCKER_ROOT -Pdocker-tag=latest
-// docker push $DOCKER_ROOT/beam_java8_sdk:latest
+// ./gradlew :sdks:java:container:java11:docker
-Pdocker-repository-root=$DOCKER_ROOT -Pdocker-tag=latest
+// docker push $DOCKER_ROOT/beam_java11_sdk:latest
//
// For runners in local mode, simply building the container using the default
// values for docker-repository-root and docker-tag will work to have it
@@ -102,7 +102,7 @@
// pipeline with the --sdk_harness_container_image_override flag for Java, or
// --environment_config flag for Go. For example:
//
-//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java8_sdk:latest"
\
+//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java11_sdk:latest"
\
// --environment_config=${DOCKER_ROOT}/beam_go_sdk:latest
package main
diff --git a/sdks/go/examples/xlang/bigquery/wordcount.go
b/sdks/go/examples/xlang/bigquery/wordcount.go
index b836efdca49..83fab1ae36d 100644
--- a/sdks/go/examples/xlang/bigquery/wordcount.go
+++ b/sdks/go/examples/xlang/bigquery/wordcount.go
@@ -76,8 +76,8 @@
// container repository, such as Docker Hub.
//
// export DOCKER_ROOT="Your Docker Repository Root"
-// ./gradlew :sdks:java:container:java8:docker
-Pdocker-repository-root=$DOCKER_ROOT -Pdocker-tag=latest
-// docker push $DOCKER_ROOT/beam_java8_sdk:latest
+// ./gradlew :sdks:java:container:java11:docker
-Pdocker-repository-root=$DOCKER_ROOT -Pdocker-tag=latest
+// docker push $DOCKER_ROOT/beam_java11_sdk:latest
//
// For runners in local mode, simply building the container using the default
values for
// docker-repository-root and docker-tag will work to have it accessible
locally.
@@ -86,7 +86,7 @@
// --sdk_harness_container_image_override flag for Java, or
--environment_config flag for Go. For
// example:
//
-//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java8_sdk:latest"
\
+//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java11_sdk:latest"
\
// --environment_config=${DOCKER_ROOT}/beam_go_sdk:latest
package main
diff --git a/sdks/go/test/build.gradle b/sdks/go/test/build.gradle
index 5b39cf81400..7a4c0a63aa4 100644
--- a/sdks/go/test/build.gradle
+++ b/sdks/go/test/build.gradle
@@ -69,7 +69,7 @@ task flinkValidatesRunner {
dependsOn ":sdks:go:test:goBuild"
dependsOn ":sdks:go:container:docker"
- dependsOn ":sdks:java:container:java8:docker"
+ dependsOn ":sdks:java:container:${project.ext.currentJavaVersion}:docker"
dependsOn
":runners:flink:${project.ext.latestFlinkVersion}:job-server:shadowJar"
dependsOn ":sdks:java:testing:expansion-service:buildTestExpansionServiceJar"
doLast {
@@ -93,7 +93,7 @@ task flinkValidatesRunner {
task samzaValidatesRunner {
(beam) branch master updated: [#31926] [java] call provision service when creating external workers. (#32198)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new cbe2b9e2dde [#31926] [java] call provision service when creating
external workers. (#32198)
cbe2b9e2dde is described below
commit cbe2b9e2dded772c3c114f1c1994f31301849ceb
Author: Robert Burke
AuthorDate: Thu Aug 15 15:59:21 2024 -0700
[#31926] [java] call provision service when creating external workers.
(#32198)
* [#31926] [java] call provision service when creating external workers.
* spotless apply
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
.../beam/fn/harness/ExternalWorkerService.java | 44 +++---
1 file changed, 38 insertions(+), 6 deletions(-)
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/ExternalWorkerService.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/ExternalWorkerService.java
index 8ef3f96b00b..dc42f7e1c78 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/ExternalWorkerService.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/ExternalWorkerService.java
@@ -20,12 +20,17 @@ package org.apache.beam.fn.harness;
import static org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull;
import java.util.Collections;
+import java.util.Set;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StopWorkerRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StopWorkerResponse;
import
org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc.BeamFnExternalWorkerPoolImplBase;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi;
+import org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.sdk.fn.channel.AddHarnessIdInterceptor;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
import org.apache.beam.sdk.fn.server.FnService;
import org.apache.beam.sdk.fn.server.GrpcFnServer;
import org.apache.beam.sdk.fn.server.ServerFactory;
@@ -35,6 +40,8 @@ import org.apache.beam.sdk.util.Sleeper;
import org.apache.beam.sdk.util.construction.Environments;
import org.apache.beam.sdk.util.construction.PipelineOptionsTranslation;
import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.stub.StreamObserver;
+import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -61,17 +68,42 @@ public class ExternalWorkerService extends
BeamFnExternalWorkerPoolImplBase impl
request.getWorkerId(),
request.getControlEndpoint().getUrl());
LOG.debug("Worker request {}.", request);
+
+Endpoints.ApiServiceDescriptor loggingEndpoint =
request.getLoggingEndpoint();
+Endpoints.ApiServiceDescriptor controlEndpoint =
request.getControlEndpoint();
+Set runnerCapabilites = Collections.emptySet();
+if (request.hasProvisionEndpoint()) {
+ ManagedChannelFactory channelFactory =
+ ManagedChannelFactory.createDefault()
+ .withInterceptors(
+
ImmutableList.of(AddHarnessIdInterceptor.create(request.getWorkerId(;
+
+ ProvisionServiceGrpc.ProvisionServiceBlockingStub provisionStub =
+ ProvisionServiceGrpc.newBlockingStub(
+ channelFactory.forDescriptor(request.getProvisionEndpoint()));
+ ProvisionApi.ProvisionInfo provisionInfo =
+ provisionStub
+
.getProvisionInfo(ProvisionApi.GetProvisionInfoRequest.newBuilder().build())
+ .getInfo();
+
+ runnerCapabilites =
Sets.newHashSet(provisionInfo.getRunnerCapabilitiesList());
+ if (provisionInfo.hasControlEndpoint()) {
+controlEndpoint = provisionInfo.getControlEndpoint();
+ }
+ if (provisionInfo.hasLoggingEndpoint()) {
+loggingEndpoint = provisionInfo.getLoggingEndpoint();
+ }
+}
+// Lambda closured variables must be final.
+final Endpoints.ApiServiceDescriptor logEndpoint = loggingEndpoint;
+final Endpoints.ApiServiceDescriptor ctrlEndpoint = controlEndpoint;
+final Set capabilities = runnerCapabilites;
Thread th =
new Thread(
() -> {
try {
FnHarness.main(
-request.getWorkerId(),
-options,
-Collections.emptySet(),
-request.getLoggingEndpoint(),
-request.getControlEndpoint(),
-null);
+
(beam) branch master updated: [prism] Catch panics in primary execution goroutines. (#32210)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 92087f2daa7 [prism] Catch panics in primary execution goroutines.
(#32210)
92087f2daa7 is described below
commit 92087f2daa7d1fdfa041c99b9047cdd195a46924
Author: Robert Burke
AuthorDate: Thu Aug 15 15:13:56 2024 -0700
[prism] Catch panics in primary execution goroutines. (#32210)
---
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go | 6 ++
sdks/go/pkg/beam/runners/prism/internal/stage.go | 8 +++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
b/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
index 0781efd5ff0..ed9c0ddc0f8 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
@@ -349,6 +349,12 @@ func (em *ElementManager) Bundles(ctx context.Context,
upstreamCancelFn context.
}()
// Watermark evaluation goroutine.
go func() {
+ defer func() {
+ // In case of panics in bundle generation, fail and
cancel the job.
+ if e := recover(); e != nil {
+ upstreamCancelFn(fmt.Errorf("panic in
ElementManager.Bundles watermark evaluation goroutine: %v", e))
+ }
+ }()
defer close(runStageCh)
// If we have a test stream, clear out existing refreshes, so
the test stream can
diff --git a/sdks/go/pkg/beam/runners/prism/internal/stage.go
b/sdks/go/pkg/beam/runners/prism/internal/stage.go
index 1a62f2f6f42..3d1e506f5e3 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/stage.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/stage.go
@@ -76,7 +76,13 @@ type stage struct {
OutputsToCoders map[string]engine.PColInfo
}
-func (s *stage) Execute(ctx context.Context, j *jobservices.Job, wk *worker.W,
comps *pipepb.Components, em *engine.ElementManager, rb engine.RunBundle) error
{
+func (s *stage) Execute(ctx context.Context, j *jobservices.Job, wk *worker.W,
comps *pipepb.Components, em *engine.ElementManager, rb engine.RunBundle) (err
error) {
+ defer func() {
+ // Convert execution panics to errors to fail the bundle.
+ if e := recover(); e != nil {
+ err = fmt.Errorf("panic in stage.Execute bundle
processing goroutine: %v, stage: %+v", e, s)
+ }
+ }()
slog.Debug("Execute: starting bundle", "bundle", rb)
var b *worker.B
(beam) branch master updated: [#32121] Support timers in interval windows. (#32180)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c197e4ffc1a [#32121] Support timers in interval windows. (#32180)
c197e4ffc1a is described below
commit c197e4ffc1a754228731093b21be6c38a608b752
Author: Robert Burke
AuthorDate: Thu Aug 15 09:34:33 2024 -0700
[#32121] Support timers in interval windows. (#32180)
* [#32121] Support timers in interval windows.
* fix test
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
sdks/go/pkg/beam/runners/prism/internal/coders.go | 12 +++-
.../pkg/beam/runners/prism/internal/coders_test.go | 16 +++--
.../prism/internal/engine/elementmanager.go| 34 +--
.../beam/runners/prism/internal/engine/timers.go | 68 ++
sdks/go/pkg/beam/runners/prism/internal/execute.go | 39 +++--
.../beam/runners/prism/internal/handlerunner.go| 2 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 26 +
7 files changed, 142 insertions(+), 55 deletions(-)
diff --git a/sdks/go/pkg/beam/runners/prism/internal/coders.go
b/sdks/go/pkg/beam/runners/prism/internal/coders.go
index 6fdaf804a34..eb8abe16ecf 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/coders.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/coders.go
@@ -26,6 +26,7 @@ import (
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/exec"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/ioutilx"
pipepb "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1"
+
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal/engine"
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal/urns"
"golang.org/x/exp/slog"
"google.golang.org/protobuf/encoding/prototext"
@@ -80,19 +81,24 @@ func makeWindowedValueCoder(pID string, comps
*pipepb.Components, coders map[str
return wvcID, nil
}
-// makeWindowCoders makes the coder pair but behavior is ultimately determined
by the strategy's windowFn.
-func makeWindowCoders(wc *pipepb.Coder) (exec.WindowDecoder,
exec.WindowEncoder) {
+// makeWindowCoders categorizes and provides the encoder, decoder pair for the
type of window.
+func makeWindowCoders(wc *pipepb.Coder) (engine.WinCoderType,
exec.WindowDecoder, exec.WindowEncoder) {
var cwc *coder.WindowCoder
+ var winCoder engine.WinCoderType
switch wc.GetSpec().GetUrn() {
case urns.CoderGlobalWindow:
+ winCoder = engine.WinGlobal
cwc = coder.NewGlobalWindow()
case urns.CoderIntervalWindow:
+ winCoder = engine.WinInterval
cwc = coder.NewIntervalWindow()
default:
+ // TODO(https://github.com/apache/beam/issues/31921): Support
custom windowfns instead of panicking here.
+ winCoder = engine.WinCustom
slog.LogAttrs(context.TODO(), slog.LevelError,
"makeWindowCoders: unknown urn", slog.String("urn", wc.GetSpec().GetUrn()))
panic(fmt.Sprintf("makeWindowCoders, unknown urn: %v",
prototext.Format(wc)))
}
- return exec.MakeWindowDecoder(cwc), exec.MakeWindowEncoder(cwc)
+ return winCoder, exec.MakeWindowDecoder(cwc),
exec.MakeWindowEncoder(cwc)
}
// lpUnknownCoders takes a coder, and populates coders with any new coders
diff --git a/sdks/go/pkg/beam/runners/prism/internal/coders_test.go
b/sdks/go/pkg/beam/runners/prism/internal/coders_test.go
index 3f9557ff836..4656a94e03e 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/coders_test.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/coders_test.go
@@ -25,6 +25,7 @@ import (
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/window"
"github.com/apache/beam/sdks/v2/go/pkg/beam/core/typex"
pipepb "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1"
+
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal/engine"
"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal/urns"
"github.com/google/go-cmp/cmp"
"google.golang.org/protobuf/testing/protocmp"
@@ -91,14 +92,15 @@ func Test_makeWindowedValueCoder(t *testing.T) {
func Test_makeWindowCoders(t *testing.T) {
tests := []struct {
- urnstring
- window typex.Window
+ urn string
+ windowtypex.Window
+ coderType engine.WinCoderType
}{
- {urns.CoderGlobalWindow, window.GlobalWindow{}},
+ {urns.CoderGlobalWindow, window.GlobalWindow{},
engi
(beam) branch master updated: [prism] Skip python tests that require an expansion service at this time. (#32182)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f1f5f775252 [prism] Skip python tests that require an expansion
service at this time. (#32182)
f1f5f775252 is described below
commit f1f5f775252a50929a96388b91e4497051aaec91
Author: Robert Burke
AuthorDate: Wed Aug 14 15:39:29 2024 -0700
[prism] Skip python tests that require an expansion service at this time.
(#32182)
* [prism] Skip tests that require an expansion service at this time.
* Remove extra python imports.
-
Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com>
---
.../runners/portability/prism_runner_test.py | 77 ++
1 file changed, 4 insertions(+), 73 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/prism_runner_test.py
b/sdks/python/apache_beam/runners/portability/prism_runner_test.py
index b179156877e..387b7ba2bec 100644
--- a/sdks/python/apache_beam/runners/portability/prism_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/prism_runner_test.py
@@ -30,11 +30,6 @@ from tempfile import mkdtemp
import pytest
import apache_beam as beam
-from apache_beam import Impulse
-from apache_beam import Map
-from apache_beam.io.external.generate_sequence import GenerateSequence
-from apache_beam.io.kafka import ReadFromKafka
-from apache_beam.io.kafka import WriteToKafka
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import PortableOptions
from apache_beam.runners.portability import portable_runner_test
@@ -42,7 +37,6 @@ from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import userstate
from apache_beam.transforms import window
-from apache_beam.transforms.sql import SqlTransform
from apache_beam.utils import timestamp
# Run as
@@ -240,79 +234,16 @@ class
PrismRunnerTest(portable_runner_test.PortableRunnerTest):
assert_that(lines, lambda lines: len(lines) > 0)
def test_external_transform(self):
-with self.create_pipeline() as p:
- res = (
- p
- | GenerateSequence(
- start=1, stop=10,
expansion_service=self.get_expansion_service()))
-
- assert_that(res, equal_to([i for i in range(1, 10)]))
+raise unittest.SkipTest("Requires an expansion service to execute.")
def test_expand_kafka_read(self):
-# We expect to fail here because we do not have a Kafka cluster handy.
-# Nevertheless, we check that the transform is expanded by the
-# ExpansionService and that the pipeline fails during execution.
-with self.assertRaises(Exception) as ctx:
- self.enable_commit = True
- with self.create_pipeline() as p:
-# pylint: disable=expression-not-assigned
-(
-p
-| ReadFromKafka(
-consumer_config={
-'bootstrap.servers': 'notvalid1:, notvalid2:3531',
-'group.id': 'any_group'
-},
-topics=['topic1', 'topic2'],
-key_deserializer='org.apache.kafka.'
-'common.serialization.'
-'ByteArrayDeserializer',
-value_deserializer='org.apache.kafka.'
-'common.serialization.'
-'LongDeserializer',
-commit_offset_in_finalize=True,
-timestamp_policy=ReadFromKafka.create_time_policy,
-expansion_service=self.get_expansion_service()))
-self.assertTrue(
-'No resolvable bootstrap urls given in bootstrap.servers' in str(
-ctx.exception),
-'Expected to fail due to invalid bootstrap.servers, but '
-'failed due to:\n%s' % str(ctx.exception))
+raise unittest.SkipTest("Requires an expansion service to execute.")
def test_expand_kafka_write(self):
-# We just test the expansion but do not execute.
-# pylint: disable=expression-not-assigned
-(
-self.create_pipeline()
-| Impulse()
-| Map(lambda input: (1, input))
-| WriteToKafka(
-producer_config={
-'bootstrap.servers': 'localhost:9092, notvalid2:3531'
-},
-topic='topic1',
-key_serializer='org.apache.kafka.'
-'common.serialization.'
-'LongSerializer',
-value_serializer='org.apache.kafka.'
-'common.serialization.'
-'ByteArraySerializer',
-
(beam) branch master updated: Bump github.com/docker/docker in /sdks (#32176)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8ff7f0d75e4 Bump github.com/docker/docker in /sdks (#32176) 8ff7f0d75e4 is described below commit 8ff7f0d75e45aa31bcc56d2bcd38ef49125295aa Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 13 23:12:52 2024 -0700 Bump github.com/docker/docker in /sdks (#32176) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 27.1.1+incompatible to 27.1.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](https://github.com/docker/docker/compare/v27.1.1...v27.1.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 3 ++- sdks/go.sum | 6 -- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 7c3ade37490..53aa6f23bfa 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -93,6 +93,7 @@ require ( github.com/minio/highwayhash v1.0.3 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect github.com/moby/sys/user v0.1.0 // indirect + github.com/moby/sys/userns v0.1.0 // indirect github.com/nats-io/jwt/v2 v2.5.8 // indirect github.com/nats-io/nkeys v0.4.7 // indirect github.com/nats-io/nuid v1.0.1 // indirect @@ -142,7 +143,7 @@ require ( github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect github.com/containerd/containerd v1.7.18 // indirect github.com/cpuguy83/dockercfg v0.3.1 // indirect - github.com/docker/docker v27.1.1+incompatible // but required to resolve issue docker has with go1.20 + github.com/docker/docker v27.1.2+incompatible // but required to resolve issue docker has with go1.20 github.com/docker/go-units v0.5.0 // indirect github.com/envoyproxy/go-control-plane v0.12.0 // indirect github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 5b9bfaef8a4..f49007e6b7d 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -779,8 +779,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= -github.com/docker/docker v27.1.1+incompatible h1:hO/M4MtV36kzKldqnA37IWhebRA+LnqqcqDja6kVaKY= -github.com/docker/docker v27.1.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v27.1.2+incompatible h1:AhGzR1xaQIy53qCkxARaFluI00WPGtXn0AJuoQsVYTY= +github.com/docker/docker v27.1.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c= github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= @@ -1065,6 +1065,8 @@ github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5 github.com/moby/sys/sequential v0.5.0/go.mod h1:tH2cOOs5V9MlPiXcQzRC+eEyab644PWKGRYaaV5ZZlo= github.com/moby/sys/user v0.1.0 h1:WmZ93f5Ux6het5iituh9x2zAG7NFY9Aqi49jjE1PaQg= github.com/moby/sys/user v0.1.0/go.mod h1:fKJhFOnsCN6xZ5gSfbM6zaHGgDJMrqt9/reuj4T7MmU= +github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g= +github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28= github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
(beam) branch dependabot/go_modules/sdks/golang.org/x/sys-0.24.0 deleted (was e02e6cc80ea)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/golang.org/x/sys-0.24.0 in repository https://gitbox.apache.org/repos/asf/beam.git was e02e6cc80ea Bump golang.org/x/sys from 0.23.0 to 0.24.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump golang.org/x/sys from 0.23.0 to 0.24.0 in /sdks (#32150)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new edf4c4f5f19 Bump golang.org/x/sys from 0.23.0 to 0.24.0 in /sdks (#32150) edf4c4f5f19 is described below commit edf4c4f5f19ef6ffd25493262261c713ba045980 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Aug 12 10:20:55 2024 -0700 Bump golang.org/x/sys from 0.23.0 to 0.24.0 in /sdks (#32150) Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.23.0 to 0.24.0. - [Commits](https://github.com/golang/sys/compare/v0.23.0...v0.24.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 5d179736baf..7c3ade37490 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -56,7 +56,7 @@ require ( golang.org/x/net v0.28.0 golang.org/x/oauth2 v0.22.0 golang.org/x/sync v0.8.0 - golang.org/x/sys v0.23.0 + golang.org/x/sys v0.24.0 golang.org/x/text v0.17.0 google.golang.org/api v0.191.0 google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf diff --git a/sdks/go.sum b/sdks/go.sum index 2b43c6d62bc..5b9bfaef8a4 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1507,8 +1507,8 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM= -golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.24.0 h1:Twjiwq9dn6R1fQcyiK+wQyHWfaz/BJB+YIpzU/Cv3Xg= +golang.org/x/sys v0.24.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
(beam) branch master updated: [#32115] Fix timer support, support timer clears. (#32119)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 17298b5572e [#32115] Fix timer support, support timer clears. (#32119)
17298b5572e is described below
commit 17298b5572e9b0d8aa8c4d0ca1e51c3f832c0067
Author: Robert Burke
AuthorDate: Fri Aug 9 14:21:32 2024 -0700
[#32115] Fix timer support, support timer clears. (#32119)
---
.../prism/internal/engine/elementmanager.go| 22 ++--
.../beam/runners/prism/internal/engine/timers.go | 115 +
.../runners/portability/prism_runner_test.py | 32 ++
3 files changed, 119 insertions(+), 50 deletions(-)
diff --git a/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
b/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
index bc8449c72b3..c73db507c79 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
@@ -869,14 +869,20 @@ func (em *ElementManager) triageTimers(d TentativeData,
inputInfo PColInfo, stag
for tentativeKey, timers := range d.timers {
keyToTimers := map[timerKey]element{}
for _, t := range timers {
- key, tag, elms := decodeTimer(inputInfo.KeyDec, true, t)
- for _, e := range elms {
- keyToTimers[timerKey{key: string(key), tag:
tag, win: e.window}] = e
- }
- if len(elms) == 0 {
- // TODO(lostluck): Determine best way to mark a
timer cleared.
- continue
- }
+ // TODO: Call in a for:range loop when Beam's minimum
Go version hits 1.23.0
+ iter := decodeTimerIter(inputInfo.KeyDec, true, t)
+ iter(func(ret timerRet) bool {
+ for _, e := range ret.elms {
+ keyToTimers[timerKey{key:
string(ret.keyBytes), tag: ret.tag, win: e.window}] = e
+ }
+ if len(ret.elms) == 0 {
+ for _, w := range ret.windows {
+ delete(keyToTimers,
timerKey{key: string(ret.keyBytes), tag: ret.tag, win: w})
+ }
+ }
+ // Indicate we'd like to continue iterating.
+ return true
+ })
}
for _, elm := range keyToTimers {
diff --git a/sdks/go/pkg/beam/runners/prism/internal/engine/timers.go
b/sdks/go/pkg/beam/runners/prism/internal/engine/timers.go
index 787d27858a0..9a3bd6f9682 100644
--- a/sdks/go/pkg/beam/runners/prism/internal/engine/timers.go
+++ b/sdks/go/pkg/beam/runners/prism/internal/engine/timers.go
@@ -31,53 +31,77 @@ import (
"google.golang.org/protobuf/encoding/protowire"
)
-// DecodeTimer extracts timers to elements for insertion into their keyed
queues.
-// Returns the key bytes, tag, window exploded elements, and the hold
timestamp.
+type timerRet struct {
+ keyBytes []byte
+ tag string
+ elms []element
+ windows []typex.Window
+}
+
+// decodeTimerIter extracts timers to elements for insertion into their keyed
queues,
+// through a go iterator function, to be called by the caller with their
processing function.
+//
+// For each timer, a key, tag, windowed elements, and the window set are
returned.
+//
// If the timer has been cleared, no elements will be returned. Any existing
timers
-// for the tag *must* be cleared from the pending queue.
-func decodeTimer(keyDec func(io.Reader) []byte, usesGlobalWindow bool, raw
[]byte) ([]byte, string, []element) {
- keyBytes := keyDec(bytes.NewBuffer(raw))
-
- d := decoder{raw: raw, cursor: len(keyBytes)}
- tag := string(d.Bytes())
-
- var ws []typex.Window
- numWin := d.Fixed32()
- if usesGlobalWindow {
- for i := 0; i < int(numWin); i++ {
- ws = append(ws, window.GlobalWindow{})
- }
- } else {
- // Assume interval windows here, since we don't understand
custom windows yet.
- for i := 0; i < int(numWin); i++ {
- ws = append(ws, d.IntervalWindow())
- }
- }
+// for the tag *must* be cleared from the pending queue. The windows
associated with
+// the clear are provided to be able to delete pending timers.
+func decodeTimerIter(keyDec func(io.Reader) []byte, usesGlobalWindow bool, raw
[]byte) func(func(timerRet) bool) {
+ return
(beam) branch master updated: Bump cloud.google.com/go/profiler from 0.4.0 to 0.4.1 in /sdks (#32125)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 27741fb6c9b Bump cloud.google.com/go/profiler from 0.4.0 to 0.4.1 in /sdks (#32125) 27741fb6c9b is described below commit 27741fb6c9b6fc109a94bb91fa4a665473f50268 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Aug 9 05:11:41 2024 -0700 Bump cloud.google.com/go/profiler from 0.4.0 to 0.4.1 in /sdks (#32125) Bumps [cloud.google.com/go/profiler](https://github.com/googleapis/google-cloud-go) from 0.4.0 to 0.4.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/v0.4.0...ai/v0.4.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/profiler dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 4 ++-- sdks/go.sum | 8 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 8849f9732af..8c7a52ca951 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -26,7 +26,7 @@ require ( cloud.google.com/go/bigquery v1.62.0 cloud.google.com/go/bigtable v1.28.0 cloud.google.com/go/datastore v1.17.1 - cloud.google.com/go/profiler v0.4.0 + cloud.google.com/go/profiler v0.4.1 cloud.google.com/go/pubsub v1.40.0 cloud.google.com/go/spanner v1.66.0 cloud.google.com/go/storage v1.43.0 @@ -152,7 +152,7 @@ require ( github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/snappy v0.0.4 // indirect github.com/google/flatbuffers v23.5.26+incompatible // indirect - github.com/google/pprof v0.0.0-20230602150820-91b7bce49751 // indirect + github.com/google/pprof v0.0.0-20240528025155-186aa0362fba // indirect github.com/google/renameio/v2 v2.0.0 // indirect github.com/google/s2a-go v0.1.7 // indirect github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 90fb958b1df..e0039f28e43 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -439,8 +439,8 @@ cloud.google.com/go/privatecatalog v0.5.0/go.mod h1:XgosMUvvPyxDjAVNDYxJ7wBW8//h cloud.google.com/go/privatecatalog v0.6.0/go.mod h1:i/fbkZR0hLN29eEWiiwue8Pb+GforiEIBnV9yrRUOKI= cloud.google.com/go/privatecatalog v0.7.0/go.mod h1:2s5ssIFO69F5csTXcwBP7NPFTZvps26xGzvQ2PQaBYg= cloud.google.com/go/privatecatalog v0.8.0/go.mod h1:nQ6pfaegeDAq/Q5lrfCQzQLhubPiZhSaNhIgfJlnIXs= -cloud.google.com/go/profiler v0.4.0 h1:ZeRDZbsOBDyRG0OiK0Op1/XWZ3xeLwJc9zjkzczUxyY= -cloud.google.com/go/profiler v0.4.0/go.mod h1:RvPlm4dilIr3oJtAOeFQU9Lrt5RoySHSDj4pTd6TWeU= +cloud.google.com/go/profiler v0.4.1 h1:Q7+lOvikTGMJ/IAWocpYYGit4SIIoILmVZfEEWTORSY= +cloud.google.com/go/profiler v0.4.1/go.mod h1:LBrtEX6nbvhv1w/e5CPZmX9ajGG9BGLtGbv56Tg4SHs= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= @@ -938,8 +938,8 @@ github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20230602150820-91b7bce49751 h1:hR7/MlvK23p6+lIw9SN1TigNLn9ZnF3W4SYRKq2gAHs= -github.com/google/pprof v0.0.0-20230602150820-91b7bce49751/go.mod h1:Jh3hGz2jkYak8qXPD19ryItVnUgpgeqzdkY/D0EaeuA= +github.com/google/pprof v0.0.0-20240528025155-186aa0362fba h1:ql1qNgCyOB7iAEk8JTNM+zJrgIbnyCKX/wdlyPufP5g= +github.com/google/pprof v0.0.0-20240528025155-186aa0362fba/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/renameio/v2 v2.0.0 h1:UifI23ZTGY8Tt29JbYFiuyIU3eX+RNFtUwefq9qAhxg= github.com/google/renameio/v2 v2.0.0/go.mod h1:BtmJXm5YlszgC+TD4HOEEUFgkJP3nLxehU6hfe7jRt4=
(beam) branch dependabot/go_modules/sdks/cloud.google.com/go/profiler-0.4.1 deleted (was 4926338ce3f)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/profiler-0.4.1 in repository https://gitbox.apache.org/repos/asf/beam.git was 4926338ce3f Bump cloud.google.com/go/profiler from 0.4.0 to 0.4.1 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump golang.org/x/net from 0.27.0 to 0.28.0 in /sdks (#32128)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new af6bf8a1423 Bump golang.org/x/net from 0.27.0 to 0.28.0 in /sdks (#32128) af6bf8a1423 is described below commit af6bf8a1423eef082a289e9a1cfcfe0bf2d8c0ed Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Thu Aug 8 23:11:22 2024 -0700 Bump golang.org/x/net from 0.27.0 to 0.28.0 in /sdks (#32128) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.27.0 to 0.28.0. - [Commits](https://github.com/golang/net/compare/v0.27.0...v0.28.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 6 +++--- sdks/go.sum | 16 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 1f53410c5e0..8849f9732af 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -53,10 +53,10 @@ require ( github.com/xitongsys/parquet-go v1.6.2 github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.16.1 - golang.org/x/net v0.27.0 + golang.org/x/net v0.28.0 golang.org/x/oauth2 v0.21.0 golang.org/x/sync v0.8.0 - golang.org/x/sys v0.22.0 + golang.org/x/sys v0.23.0 golang.org/x/text v0.17.0 google.golang.org/api v0.189.0 google.golang.org/genproto v0.0.0-20240725223205-93522f1f2a9f @@ -183,7 +183,7 @@ require ( github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect github.com/zeebo/xxh3 v1.0.2 // indirect go.opencensus.io v0.24.0 // indirect - golang.org/x/crypto v0.25.0 // indirect + golang.org/x/crypto v0.26.0 // indirect golang.org/x/mod v0.18.0 // indirect golang.org/x/tools v0.22.0 // indirect golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 8a5072a7440..90fb958b1df 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1255,8 +1255,8 @@ golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30= -golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M= +golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw= +golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54= golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -1377,8 +1377,8 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= -golang.org/x/net v0.27.0 h1:5K3Njcw06/l2y9vpGCSdcxWOYHOUk3dVNGDXN+FvAys= -golang.org/x/net v0.27.0/go.mod h1:dDi0PyhWNoiUOrAS8uXv/vnScO4wnHQO4mj9fn/RytE= +golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE= +golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -1515,8 +1515,8 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI= -golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTL
(beam) branch master updated: Bump cloud.google.com/go/spanner from 1.64.0 to 1.66.0 in /sdks (#32126)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6bdf63a9d2d Bump cloud.google.com/go/spanner from 1.64.0 to 1.66.0 in /sdks (#32126) 6bdf63a9d2d is described below commit 6bdf63a9d2deb00d6b33704c8a8416cf6783f4b6 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Thu Aug 8 22:44:50 2024 -0700 Bump cloud.google.com/go/spanner from 1.64.0 to 1.66.0 in /sdks (#32126) Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.64.0 to 1.66.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.64.0...spanner/v1.66.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/spanner dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 18 ++ 2 files changed, 3 insertions(+), 17 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index bd25b3beab5..1f53410c5e0 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -28,7 +28,7 @@ require ( cloud.google.com/go/datastore v1.17.1 cloud.google.com/go/profiler v0.4.0 cloud.google.com/go/pubsub v1.40.0 - cloud.google.com/go/spanner v1.64.0 + cloud.google.com/go/spanner v1.66.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.30.3 github.com/aws/aws-sdk-go-v2/config v1.27.27 diff --git a/sdks/go.sum b/sdks/go.sum index a50c8ce9230..8a5072a7440 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -540,8 +540,8 @@ cloud.google.com/go/shell v1.6.0/go.mod h1:oHO8QACS90luWgxP3N9iZVuEiSF84zNyLytb+ cloud.google.com/go/spanner v1.41.0/go.mod h1:MLYDBJR/dY4Wt7ZaMIQ7rXOTLjYrmxLE/5ve9vFfWos= cloud.google.com/go/spanner v1.44.0/go.mod h1:G8XIgYdOK+Fbcpbs7p2fiprDw4CaZX63whnSMLVBxjk= cloud.google.com/go/spanner v1.45.0/go.mod h1:FIws5LowYz8YAE1J8fOS7DJup8ff7xJeetWEo5REA2M= -cloud.google.com/go/spanner v1.64.0 h1:ltyPbHA/nRAtAhU/o742dXBCI1eNHPeaRY09Ja8B+hM= -cloud.google.com/go/spanner v1.64.0/go.mod h1:TOFx3pb2UwPsDGlE1gTehW+y6YlU4IFk+VdDHSGQS/M= +cloud.google.com/go/spanner v1.66.0 h1:PF1upR8n+DVUO9mUpCc1j5kyHn1Xfq0A53ZrnM0AmeU= +cloud.google.com/go/spanner v1.66.0/go.mod h1:gu+weqqrnoBsVlxOmMG5pzDZ2nkpqqJx4MsnmIacH5w= cloud.google.com/go/speech v1.6.0/go.mod h1:79tcr4FHCimOp56lwC01xnt/WPJZc4v3gzyT7FoBkCM= cloud.google.com/go/speech v1.7.0/go.mod h1:KptqL+BAQIhMsj1kOP2la5DSEEerPDuOP/2mmkhHhZQ= cloud.google.com/go/speech v1.8.0/go.mod h1:9bYIl1/tjsAnMgKGHKmBZzXKEkGgtU+MpdDPTE9f7y0= @@ -681,13 +681,9 @@ github.com/aws/aws-sdk-go v1.34.0/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU github.com/aws/aws-sdk-go-v2 v1.7.1/go.mod h1:L5LuPC1ZgDr2xQS7AmIec/Jlc7O/Y1u2KxJyNVab250= github.com/aws/aws-sdk-go-v2 v1.30.3 h1:jUeBtG0Ih+ZIFH0F4UkmL9w3cSpaMv9tYYDbzILP8dY= github.com/aws/aws-sdk-go-v2 v1.30.3/go.mod h1:nIQjQVp5sfpQcTc9mPSr1B0PaWK5ByX9MOoDadSN4lc= -github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 h1:ZY3108YtBNq96jNZTICHxN1gSBSbnvIdYwwqnvCV4Mc= -github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1/go.mod h1:t8PYl/6LzdAqsU4/9tz28V/kU+asFePvpOMkdul0gEQ= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 h1:tW1/Rkad38LA15X4UQtjXZXNKsCgkshC3EbmcUmghTg= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3/go.mod h1:UbnqO+zjqk3uIt9yCACHJ9IVNhyhOCnYk8yA19SAWrM= github.com/aws/aws-sdk-go-v2/config v1.5.0/go.mod h1:RWlPOAW3E3tbtNAqTwvSW54Of/yP3oiZXMI0xfUdjyA= -github.com/aws/aws-sdk-go-v2/config v1.27.4 h1:AhfWb5ZwimdsYTgP7Od8E9L1u4sKmDW2ZVeLcf2O42M= -github.com/aws/aws-sdk-go-v2/config v1.27.4/go.mod h1:zq2FFXK3A416kiukwpsd+rD4ny6JC7QSkp4QdN1Mp2g= github.com/aws/aws-sdk-go-v2/config v1.27.27 h1:HdqgGt1OAP0HkEDDShEl0oSYa9ZZBSOmKpdpsDMdO90= github.com/aws/aws-sdk-go-v2/config v1.27.27/go.mod h1:MVYamCg76dFNINkZFu4n4RjDixhVr51HLj4ErWzrVwg= github.com/aws/aws-sdk-go-v2/credentials v1.3.1/go.mod h1:r0n73xwsIVagq8RsxmZbGSRQFj9As3je72C2WzUIToc= @@ -697,8 +693,6 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.3.0/go.mod h1:2LAuqPx1I6jNfaGDu github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 h1:KreluoV8FZDEtI6Co2xuNk/UqI9iwMrOx/87PBNIKqw= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11/go.mod h1:SeSUYBLsMYFoRvHE0Tjvn7kbxaUhl75CJi1sbfhMxkU= github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.3.2/go.mod h1:qaqQiHSrOUVOfKe6fhgQ6UzhxjwqVW8aHNegd6Ws4w4= -github.com/aws/aws-sdk-go-v2/feature/s3/manager
(beam) branch master updated: [Go SDK] s3 filesystem: Fix nillable content length, update deps. (#32111)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a6de47572b9 [Go SDK] s3 filesystem: Fix nillable content length, update deps. (#32111) a6de47572b9 is described below commit a6de47572b927c8a1c3fdaf11f15b6d02473c3e8 Author: Robert Burke AuthorDate: Thu Aug 8 10:08:21 2024 -0700 [Go SDK] s3 filesystem: Fix nillable content length, update deps. (#32111) Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com> --- sdks/go.mod | 14 +++--- sdks/go.sum | 14 ++ sdks/go/pkg/beam/io/filesystem/s3/s3.go | 10 +++--- 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 26c1d9d3769..bd25b3beab5 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -31,10 +31,10 @@ require ( cloud.google.com/go/spanner v1.64.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.30.3 - github.com/aws/aws-sdk-go-v2/config v1.27.4 + github.com/aws/aws-sdk-go-v2/config v1.27.27 github.com/aws/aws-sdk-go-v2/credentials v1.17.27 - github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.13.8 - github.com/aws/aws-sdk-go-v2/service/s3 v1.42.2 + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.10 + github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3 github.com/aws/smithy-go v1.20.3 github.com/docker/go-connections v0.5.0 github.com/dustin/go-humanize v1.0.1 @@ -123,16 +123,16 @@ require ( github.com/apache/arrow/go/arrow v0.0.0-20200730104253-651201b0f516 // indirect github.com/apache/thrift v0.17.0 // indirect github.com/aws/aws-sdk-go v1.34.0 // indirect - github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect + github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 // indirect github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect - github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.3 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15 // indirect github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 // indirect - github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.3 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.17 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 // indirect - github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.3 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.15 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.22.4 // indirect github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 1f57de7507c..a50c8ce9230 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -683,9 +683,13 @@ github.com/aws/aws-sdk-go-v2 v1.30.3 h1:jUeBtG0Ih+ZIFH0F4UkmL9w3cSpaMv9tYYDbzILP github.com/aws/aws-sdk-go-v2 v1.30.3/go.mod h1:nIQjQVp5sfpQcTc9mPSr1B0PaWK5ByX9MOoDadSN4lc= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 h1:ZY3108YtBNq96jNZTICHxN1gSBSbnvIdYwwqnvCV4Mc= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1/go.mod h1:t8PYl/6LzdAqsU4/9tz28V/kU+asFePvpOMkdul0gEQ= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 h1:tW1/Rkad38LA15X4UQtjXZXNKsCgkshC3EbmcUmghTg= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3/go.mod h1:UbnqO+zjqk3uIt9yCACHJ9IVNhyhOCnYk8yA19SAWrM= github.com/aws/aws-sdk-go-v2/config v1.5.0/go.mod h1:RWlPOAW3E3tbtNAqTwvSW54Of/yP3oiZXMI0xfUdjyA= github.com/aws/aws-sdk-go-v2/config v1.27.4 h1:AhfWb5ZwimdsYTgP7Od8E9L1u4sKmDW2ZVeLcf2O42M= github.com/aws/aws-sdk-go-v2/config v1.27.4/go.mod h1:zq2FFXK3A416kiukwpsd+rD4ny6JC7QSkp4QdN1Mp2g= +github.com/aws/aws-sdk-go-v2/config v1.27.27 h1:HdqgGt1OAP0HkEDDShEl0oSYa9ZZBSOmKpdpsDMdO90= +github.com/aws/aws-sdk-go-v2/config v1.27.27/go.mod h1:MVYamCg76dFNINkZFu4n4RjDixhVr51HLj4ErWzrVwg= github.com/aws/aws-sdk-go-v2/credentials v1.3.1/go.mod h1:r0n73xwsIVagq8RsxmZbGSRQFj9As3je72C2WzUIToc= github.com/aws/aws-sdk-go-v2/credentials v1.17.27 h1:2raNba6gr2IfA0eqqiP2XiQ0UVOpGPgDSi0I9iAP+UI= github.com/aws/aws-sdk-go-v2/credentials v1.17.27/go.mod h1:gniiwbGahQByxan6YjQUMcW4Aov6bLC3m+evgcoN4r4= @@ -695,6 +699,8 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11/go.mod h1:SeSUYBLsMYFoRvH github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.3.2/go.m
(beam) branch dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.28.0 deleted (was d8054815ae4)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/bigtable-1.28.0 in repository https://gitbox.apache.org/repos/asf/beam.git was d8054815ae4 Bump cloud.google.com/go/bigtable from 1.25.0 to 1.28.0 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump cloud.google.com/go/bigtable from 1.25.0 to 1.28.0 in /sdks (#32105)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new adc3b2b4a5f Bump cloud.google.com/go/bigtable from 1.25.0 to 1.28.0 in /sdks (#32105) adc3b2b4a5f is described below commit adc3b2b4a5ffaeac50e73bce32027c85e8637cac Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Aug 7 22:37:13 2024 -0700 Bump cloud.google.com/go/bigtable from 1.25.0 to 1.28.0 in /sdks (#32105) Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.25.0 to 1.28.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.25.0...pubsub/v1.28.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigtable dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 38 +++- sdks/go.sum | 84 - 2 files changed, 64 insertions(+), 58 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 654a4562854..5aeb14606c9 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -23,12 +23,12 @@ module github.com/apache/beam/sdks/v2 go 1.21 require ( - cloud.google.com/go/bigquery v1.61.0 - cloud.google.com/go/bigtable v1.25.0 + cloud.google.com/go/bigquery v1.62.0 + cloud.google.com/go/bigtable v1.28.0 cloud.google.com/go/datastore v1.17.1 cloud.google.com/go/profiler v0.4.0 - cloud.google.com/go/pubsub v1.39.0 - cloud.google.com/go/spanner v1.63.0 + cloud.google.com/go/pubsub v1.40.0 + cloud.google.com/go/spanner v1.64.0 cloud.google.com/go/storage v1.43.0 github.com/aws/aws-sdk-go-v2 v1.30.3 github.com/aws/aws-sdk-go-v2/config v1.27.4 @@ -53,13 +53,13 @@ require ( github.com/xitongsys/parquet-go v1.6.2 github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c go.mongodb.org/mongo-driver v1.16.0 - golang.org/x/net v0.26.0 + golang.org/x/net v0.27.0 golang.org/x/oauth2 v0.21.0 golang.org/x/sync v0.8.0 golang.org/x/sys v0.22.0 golang.org/x/text v0.17.0 - google.golang.org/api v0.187.0 - google.golang.org/genproto v0.0.0-20240624140628-dc46fd24d27d + google.golang.org/api v0.189.0 + google.golang.org/genproto v0.0.0-20240725223205-93522f1f2a9f google.golang.org/grpc v1.65.0 google.golang.org/protobuf v1.34.2 gopkg.in/yaml.v2 v2.4.0 @@ -74,8 +74,9 @@ require ( require ( cel.dev/expr v0.15.0 // indirect - cloud.google.com/go/auth v0.6.1 // indirect - cloud.google.com/go/auth/oauth2adapt v0.2.2 // indirect + cloud.google.com/go/auth v0.7.2 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect + cloud.google.com/go/monitoring v1.20.2 // indirect dario.cat/mergo v1.0.0 // indirect filippo.io/edwards25519 v1.1.0 // indirect github.com/GoogleCloudPlatform/grpc-gcp-go/grpcgcp v1.5.0 // indirect @@ -84,7 +85,7 @@ require ( github.com/containerd/errdefs v0.1.0 // indirect github.com/containerd/log v0.1.0 // indirect github.com/distribution/reference v0.6.0 // indirect - github.com/go-logr/logr v1.4.1 // indirect + github.com/go-logr/logr v1.4.2 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-ole/go-ole v1.2.6 // indirect github.com/golang/protobuf v1.5.4 // indirect @@ -107,15 +108,16 @@ require ( go.opentelemetry.io/otel v1.24.0 // indirect go.opentelemetry.io/otel/metric v1.24.0 // indirect go.opentelemetry.io/otel/sdk v1.24.0 // indirect + go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect go.opentelemetry.io/otel/trace v1.24.0 // indirect golang.org/x/time v0.5.0 // indirect ) require ( cloud.google.com/go v0.115.0 // indirect - cloud.google.com/go/compute/metadata v0.3.0 // indirect - cloud.google.com/go/iam v1.1.8 // indirect - cloud.google.com/go/longrunning v0.5.7 // indirect + cloud.google.com/go/compute/metadata v0.5.0 // indirect + cloud.google.com/go/iam v1.1.11 // indirect + cloud.google.com/go/longrunning v0.5.10 // indirect github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/Microsoft/go-winio v0.6.2 // indirect github.com/apa
(beam) branch master updated (07e692b56fb -> 656a296a82d)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 07e692b56fb Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks (#31709) add 656a296a82d Enable Job management for the Prism runner (#32091) No new revisions were added by this update. Summary of changes: .../apache/beam/runners/prism/PrismJobManager.java | 160 .../beam/runners/prism/PrismJobManagerTest.java| 211 + 2 files changed, 371 insertions(+) create mode 100644 runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismJobManager.java create mode 100644 runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismJobManagerTest.java
svn commit: r70742 [1/2] - in /release/beam: 2.57.0/ 2.58.0/ 2.58.0/prism/ 2.58.0/prism/darwin/ 2.58.0/prism/darwin/amd64/ 2.58.0/prism/darwin/arm64/ 2.58.0/prism/linux/ 2.58.0/prism/linux/amd64/ 2.58
Author: lostluck Date: Wed Aug 7 19:26:59 2024 New Revision: 70742 Log: Adding artifacts for the 2.58.0 release and removing old artifacts Added: release/beam/2.58.0/ release/beam/2.58.0/apache-beam-2.58.0-source-release.zip (with props) release/beam/2.58.0/apache-beam-2.58.0-source-release.zip.asc release/beam/2.58.0/apache-beam-2.58.0-source-release.zip.sha512 release/beam/2.58.0/prism/ release/beam/2.58.0/prism/darwin/ release/beam/2.58.0/prism/darwin/amd64/ release/beam/2.58.0/prism/darwin/amd64/apache_beam-v2.58.0-prism-darwin-amd64.zip (with props) release/beam/2.58.0/prism/darwin/amd64/apache_beam-v2.58.0-prism-darwin-amd64.zip.asc release/beam/2.58.0/prism/darwin/amd64/apache_beam-v2.58.0-prism-darwin-amd64.zip.sha512 release/beam/2.58.0/prism/darwin/arm64/ release/beam/2.58.0/prism/darwin/arm64/apache_beam-v2.58.0-prism-darwin-arm64.zip (with props) release/beam/2.58.0/prism/darwin/arm64/apache_beam-v2.58.0-prism-darwin-arm64.zip.asc release/beam/2.58.0/prism/darwin/arm64/apache_beam-v2.58.0-prism-darwin-arm64.zip.sha512 release/beam/2.58.0/prism/linux/ release/beam/2.58.0/prism/linux/amd64/ release/beam/2.58.0/prism/linux/amd64/apache_beam-v2.58.0-prism-linux-amd64.zip (with props) release/beam/2.58.0/prism/linux/amd64/apache_beam-v2.58.0-prism-linux-amd64.zip.asc release/beam/2.58.0/prism/linux/amd64/apache_beam-v2.58.0-prism-linux-amd64.zip.sha512 release/beam/2.58.0/prism/linux/arm64/ release/beam/2.58.0/prism/linux/arm64/apache_beam-v2.58.0-prism-linux-arm64.zip (with props) release/beam/2.58.0/prism/linux/arm64/apache_beam-v2.58.0-prism-linux-arm64.zip.asc release/beam/2.58.0/prism/linux/arm64/apache_beam-v2.58.0-prism-linux-arm64.zip.sha512 release/beam/2.58.0/prism/windows/ release/beam/2.58.0/prism/windows/amd64/ release/beam/2.58.0/prism/windows/amd64/apache_beam-v2.58.0-prism-windows-amd64.zip (with props) release/beam/2.58.0/prism/windows/amd64/apache_beam-v2.58.0-prism-windows-amd64.zip.asc release/beam/2.58.0/prism/windows/amd64/apache_beam-v2.58.0-prism-windows-amd64.zip.sha512 release/beam/2.58.0/prism/windows/arm64/ release/beam/2.58.0/prism/windows/arm64/apache_beam-v2.58.0-prism-windows-arm64.zip (with props) release/beam/2.58.0/prism/windows/arm64/apache_beam-v2.58.0-prism-windows-arm64.zip.asc release/beam/2.58.0/prism/windows/arm64/apache_beam-v2.58.0-prism-windows-arm64.zip.sha512 release/beam/2.58.0/python/ release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-macosx_10_9_x86_64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-macosx_10_9_x86_64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-macosx_10_9_x86_64.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win32.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win32.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win32.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win_amd64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win_amd64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp310-cp310-win_amd64.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311-macosx_10_9_x86_64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311-macosx_10_9_x86_64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311-macosx_10_9_x86_64.whl.sha512 release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.58.0/python/apache_beam-2.58.0-cp311-cp311
svn commit: r70742 [2/2] - in /release/beam: 2.57.0/ 2.58.0/ 2.58.0/prism/ 2.58.0/prism/darwin/ 2.58.0/prism/darwin/amd64/ 2.58.0/prism/darwin/arm64/ 2.58.0/prism/linux/ 2.58.0/prism/linux/amd64/ 2.58
Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc == --- release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc (added) +++ release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc Wed Aug 7 19:26:59 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmaoTuoYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQicbIQAInIlKXSPvWIUY6d9F8ThH4w +XqGYCwlOPsHNgYsDUb4+ho1TEQI4xJN26DZ837j4L7vvR0s+aS+8e4DLZYpYJHek +cCzNL7TcC3E1d6l8rx59/LSM0qGGsKQmrBaQfFbmni1qA4Ifyp9JBeIqu396pS24 +IBNPnkAR3J2snBZTJTn/AcWf1Ax3qs7ToXi2WZZhMLwHcw2Ye0HxnW4vA7BpqXFN +YJBM39VKNBnhfkfLiSQZUeQk0h3JyPda9tPfV1Y7D7FaG3G3clmwwjNfRUk8qD9p +a3uvky5K54Q11dyFtfuca+zH4ROK+5JopGWQdzO8gVJkrvaDoNA7be7TsITDwaQL +g8/l7Nh2uOIotCqzC1biuou5tZQjoRg429/NJA5Ce4bRpbr7xPZsipC/fMuE7PDh +PcI7WnNtvdEmQnxz9PtZ2KVtGF+BqvqVQvUXLc0UeM67oDyQ77dsg5YHhKd6rDf+ +1hM9LqJjuQ/xZSfBbJfZNZAQjf/g9ygeoVDAIXbe0ZqBM0LjwMHdxqPatcS3CD+J +l+mh1Ys8+AQ3BBXtDE8Zn/gHDpvJG09IiNWwbdm3gf4OLz68DI8APZx5dPFl8pQo +VyQ9KglbzRgrM6wqfGebMajR9ViTMy//vF5/41K2IF03dUWUwVWbwhImpO4GjliS +nsLHiGg/Iy3woC8VTRGz +=PsTY +-END PGP SIGNATURE- Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 == --- release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 (added) +++ release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 Wed Aug 7 19:26:59 2024 @@ -0,0 +1 @@ +4d12ec7f155dc7144721ae383704a933a8c951cb634b2f72d2738c4177c1a4af21989b3bd805bcebf1624dc6b41aab458f879ae2224eba8624cc7c6bed234ae1 apache_beam-2.58.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl == Binary file - no diff available. Propchange: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc == --- release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc (added) +++ release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc Wed Aug 7 19:26:59 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmaoTuoYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQi8WYP+wTKVjta4TvZsyd1qFx24TBY +YaSlGhyzLr5zmAcvfePuatxAjhGzkvRW1vTn3m/LrcALOGsj3duyiCJVhkSfHdi8 +6p71TB554H3uYDhJHPw2268yBy1OD/o0QN+nig18asNOuBszspLdIQa8usX8qFku +378eDHwO6aDqIorIG+J4Kz9ziOCATPjPkSLEGabKxbdmujhIftslI3z7rbsGGweo +lA2BmCMUL6eiLejUZWfyDJGjYS9uMv+UXIRl7+ZyYpBtEF/l27x7wjdRpYNcQEuK +jCKrg7ZQZhtp19zFjhD8iluZiMtQ5auc11mQuBx2+B3PAuljhX1k7LHLvrBMqRQJ +JjuLLDN749HMgXXRihauERuwPc29fcIlSFZ5aWajrRzYVVZd4IVMOJZIl0rqyrwl +HnAggunysXDlL5hOhfQ3hRWFsoYwO2iIDdCVuneBw4zT+lJ/NsC+uKcI1nqC0JBH +Tz33eK0l3Rd+BjxynjMZunK0y2lcElJuz5LHh5w7EkwOwvhAHqQgS1ddqoJnB6E6 +mOPdws9KJTTA3UVxEChQMoSFfql5AHJ0IDuDx42v9bB/yKs3SZpX0Mke+pOrk+pa +oQFh+7k3c854OJlJeU/y2kkFsNI6kqtTQLGbquGAvoXn4SiCnjxPz0fwY1QYvTlJ +dY/TYddI8jeYTbepMGKD +=ve5v +-END PGP SIGNATURE- Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 == --- release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 (added) +++ release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 Wed Aug 7 19:26:59 2024 @@ -0,0 +1 @@ +45eb03e428d8ee9ec8a2b8ba3a5195327be1ac4182acf5876866f947861e6dacb59ae38df98583d3dc5e7e69b21c45873508473d1dcc88af484e0cd5051c89b8 apache_beam-2.58.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-win32.whl == Binary file - no diff available. Propchange: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-win32.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.58.0/python/apache_beam-2.58.0-cp39-cp39-win32.whl.asc ==
(beam) branch dependabot/go_modules/sdks/github.com/nats-io/nats-server/v2-2.10.17 deleted (was 5e99979520e)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch dependabot/go_modules/sdks/github.com/nats-io/nats-server/v2-2.10.17 in repository https://gitbox.apache.org/repos/asf/beam.git was 5e99979520e Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks (#31709)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 07e692b56fb Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks (#31709) 07e692b56fb is described below commit 07e692b56fb19550c40eede5d39b951851eb8980 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Wed Aug 7 10:06:22 2024 -0700 Bump github.com/nats-io/nats-server/v2 from 2.10.12 to 2.10.17 in /sdks (#31709) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.10.12 to 2.10.17. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](https://github.com/nats-io/nats-server/compare/v2.10.12...v2.10.17) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 14 +++--- sdks/go.sum | 32 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index fb0b7f85f3d..654a4562854 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -44,8 +44,8 @@ require ( github.com/johannesboyne/gofakes3 v0.0.0-20221110173912-32fb85c5aed6 github.com/lib/pq v1.10.9 github.com/linkedin/goavro/v2 v2.13.0 - github.com/nats-io/nats-server/v2 v2.10.16 - github.com/nats-io/nats.go v1.35.0 + github.com/nats-io/nats-server/v2 v2.10.18 + github.com/nats-io/nats.go v1.36.0 github.com/proullon/ramsql v0.1.3 github.com/spf13/cobra v1.8.1 github.com/testcontainers/testcontainers-go v0.32.0 @@ -56,7 +56,7 @@ require ( golang.org/x/net v0.26.0 golang.org/x/oauth2 v0.21.0 golang.org/x/sync v0.8.0 - golang.org/x/sys v0.21.0 + golang.org/x/sys v0.22.0 golang.org/x/text v0.17.0 google.golang.org/api v0.187.0 google.golang.org/genproto v0.0.0-20240624140628-dc46fd24d27d @@ -89,10 +89,10 @@ require ( github.com/go-ole/go-ole v1.2.6 // indirect github.com/golang/protobuf v1.5.4 // indirect github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect - github.com/minio/highwayhash v1.0.2 // indirect + github.com/minio/highwayhash v1.0.3 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect github.com/moby/sys/user v0.1.0 // indirect - github.com/nats-io/jwt/v2 v2.5.7 // indirect + github.com/nats-io/jwt/v2 v2.5.8 // indirect github.com/nats-io/nkeys v0.4.7 // indirect github.com/nats-io/nuid v1.0.1 // indirect github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect @@ -158,7 +158,7 @@ require ( github.com/gorilla/handlers v1.5.2 // indirect github.com/gorilla/mux v1.8.1 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect - github.com/klauspost/compress v1.17.8 // indirect + github.com/klauspost/compress v1.17.9 // indirect github.com/klauspost/cpuid/v2 v2.2.6 // indirect github.com/magiconair/properties v1.8.7 // indirect github.com/moby/patternmatcher v0.6.0 // indirect @@ -181,7 +181,7 @@ require ( github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect github.com/zeebo/xxh3 v1.0.2 // indirect go.opencensus.io v0.24.0 // indirect - golang.org/x/crypto v0.24.0 // indirect + golang.org/x/crypto v0.25.0 // indirect golang.org/x/mod v0.17.0 // indirect golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index 6afc175732f..1c03d5afc89 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1019,8 +1019,8 @@ github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j github.com/klauspost/compress v1.9.7/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= github.com/klauspost/compress v1.13.1/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= -github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU= -github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw= +github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA= +github.com/klauspost/compress v1.17.
