[beam] branch nightly-refs/heads/master updated (ffb43321a0a -> 3b890399e11)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from ffb43321a0a [YAML] Add optional output type parameter to mappings. (#29077) add 8b8af7fee89 Change level of initialization exception to error, make errors obvious (#29078) add 1ff014d6101 Move metrics report fully to GHA (#29136) add 60c2ba56f60 Set correct markers for the other test scenario. (#29141) add a411a71bc46 [RRIO] create RedisClient utility (#29081) add 16d68c120fc [yaml] Normalize JdbcIO (#28971) add 32d1cc0a381 Bump Dataflow dev container versions to beam-master-20231026 (#29153) add f4574bc9721 Fix cleanup dataproc cluster name pattern (#29154) add 3b890399e11 Automatically switch to upload_graph when the graph is large (#28621) No new revisions were added by this update. Summary of changes: .github/workflows/beam_Metrics_Report.yml | 26 +-- .test-infra/dataproc/cleanup.sh| 7 +- .test-infra/jenkins/job_Metrics_Report.groovy | 77 CHANGES.md | 1 + runners/google-cloud-dataflow-java/build.gradle| 4 +- .../beam/runners/dataflow/DataflowRunner.java | 24 ++- .../beam/runners/dataflow/DataflowRunnerTest.java | 36 +++- .../java/org/apache/beam/fn/harness/FnHarness.java | 5 +- .../io/jdbc/JdbcReadSchemaTransformProvider.java | 13 ++ .../io/jdbc/JdbcWriteSchemaTransformProvider.java | 39 +++- sdks/java/io/rrio/build.gradle | 12 ++ .../beam/io/requestresponse/RedisClient.java | 188 ++ .../beam/io/requestresponse/RedisClientTestIT.java | 209 + .../RedisExternalResourcesRule.java| 71 +++ .../ml/inference/vertex_ai_inference_it_test.py| 3 +- sdks/python/apache_beam/yaml/standard_io.yaml | 33 16 files changed, 627 insertions(+), 121 deletions(-) delete mode 100644 .test-infra/jenkins/job_Metrics_Report.groovy create mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/RedisClient.java create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RedisClientTestIT.java create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RedisExternalResourcesRule.java
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 7cdac6816be Updating config from bot 7cdac6816be is described below commit 7cdac6816bef2a0313c3d80615c4ece2c50380a1 Author: github-actions AuthorDate: Fri Oct 27 05:05:59 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index df0dd30c684..c23afa9f4c4 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -5,6 +5,6 @@ "jrmccluskey": 1698210371158, "youngoli": 1657688896155, "damccorm": 1680501930289, -"riteshghorse": 1697778356534 +"riteshghorse": 1698383156356 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9be4a60576d Updating config from bot
9be4a60576d is described below
commit 9be4a60576d84eda527beea3e55c20bb2e36803d
Author: github-actions
AuthorDate: Fri Oct 27 05:06:02 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29158.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29158.json
b/scripts/ci/pr-bot/state/pr-state/pr-29158.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29158.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e6b5ece43e8 Updating config from bot
e6b5ece43e8 is described below
commit e6b5ece43e8c502192501ac2849a4aec44cd587f
Author: github-actions
AuthorDate: Fri Oct 27 05:05:58 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29159.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29159.json
b/scripts/ci/pr-bot/state/pr-state/pr-29159.json
new file mode 100644
index 000..aa3638a74ec
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29159.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "riteshghorse"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: [YAML] Kafka Read Provider (#28865)
This is an automated email from the ASF dual-hosted git repository.
austin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e98e37f15b8 [YAML] Kafka Read Provider (#28865)
e98e37f15b8 is described below
commit e98e37f15b898c55b63af680aff33f528d2ede38
Author: Ferran Fernández Garrido
AuthorDate: Fri Oct 27 06:46:38 2023 +0200
[YAML] Kafka Read Provider (#28865)
* [YAML] Kafka provider
---
.../KafkaReadSchemaTransformConfiguration.java | 2 +-
.../io/kafka/KafkaReadSchemaTransformProvider.java | 54 +-
.../KafkaReadSchemaTransformProviderTest.java | 18
sdks/python/apache_beam/yaml/standard_io.yaml | 21 +
4 files changed, 92 insertions(+), 3 deletions(-)
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
index f7e09158068..299a4f222e3 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
@@ -39,7 +39,7 @@ public abstract class KafkaReadSchemaTransformConfiguration {
public static final Set VALID_START_OFFSET_VALUES =
Sets.newHashSet("earliest", "latest");
- public static final String VALID_FORMATS_STR = "AVRO,JSON";
+ public static final String VALID_FORMATS_STR = "RAW,AVRO,JSON";
public static final Set VALID_DATA_FORMATS =
Sets.newHashSet(VALID_FORMATS_STR.split(","));
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
index 0c091bf9ba8..fcba5c355df 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java
@@ -49,6 +49,7 @@ import org.apache.beam.sdk.transforms.DoFn.FinishBundle;
import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionRowTuple;
@@ -112,16 +113,56 @@ public class KafkaReadSchemaTransformProvider
consumerConfigs.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, 100);
consumerConfigs.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
autoOffsetReset);
+String format = configuration.getFormat();
+
+if (format != null && format.equals("RAW")) {
+ if (inputSchema != null) {
+throw new IllegalArgumentException(
+"To read from Kafka in RAW format, you can't provide a schema.");
+ }
+ Schema rawSchema = Schema.builder().addField("payload",
Schema.FieldType.BYTES).build();
+ SerializableFunction valueMapper =
getRawBytesToRowFunction(rawSchema);
+ return new SchemaTransform() {
+@Override
+public PCollectionRowTuple expand(PCollectionRowTuple input) {
+ KafkaIO.Read kafkaRead =
+ KafkaIO.readBytes()
+ .withConsumerConfigUpdates(consumerConfigs)
+ .withConsumerFactoryFn(new
ConsumerFactoryWithGcsTrustStores())
+ .withTopic(configuration.getTopic())
+ .withBootstrapServers(configuration.getBootstrapServers());
+ if (isTest) {
+kafkaRead =
kafkaRead.withMaxReadTime(Duration.standardSeconds(testTimeoutSecs));
+ }
+
+ PCollection kafkaValues =
+
input.getPipeline().apply(kafkaRead.withoutMetadata()).apply(Values.create());
+
+ PCollectionTuple outputTuple =
+ kafkaValues.apply(
+ ParDo.of(new ErrorFn("Kafka-read-error-counter",
valueMapper))
+ .withOutputTags(OUTPUT_TAG, TupleTagList.of(ERROR_TAG)));
+
+ return PCollectionRowTuple.of(
+ "output",
+ outputTuple.get(OUTPUT_TAG).setRowSchema(rawSchema),
+ "errors",
+ outputTuple.get(ERROR_TAG).setRowSchema(ERROR_SCHEMA));
+}
+ };
+}
+
if (inputSchema != null && !inputSchema.isEmpty()) {
assert
Strings.isNullOrEmpty(configuration.getConfluentSchemaRegistryUrl())
: "To read from Kafka, a schema must be provided directly or though
Confluent "
+ "Schema Registry, but not both.";
+
final Schema beamSchema =
- Objects.equals(configuration.getFormat(), "JSON")
+
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ebe057aa70f Updating config from bot
ebe057aa70f is described below
commit ebe057aa70f1659a7d25a0a96eb722021279da94
Author: github-actions
AuthorDate: Fri Oct 27 04:34:51 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29157.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29157.json
b/scripts/ci/pr-bot/state/pr-state/pr-29157.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29157.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/sdks/github.com/google/uuid-1.4.0 created (now 713002fee05)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/google/uuid-1.4.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 713002fee05 Bump github.com/google/uuid from 1.3.1 to 1.4.0 in /sdks No new revisions were added by this update.
[beam] branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.26.0 created (now e04579d7ec5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.26.0 in repository https://gitbox.apache.org/repos/asf/beam.git at e04579d7ec5 Bump github.com/testcontainers/testcontainers-go in /sdks No new revisions were added by this update.
[beam] branch master updated: Automatically switch to upload_graph when the graph is large (#28621)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3b890399e11 Automatically switch to upload_graph when the graph is
large (#28621)
3b890399e11 is described below
commit 3b890399e11d65695d04ae679e2fdf56555630a8
Author: liferoad
AuthorDate: Thu Oct 26 22:14:58 2023 -0400
Automatically switch to upload_graph when the graph is large (#28621)
* Automatically Switch to upload_graph when the graph is large
* fix the formats
* updated the CHANGES.md
* added one test
* fixed the styles
* fixed the styles
* fixed the styles
* addressed the comments
---
CHANGES.md | 1 +
.../beam/runners/dataflow/DataflowRunner.java | 24 ++-
.../beam/runners/dataflow/DataflowRunnerTest.java | 36 +-
3 files changed, 53 insertions(+), 8 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 49b4fdfe89b..a97035fdc29 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -71,6 +71,7 @@ should handle this.
([#25252](https://github.com/apache/beam/issues/25252)).
## New Features / Improvements
* X feature added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
+* `upload_graph` as one of the Experiments options for DataflowRunner is no
longer required when the graph is larger than 10MB for Java SDK
([PR#28621](https://github.com/apache/beam/pull/28621).
## Breaking Changes
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 891b4c0454c..80b4e4cfd8b 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -168,7 +168,6 @@ import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.Vi
import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Joiner;
import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Utf8;
import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterables;
@@ -1330,15 +1329,26 @@ public class DataflowRunner extends
PipelineRunner {
hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
}
+// enable upload_graph when the graph is too large
+byte[] jobGraphBytes =
DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8);
+int jobGraphByteSize = jobGraphBytes.length;
+if (jobGraphByteSize >= CREATE_JOB_REQUEST_LIMIT_BYTES
+&& !hasExperiment(options, "upload_graph")) {
+ List experiments = firstNonNull(options.getExperiments(), new
ArrayList<>());
+ experiments.add("upload_graph");
+ options.setExperiments(ImmutableList.copyOf(experiments));
+ LOG.info(
+ "The job graph size ({} in bytes) is larger than {}. Automatically
add "
+ + "the upload_graph option to experiments.",
+ jobGraphByteSize,
+ CREATE_JOB_REQUEST_LIMIT_BYTES);
+}
+
// Upload the job to GCS and remove the graph object from the API call.
The graph
// will be downloaded from GCS by the service.
if (hasExperiment(options, "upload_graph")) {
DataflowPackage stagedGraph =
- options
- .getStager()
- .stageToFile(
-
DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8),
- DATAFLOW_GRAPH_FILE_NAME);
+ options.getStager().stageToFile(jobGraphBytes,
DATAFLOW_GRAPH_FILE_NAME);
newJob.getSteps().clear();
newJob.setStepsLocation(stagedGraph.getLocation());
}
@@ -1398,7 +1408,7 @@ public class DataflowRunner extends
PipelineRunner {
} catch (GoogleJsonResponseException e) {
String errorMessages = "Unexpected errors";
if (e.getDetails() != null) {
-if (Utf8.encodedLength(newJob.toString()) >=
CREATE_JOB_REQUEST_LIMIT_BYTES) {
+if (jobGraphByteSize >= CREATE_JOB_REQUEST_LIMIT_BYTES) {
errorMessages =
"The size of the serialized JSON representation of the pipeline "
+ "exceeds the allowable limit. "
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
b/runne
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 3d0db6c2b51 Updating config from bot
3d0db6c2b51 is described below
commit 3d0db6c2b51845494f0b0b688c9cf3d8e1e126d8
Author: github-actions
AuthorDate: Fri Oct 27 01:26:09 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29156.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29156.json
b/scripts/ci/pr-bot/state/pr-state/pr-29156.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29156.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch asf-site updated: Publishing website 2023/10/26 23:39:09 at commit f4574bc
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 39747662f5a Publishing website 2023/10/26 23:39:09 at commit f4574bc 39747662f5a is described below commit 39747662f5a1817c71f642018a4d7f56287a6e21 Author: runner AuthorDate: Thu Oct 26 23:39:09 2023 + Publishing website 2023/10/26 23:39:09 at commit f4574bc --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 54d8b2e6c83..505fadfccbc 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.51.0/2023-10-26T09:48:51-07:00/categories/blog/2023-10-26T09:48:51-07:00/blog/2023-10-26T09:48:51-07:00/categories/2023-10-26T09:48:51-07:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.51.0/2023-10-26T17:22:48-04:00/categories/blog/2023-10-26T17:22:48-04:00/blog/2023-10-26T17:22:48-04:00/categories/2023-10-26T17:22:48-04:00/catego [...] \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bc917900750 Updating config from bot
bc917900750 is described below
commit bc917900750d182199642cc413a789d043a28b35
Author: github-actions
AuthorDate: Thu Oct 26 23:34:43 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index a8933303029..761edb73bb7 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1698236136224,
+"damccorm": 1698363279624,
"Abacn": 1698262498472
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e8581b36557 Updating config from bot
e8581b36557 is described below
commit e8581b365573450182273d8510f13d9a8505994a
Author: github-actions
AuthorDate: Thu Oct 26 23:34:42 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29155.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29155.json
b/scripts/ci/pr-bot/state/pr-state/pr-29155.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29155.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Fix cleanup dataproc cluster name pattern (#29154)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f4574bc9721 Fix cleanup dataproc cluster name pattern (#29154)
f4574bc9721 is described below
commit f4574bc9721e82192e81bdb1f5e97c4b83d20fbc
Author: Yi Hu
AuthorDate: Thu Oct 26 17:22:48 2023 -0400
Fix cleanup dataproc cluster name pattern (#29154)
---
.test-infra/dataproc/cleanup.sh | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/.test-infra/dataproc/cleanup.sh b/.test-infra/dataproc/cleanup.sh
index e535fabb777..3efb80d8613 100755
--- a/.test-infra/dataproc/cleanup.sh
+++ b/.test-infra/dataproc/cleanup.sh
@@ -20,10 +20,7 @@ clustersList=( )
toDeleteList=( )
-generatedResources=("beam-loadtests-go-cogbk-flink"
"beam-loadtests-python-cogbk-flink" \
-"beam-loadtests-go-combine-flink" "beam-loadtests-python-combine-flink" \
-"beam-loadtests-go-gbk-flink" "beam-loadtests-python-gbk-flink" \
-"beam-loadtests-go-pardo-flink" "beam-loadtests-python-pardo-flink" \
+generatedResources=("beam-loadtests-go-*-flink"
"beam-loadtests-python-*-flink" \
"beam-postcommit-python-chicago" )
function deleteFilteredClusters(){
@@ -51,7 +48,7 @@ function filterClusters(){
if [[ $elapsedHours -ge 2 ]]; then
for name in ${generatedResources[@]}; do
# Only resources generated by the groovy jobs
set are queued for deletion
- if [[ "$cluster" == *"$name"* ]]; then
+ if [[ "$cluster" == *${name}* ]]; then
toDeleteList+=( "$cluster" )
break
fi
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new fe48bf84516 Updating config from bot
fe48bf84516 is described below
commit fe48bf84516968e758a304177ee665578db21ce4
Author: github-actions
AuthorDate: Thu Oct 26 20:57:28 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29154.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29154.json
b/scripts/ci/pr-bot/state/pr-state/pr-29154.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29154.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Bump Dataflow dev container versions to beam-master-20231026 (#29153)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 32d1cc0a381 Bump Dataflow dev container versions to
beam-master-20231026 (#29153)
32d1cc0a381 is described below
commit 32d1cc0a3816a0dd2164a91ecceaeabf3e921b86
Author: Yi Hu
AuthorDate: Thu Oct 26 16:30:33 2023 -0400
Bump Dataflow dev container versions to beam-master-20231026 (#29153)
---
runners/google-cloud-dataflow-java/build.gradle | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index 9e4d28ae5b4..69759fc76a6 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -51,8 +51,8 @@ evaluationDependsOn(":sdks:java:container:java11")
ext.dataflowLegacyEnvironmentMajorVersion = '8'
ext.dataflowFnapiEnvironmentMajorVersion = '8'
-ext.dataflowLegacyContainerVersion = 'beam-master-20230809'
-ext.dataflowFnapiContainerVersion = 'beam-master-20230809'
+ext.dataflowLegacyContainerVersion = 'beam-master-20231026'
+ext.dataflowFnapiContainerVersion = 'beam-master-20231026'
ext.dataflowContainerBaseRepository = 'gcr.io/cloud-dataflow/v1beta3'
processResources {
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5718b104eb1 Updating config from bot
5718b104eb1 is described below
commit 5718b104eb1aa90ead7ccd7fc38f1aafd4cbaebf
Author: github-actions
AuthorDate: Thu Oct 26 19:33:10 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29153.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29153.json
b/scripts/ci/pr-bot/state/pr-state/pr-29153.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29153.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (a411a71bc46 -> 16d68c120fc)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a411a71bc46 [RRIO] create RedisClient utility (#29081) add 16d68c120fc [yaml] Normalize JdbcIO (#28971) No new revisions were added by this update. Summary of changes: .../io/jdbc/JdbcReadSchemaTransformProvider.java | 13 .../io/jdbc/JdbcWriteSchemaTransformProvider.java | 39 +++--- sdks/python/apache_beam/yaml/standard_io.yaml | 33 ++ 3 files changed, 80 insertions(+), 5 deletions(-)
[beam] branch dependabot/gradle/com.gradle.enterprise-3.15.1 updated (26b1e90f2f0 -> 6f6b5e4ab32)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/gradle/com.gradle.enterprise-3.15.1
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 26b1e90f2f0 Bump com.gradle.enterprise from 3.13.2 to 3.15.1
add 05861613f48 Fix typo and remove extra button (#28965)
add a7c6728b004 Fix publish snapshots github actions (#28961)
add a342d2817a9 Github Workflow Replacement for Jenkins Jobs,
beam_LoadTests_Python_GBK_Flink_Batch; beam_Java_LoadTests_Combine_Smoke_PR
(#28897)
add 2dc766512c0 Bump github.com/aws/aws-sdk-go-v2 from 1.21.1 to 1.21.2 in
/sdks (#28978)
add 3a749a9a6ba Add GitHub Workflow Replacements for Jenkins
job_Python_LoadTests_Smoke and job_Java_LoadTests_GBK_Smoke (#28854)
add 71b8e93b6d3 Add GitHub Workflow Replacements for Jenkins
job_Python_ParDo_DataFlow and job_Python_ParDo_Flink (#28894)
add c0af78a9e98 Revert "Setup Java 21 container (#28833)" (#28968)
add dad13cfbc9a Fix `watch_file_pattern` condition in RunInference (#28948)
add 58e3bdd678b Shutdown more Jenkins PreCommits (#28840)
add 3a45ecf4b27 Updated the DoFn documentation with pickling (#28970)
add f4f1c3c22d9 Add LoadTests Java ParDo Dataflow V2 workflows (#28744)
add dd40403752f beam_LoadTests_Python_GBK_reiterate_Dataflow (#28861)
add 155f8507d07 Add CloudML Benchmarks Dataflow workflow (#28941)
add d636dd32aca Add KeyedModelHandler content to the multi-model ML page
(#28987)
add 250483c5f1f Bump org.checkerframework:checkerframework-gradle-plugin
(#28979)
add 47d0fd566f8 Set MIN_PER_PARTITION_MEMORY to 10 MiB instead of 4 MiB to
match the change in the backend limit. (#28964)
add f93f62e9d82 reorder args to avoid newline issues (#28994)
add 8de14655f78 [RRIO] Stub the RequestResponseIO transform (#28950)
add 6b32a3fb7d9 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#29002)
add 6a57d0d49e6 [JdbcIO] Allow fetchSize to be set for partitioned reads
(#28999)
add 6758aeb0eb4 Support metric_name as list for perf alert tool (#28902)
add 5e06f54371b Update .htaccess to redirect
https://beam.apache.org/contribute/release-guide/
add 41fb7e25163 Merge pull request #28998: Update .htaccess to redirect
the release guide
add 6ca14e5ce46 Bump google.golang.org/api from 0.146.0 to 0.147.0 in
/sdks (#28975)
add 7ac5c175871 Update Google Cloud Java Libraries BOM
add fddab443cca Merge pull request #28656: Update Google Cloud Java
Libraries BOM from 26.22.0 to 26.23.0
add 420418cf23e Fix concurrency group (#28989)
add 9676fccc6e2 Change the if condition to check if build is installed
(#28992)
add d38fdd8e7d0 Add no-cache-dir to tox install
add 75508e5a92c Revert "Add no-cache-dir to tox install" (#29023)
add 1b50ebbe157 Fix concurrency groups in Readme (#29021)
add 7df9b24c6c6 [RRIO] [Cache] Stub CacheWrite (#29012)
add a6917d410d3 Add no-cache-dir to tox install (#29024)
add d32a8e55183 Update the Beam ML readme (#29031)
add 48ad7e338b7 Fix registry (#29013)
add 2a764dd2a05 Change upload test report step (#29036)
add 83355135b42 Add raw format option.
add 5bfd422c8d7 Timestamp and id attributes.
add 0940e900681 Attribute handling.
add 868f06da01d Add configurable error handling.
add 784e9278b15 guava
add f12f529882d cleanup
add ef97c4d61f9 Merge pull request #28782 [YAML] Java PubSubIO
add bde60059f09 Log warning when gen_protos is not found (#29025)
add 0b50a173608 Fix num workers (#29006)
add f8ca45ebb7b [Website] add linkedin case-study
add 36574ce3109 [Website] add linkedIn case-study (#28988)
add 5f1659af9ab Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#29032)
add 3097f50230a Bump cloud.google.com/go/datastore from 1.14.0 to 1.15.0
in /sdks (#28910)
add ce28a36d23f Add cleanup stale k8s namespace to CleanUpGCPResource
workflow (#29038)
add 587fa177b69 Fix beam_LoadTests_Go_GBK_Dataflow_Batch job (#28960)
add 67c30a447cf Fix bug causing job failure (#28982)
add 416a4f268a2 [RRIO] [Call] Create CallShouldBackoff and default
implementation (#28952)
add eb7997145e4 [RRIO] [Cache] Stub CacheRead transform (#29011)
add 89115954f67 Rename rrio package (#29040)
add 574255a0411 Bump google.golang.org/grpc from 1.58.3 to 1.59.0 in /sdks
(#29047)
add 1cb87c34dbc old py, java, spotbugs plugins fix (#29008)
add 4e6f10b5be5 Add withAppProfileId to BigtableIO Read and Write (#28864)
add 5fbf65afd97 Add `requirementsFile` parameter to LaunchConfig. (#28732)
add 9745ea22497 Bump urllib3 from 1.26.17 to 1.26.18 in
/sdks/python/container/py310 (#29045)
add dbc5c4e15b1 [Python] Fail WindowInto when there are no inputs to it
(#28946)
add 130ad8b3ec6 Add MetricsContainer::getPerWorker{Counter|Histogram}
(#28903) (#28923)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2e94f3ce396 Updating config from bot
2e94f3ce396 is described below
commit 2e94f3ce3965bfb6c8ac7d109249793e8c205dee
Author: github-actions
AuthorDate: Thu Oct 26 18:53:39 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29152.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29152.json
b/scripts/ci/pr-bot/state/pr-state/pr-29152.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29152.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new d4a6b333dae Updating config from bot
d4a6b333dae is described below
commit d4a6b333daecac6c925d53f7ad77938c43f2eed2
Author: github-actions
AuthorDate: Thu Oct 26 18:40:56 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29151.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29151.json
b/scripts/ci/pr-bot/state/pr-state/pr-29151.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29151.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (60c2ba56f60 -> a411a71bc46)
This is an automated email from the ASF dual-hosted git repository. damondouglas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 60c2ba56f60 Set correct markers for the other test scenario. (#29141) add a411a71bc46 [RRIO] create RedisClient utility (#29081) No new revisions were added by this update. Summary of changes: sdks/java/io/rrio/build.gradle | 12 ++ .../beam/io/requestresponse/RedisClient.java | 188 ++ .../beam/io/requestresponse/RedisClientTestIT.java | 209 + .../RedisExternalResourcesRule.java| 71 +++ 4 files changed, 480 insertions(+) create mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/RedisClient.java create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RedisClientTestIT.java create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RedisExternalResourcesRule.java
[beam] branch tvalentyn-patch-2 deleted (was f2947693cdf)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git was f2947693cdf Set correct markers for the other test scenario. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Set correct markers for the other test scenario. (#29141)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 60c2ba56f60 Set correct markers for the other test scenario. (#29141) 60c2ba56f60 is described below commit 60c2ba56f60ccfd34384711302daf790cd697db7 Author: tvalentyn AuthorDate: Thu Oct 26 09:39:32 2023 -0700 Set correct markers for the other test scenario. (#29141) --- sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py b/sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py index 168ab031abb..7c96dbe8b84 100644 --- a/sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py +++ b/sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py @@ -65,8 +65,7 @@ class VertexAIInference(unittest.TestCase): test_pipeline.get_full_options_as_args(**extra_opts)) self.assertEqual(FileSystems().exists(output_file), True) - @pytest.mark.uses_vertex_ai - @pytest.mark.it_postcommit + @pytest.mark.vertex_ai_postcommit def test_vertex_ai_run_llm_text_classification(self): output_file = '/'.join([_OUTPUT_DIR, str(uuid.uuid4()), 'output.txt'])
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 8567e9813a0 Updating config from bot
8567e9813a0 is described below
commit 8567e9813a0d62d29a011e9e5cc3bd13b176ba0f
Author: github-actions
AuthorDate: Thu Oct 26 16:35:29 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index 0d75d06d2e2..b8e7f85a22d 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -1,7 +1,7 @@
{
"label": "io",
"dateOfLastReviewAssignment": {
-"chamikaramj": 1697648774461,
+"chamikaramj": 1698338123236,
"johnjcasey": 1697819482751,
"pabloem": 1691787951165,
"Abacn": 1697861168643,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1a7d417c86a Updating config from bot
1a7d417c86a is described below
commit 1a7d417c86a8115804b4bd406946257adeb64695
Author: github-actions
AuthorDate: Thu Oct 26 16:35:26 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 87b873235cd..b9f97490bad 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -2,7 +2,7 @@
"label": "java",
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
-"kennknowles": 1697861168642,
+"kennknowles": 1698338123236,
"robertwb": 1698154559603,
"kileys": 1674428678843,
"apilloud": 1678822446183,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new d98652f2105 Updating config from bot
d98652f2105 is described below
commit d98652f21050c2852b5b05713dd9d4771049963e
Author: github-actions
AuthorDate: Thu Oct 26 16:35:25 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29144.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29144.json
b/scripts/ci/pr-bot/state/pr-state/pr-29144.json
new file mode 100644
index 000..51c84381040
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29144.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "kennknowles",
+"io": "chamikaramj"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5d4674dfb8f Updating config from bot
5d4674dfb8f is described below
commit 5d4674dfb8fdd8dddc7f331047af938be7773c64
Author: github-actions
AuthorDate: Thu Oct 26 15:53:23 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-29148.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-29148.json
b/scripts/ci/pr-bot/state/pr-state/pr-29148.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-29148.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/playground/backend/google.golang.org/grpc-1.56.3 updated (312342b2e6c -> 80fe7ea5b3d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/playground/backend/google.golang.org/grpc-1.56.3 in repository https://gitbox.apache.org/repos/asf/beam.git omit 312342b2e6c Bump google.golang.org/grpc from 1.54.0 to 1.56.3 in /playground/backend add ffb43321a0a [YAML] Add optional output type parameter to mappings. (#29077) add 8b8af7fee89 Change level of initialization exception to error, make errors obvious (#29078) add 1ff014d6101 Move metrics report fully to GHA (#29136) new 80fe7ea5b3d Bump google.golang.org/grpc from 1.54.0 to 1.56.3 in /playground/backend This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (312342b2e6c) \ N -- N -- N refs/heads/dependabot/go_modules/playground/backend/google.golang.org/grpc-1.56.3 (80fe7ea5b3d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/beam_Metrics_Report.yml | 26 ++-- .test-infra/jenkins/job_Metrics_Report.groovy | 77 -- .../java/org/apache/beam/fn/harness/FnHarness.java | 5 +- sdks/python/apache_beam/transforms/core.py | 11 +++- sdks/python/apache_beam/yaml/yaml_mapping.md | 44 + sdks/python/apache_beam/yaml/yaml_mapping.py | 68 +-- sdks/python/apache_beam/yaml/yaml_mapping_test.py | 21 ++ 7 files changed, 145 insertions(+), 107 deletions(-) delete mode 100644 .test-infra/jenkins/job_Metrics_Report.groovy
[beam] 01/01: Bump google.golang.org/grpc from 1.54.0 to 1.56.3 in /playground/backend
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch dependabot/go_modules/playground/backend/google.golang.org/grpc-1.56.3 in repository https://gitbox.apache.org/repos/asf/beam.git commit 80fe7ea5b3d2be179383ca3c91cd9776b7a4d6fb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Thu Oct 26 14:10:39 2023 + Bump google.golang.org/grpc from 1.54.0 to 1.56.3 in /playground/backend Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.54.0 to 1.56.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.54.0...v1.56.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- playground/backend/go.mod | 18 +- playground/backend/go.sum | 27 ++- 2 files changed, 27 insertions(+), 18 deletions(-) diff --git a/playground/backend/go.mod b/playground/backend/go.mod index 9f5fb433ab7..bcb3a78fbd4 100644 --- a/playground/backend/go.mod +++ b/playground/backend/go.mod @@ -18,7 +18,7 @@ module beam.apache.org/playground/backend go 1.20 require ( - cloud.google.com/go/datastore v1.10.0 + cloud.google.com/go/datastore v1.11.0 cloud.google.com/go/logging v1.7.0 github.com/GoogleCloudPlatform/functions-framework-go v1.6.1 github.com/confluentinc/confluent-kafka-go/v2 v2.1.1 @@ -33,16 +33,16 @@ require ( github.com/spf13/viper v1.14.0 github.com/stretchr/testify v1.8.2 go.uber.org/goleak v1.2.0 - google.golang.org/grpc v1.54.0 + google.golang.org/grpc v1.56.3 google.golang.org/protobuf v1.30.0 gopkg.in/yaml.v3 v3.0.1 ) require ( cloud.google.com/go v0.110.0 // indirect - cloud.google.com/go/compute v1.19.0 // indirect + cloud.google.com/go/compute v1.19.1 // indirect cloud.google.com/go/compute/metadata v0.2.3 // indirect - cloud.google.com/go/functions v1.12.0 // indirect + cloud.google.com/go/functions v1.13.0 // indirect cloud.google.com/go/longrunning v0.4.1 // indirect github.com/cenkalti/backoff/v4 v4.1.3 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect @@ -75,15 +75,15 @@ require ( go.uber.org/atomic v1.9.0 // indirect go.uber.org/multierr v1.8.0 // indirect go.uber.org/zap v1.21.0 // indirect - golang.org/x/net v0.8.0 // indirect - golang.org/x/oauth2 v0.6.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/oauth2 v0.7.0 // indirect golang.org/x/sync v0.1.0 // indirect - golang.org/x/sys v0.6.0 // indirect - golang.org/x/text v0.8.0 // indirect + golang.org/x/sys v0.7.0 // indirect + golang.org/x/text v0.9.0 // indirect golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/api v0.114.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/genproto v0.0.0-20230331144136-dcfb400f0633 // indirect + google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect nhooyr.io/websocket v1.8.6 // indirect diff --git a/playground/backend/go.sum b/playground/backend/go.sum index 779dc5a38c1..516388aeb27 100644 --- a/playground/backend/go.sum +++ b/playground/backend/go.sum @@ -168,8 +168,9 @@ cloud.google.com/go/compute v1.13.0/go.mod h1:5aPTS0cUNMIc1CE546K+Th6weJUNQErARy cloud.google.com/go/compute v1.14.0/go.mod h1:YfLtxrj9sU4Yxv+sXzZkyPjEyPBZfXHUvjxega5vAdo= cloud.google.com/go/compute v1.15.1/go.mod h1:bjjoF/NtFUrkD/urWfdHaKuOPDR5nWIs63rR+SXhcpA= cloud.google.com/go/compute v1.18.0/go.mod h1:1X7yHxec2Ga+Ss6jPyjxRxpu2uu7PLgsOVXvgU0yacs= -cloud.google.com/go/compute v1.19.0 h1:+9zda3WGgW1ZSTlVppLCYFIr48Pa35q1uG2N1itbCEQ= cloud.google.com/go/compute v1.19.0/go.mod h1:rikpw2y+UMidAe9tISo04EHNOIf42RLYF/q8Bs93scU= +cloud.google.com/go/compute v1.19.1 h1:am86mquDUgjGNWxiGn+5PGLbmgiWXlE/yNWpIpNvuXY= +cloud.google.com/go/compute v1.19.1/go.mod h1:6ylj3a05WF8leseCdIf77NK0g1ey+nj5IKd5/kvShxE= cloud.google.com/go/compute/metadata v0.1.0/go.mod h1:Z1VN+bulIf6bt4P/C37K4DyZYZEXYonfTBHHFPO/4UU= cloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k= cloud.google.com/go/compute/metadata v0.2.1/go.mod h1:jgHgmJd2RKBGzXqF5LR2EZMGxBkeanZ9wwa75XHJgOM= @@ -220,8 +221,9 @@ cloud.google.com/go/dataqna v0.6.0/go.mod h1:1lqNpM7rqNLVgWBJyk5NF6Uen2PHym0jtVJ cloud.google.com/go/dataqna v0.7.0/go.mod h1:Lx9OcIIeqCrw1a6KdO3/5KMP1wAmTc0slZWwP12Qq3c= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0
[beam] branch master updated: Move metrics report fully to GHA (#29136)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1ff014d6101 Move metrics report fully to GHA (#29136)
1ff014d6101 is described below
commit 1ff014d61011d807a46a978dec3ebcbe7b946487
Author: Danny McCormick
AuthorDate: Thu Oct 26 09:41:50 2023 -0400
Move metrics report fully to GHA (#29136)
* Move metrics report fully to GHA
* Remove issue comment trigger
* Remove issue comment trigger
* Simplify
---
.github/workflows/beam_Metrics_Report.yml | 26 +++--
.test-infra/jenkins/job_Metrics_Report.groovy | 77 ---
2 files changed, 6 insertions(+), 97 deletions(-)
diff --git a/.github/workflows/beam_Metrics_Report.yml
b/.github/workflows/beam_Metrics_Report.yml
index 9c4a540e7ef..1502c1fa326 100644
--- a/.github/workflows/beam_Metrics_Report.yml
+++ b/.github/workflows/beam_Metrics_Report.yml
@@ -18,10 +18,8 @@
name: Beam Metrics Report
on:
- # issue_comment:
- # types: [created]
- # schedule:
- # - cron: '0 */23 * * *'
+ schedule:
+- cron: '0 11 * * 2'
workflow_dispatch:
# This allows a subsequently queued workflow run to interrupt previous runs
@@ -52,26 +50,15 @@ permissions:
jobs:
beam_Metrics_Report:
-name: ${{matrix.job_name}} (${{matrix.job_phrase}})
+name: beam_Metrics_Report
runs-on: [self-hosted, ubuntu-20.04, main]
timeout-minutes: 100
-strategy:
- matrix:
-job_name: [beam_Metrics_Report]
-job_phrase: [Run Metrics Report]
if: |
(github.event_name == 'schedule' ||
- github.event_name == 'workflow_dispatch' ||
- github.event.comment.body == 'Run Metrics Report') &&
+ github.event_name == 'workflow_dispatch') &&
github.repository == 'apache/beam'
steps:
- uses: actions/checkout@v3
- - name: Setup repository
-uses: ./.github/actions/setup-action
-with:
- comment_phrase: ${{ matrix.job_phrase }}
- github_token: ${{ secrets.GITHUB_TOKEN }}
- github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
@@ -102,7 +89,6 @@ jobs:
username: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_ADDRESS }}
password: ${{ secrets.ISSUE_REPORT_SENDER_EMAIL_PASSWORD }}
subject: Beam Metrics Report ${{ env.date }}
- to: dannymccorm...@google.com #d...@beam.apache.org
- cc: vlado.dje...@akvelon.com
- from: gacti...@beam.apache.org #
+ to: d...@beam.apache.org
+ from: beamacti...@gmail.com
html_body: file://${{ github.workspace
}}/.test-infra/jenkins/metrics_report/beam-metrics_report.html
diff --git a/.test-infra/jenkins/job_Metrics_Report.groovy
b/.test-infra/jenkins/job_Metrics_Report.groovy
deleted file mode 100644
index d1d35822121..000
--- a/.test-infra/jenkins/job_Metrics_Report.groovy
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-import InfluxDBCredentialsHelper
-
-job('beam_Metrics_Report') {
- description('Runs Beam metrics report.')
-
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(
- delegate, 'master', 100, true, 'beam', false)
- InfluxDBCredentialsHelper.useCredentials(delegate)
-
- def influxDb = InfluxDBCredentialsHelper.InfluxDBDatabaseName
- def influxHost = InfluxDBCredentialsHelper.InfluxDBHost
- def influxPort = InfluxDBCredentialsHelper.InfluxDBPort
-
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Beam Metrics Report',
- 'Run Metrics Report',
- false
- )
-
- commonJobProperties.setAutoJob(
- delegate,
- '@weekly')
-
- steps {
-gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- commonJobProperties.setGradleSwitches(delegate)
- switch
[beam-starter-go] branch dependabot/go_modules/google.golang.org/grpc-1.58.3 deleted (was 2e48d6c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/google.golang.org/grpc-1.58.3 in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git was 2e48d6c Bump google.golang.org/grpc from 1.58.1 to 1.58.3 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam-starter-go] branch main updated (c34fa8f -> 5c52ca6)
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git from c34fa8f Bump golang.org/x/net from 0.14.0 to 0.17.0 (#22) add 2e48d6c Bump google.golang.org/grpc from 1.58.1 to 1.58.3 new 5c52ca6 Merge pull request #24 from apache/dependabot/go_modules/google.golang.org/grpc-1.58.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
[beam-starter-go] 01/01: Merge pull request #24 from apache/dependabot/go_modules/google.golang.org/grpc-1.58.3
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-go.git commit 5c52ca6babe5c27bacbe656b424441dc387d0ade Merge: c34fa8f 2e48d6c Author: Ritesh Ghorse AuthorDate: Thu Oct 26 09:27:34 2023 -0400 Merge pull request #24 from apache/dependabot/go_modules/google.golang.org/grpc-1.58.3 Bump google.golang.org/grpc from 1.58.1 to 1.58.3 go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (ffb43321a0a -> 8b8af7fee89)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ffb43321a0a [YAML] Add optional output type parameter to mappings. (#29077) add 8b8af7fee89 Change level of initialization exception to error, make errors obvious (#29078) No new revisions were added by this update. Summary of changes: .../harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
