[beam] branch nightly-refs/heads/master updated (0146a8389b4 -> f635ade0e70)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 0146a8389b4 cleanup jobs (#28528)
add 04a26da777f [Python BQ] Allow setting a fixed number of Storage API
streams (#28592)
add 426dbd3955e Revert "[Python BQ] Allow setting a fixed number of
Storage API streams (#28592)" (#28613)
add eb36dcac7a9 Rearrange Java Example tests on Dataflow (#28565)
add 97dea715f8f Add all GHA runs to
http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1
(#28615)
add c2677e28c66 Bump google.golang.org/grpc from 1.58.1 to 1.58.2 in /sdks
(#28607)
add 1b6de5d407b [Python BQ] Follow up of #28592: Allow setting a fixed
number of Storage API streams (#28618)
add 0be5a67243d Move release guide from website to new contributor-docs/
home
add f635ade0e70 Merge pull request #28619: Move release guide from website
to new contributor-docs/ home
No new revisions were added by this update.
Summary of changes:
.github/workflows/README.md| 7 +-
... => beam_PostCommit_Java_Examples_Dataflow.yml} | 58 ++--
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java17.yml | 2 +-
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
..._Precommit_Java_Examples_Dataflow_Java17.groovy | 2 +-
.test-infra/metrics/sync/github/sync_workflows.py | 18 +-
build.gradle.kts | 1 +
.../images/cut-release-branch.png | Bin
.../images/release-guide-1.png | Bin
.../images/tag-rc-commit.png | Bin
.../release-guide.md | 344 ++---
runners/google-cloud-dataflow-java/build.gradle| 25 +-
.../examples/build.gradle | 40 ++-
sdks/go.mod| 2 +-
sdks/go.sum| 4 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 37 ++-
.../io/external/xlang_bigqueryio_it_test.py| 44 ++-
sdks/python/apache_beam/io/gcp/bigquery.py | 9 +
.../documentation/io/built-in/google-bigquery.md | 2 +-
.../partials/section-menu/en/contribute.html | 1 -
23 files changed, 332 insertions(+), 272 deletions(-)
copy .github/workflows/{beam_PostCommit_Java_Examples_Dataflow_V2.yml =>
beam_PostCommit_Java_Examples_Dataflow.yml} (65%)
rename {website/www/site/static =>
contributor-docs}/images/cut-release-branch.png (100%)
rename {website/www/site/static =>
contributor-docs}/images/release-guide-1.png (100%)
rename {website/www/site/static => contributor-docs}/images/tag-rc-commit.png
(100%)
rename {website/www/site/content/en/contribute =>
contributor-docs}/release-guide.md (82%)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new cbb5be4d8a5 Updating config from bot
cbb5be4d8a5 is described below
commit cbb5be4d8a5d7e55778445fe1331817dc0965f51
Author: github-actions
AuthorDate: Sat Sep 23 04:08:38 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28634.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28634.json
b/scripts/ci/pr-bot/state/pr-state/pr-28634.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28634.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: [Fix Python Bigtable dataloss bug] Stop unsetting timestamps of -1 (#28624)
This is an automated email from the ASF dual-hosted git repository.
ahmedabualsaud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 68cf802626a [Fix Python Bigtable dataloss bug] Stop unsetting
timestamps of -1 (#28624)
68cf802626a is described below
commit 68cf802626a1ef7d41aadca13d009f0d1b609b33
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Sat Sep 23 03:56:19 2023 +
[Fix Python Bigtable dataloss bug] Stop unsetting timestamps of -1 (#28624)
* add test for unset timestamp
* pass thru all timestamps
* default to -1 in schematransform
---
.../BigtableWriteSchemaTransformProvider.java | 13 +++--
.../BigtableWriteSchemaTransformProviderIT.java | 19 +++
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +++
sdks/python/apache_beam/io/gcp/bigtableio_it_test.py | 18 ++
4 files changed, 39 insertions(+), 18 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
index d38bdae2f09..b99b69621a8 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
@@ -179,12 +179,13 @@ public class BigtableWriteSchemaTransformProvider
.setColumnQualifier(
ByteString.copyFrom(ofNullable(mutation.get("column_qualifier")).get()))
.setFamilyNameBytes(
-
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()));
-if (mutation.containsKey("timestamp_micros")) {
- setMutation =
- setMutation.setTimestampMicros(
-
Longs.fromByteArray(ofNullable(mutation.get("timestamp_micros")).get()));
-}
+
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()))
+// Use timestamp if provided, else default to -1 (current
Bigtable server time)
+.setTimestampMicros(
+mutation.containsKey("timestamp_micros")
+? Longs.fromByteArray(
+
ofNullable(mutation.get("timestamp_micros")).get())
+: -1);
bigtableMutation =
Mutation.newBuilder().setSetCell(setMutation.build()).build();
break;
case "DeleteFromColumn":
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
index 14bb04b0315..1a60fe661b5 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
@@ -154,8 +154,8 @@ public class BigtableWriteSchemaTransformProviderIT {
public void testSetMutationsExistingColumn() {
RowMutation rowMutation =
RowMutation.create(tableId, "key-1")
-.setCell(COLUMN_FAMILY_NAME_1, "col_a", "val-1-a")
-.setCell(COLUMN_FAMILY_NAME_2, "col_c", "val-1-c");
+.setCell(COLUMN_FAMILY_NAME_1, "col_a", 1000, "val-1-a")
+.setCell(COLUMN_FAMILY_NAME_2, "col_c", 1000, "val-1-c");
dataClient.mutateRow(rowMutation);
List> mutations = new ArrayList<>();
@@ -165,13 +165,15 @@ public class BigtableWriteSchemaTransformProviderIT {
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-a".getBytes(StandardCharsets.UTF_8),
"column_qualifier", "col_a".getBytes(StandardCharsets.UTF_8),
-"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8)));
+"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8),
+"timestamp_micros", Longs.toByteArray(2000)));
mutations.add(
ImmutableMap.of(
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-c".getBytes(StandardCharsets.UTF_8),
"column_qualifier", "col_c".getBytes(StandardCharsets.UTF_8),
-"family_name",
COLUMN_FAMILY_NAME_2.getBytes(StandardCharsets.UTF_8)));
+"family_name",
COLUMN_FAMILY_NAME_2.getBytes(StandardCharsets.UTF_8),
+
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4c7998c672f Updating config from bot
4c7998c672f is described below
commit 4c7998c672fd28b963c0a163ddb1963acdd77ae1
Author: github-actions
AuthorDate: Sat Sep 23 01:08:56 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28633.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28633.json
b/scripts/ci/pr-bot/state/pr-state/pr-28633.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28633.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 93e784cf7de Updating config from bot 93e784cf7de is described below commit 93e784cf7dea0c7536688f14f1b459675a6cf863 Author: github-actions AuthorDate: Fri Sep 22 22:34:43 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 7bb0a1d739e..395c557aad7 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -11,6 +11,6 @@ "damccorm": 1695335680296, "jrmccluskey": 1695396898795, "riteshghorse": 1695333945866, -"liferoad": 1695324880885 +"liferoad": 1695422080469 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 212426e5361 Updating config from bot 212426e5361 is described below commit 212426e5361c8317194f59c06cf2cbb1c97950f4 Author: github-actions AuthorDate: Fri Sep 22 22:34:45 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 844ed8f8004..e102b5b99de 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -6,7 +6,7 @@ "pabloem": 1691787951165, "Abacn": 1695375280928, "ahmedabu98": 1695105814268, -"bvolpato": 1695137718373, +"bvolpato": 1695422080469, "manavgarg": 1690826779210 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 6f04a5e4ca1 Updating config from bot
6f04a5e4ca1 is described below
commit 6f04a5e4ca1a50b31666f3a682d1ad761cbd4773
Author: github-actions
AuthorDate: Fri Sep 22 22:34:42 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28624.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28624.json
b/scripts/ci/pr-bot/state/pr-state/pr-28624.json
new file mode 100644
index 000..9ec9b497254
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28624.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "liferoad",
+"io": "bvolpato"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new b4a0758d12e Updating config from bot b4a0758d12e is described below commit b4a0758d12e1e5c4408d30ae8e29afb1795008bb Author: github-actions AuthorDate: Fri Sep 22 21:34:45 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index c39035ddb25..7bb0a1d739e 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -5,7 +5,7 @@ "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, -"tvalentyn": 1695259574345, +"tvalentyn": 1695418481493, "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1695335680296,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ce6d8dcc25e Updating config from bot
ce6d8dcc25e is described below
commit ce6d8dcc25eb10e1a1c3e623de0a7b46f9b7b333
Author: github-actions
AuthorDate: Fri Sep 22 21:34:43 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28628.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28628.json
b/scripts/ci/pr-bot/state/pr-state/pr-28628.json
new file mode 100644
index 000..321b31c0f05
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28628.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "tvalentyn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 977cc91d0f1 Updating config from bot
977cc91d0f1 is described below
commit 977cc91d0f1d48f4796b2c24a46280a3eb915100
Author: github-actions
AuthorDate: Fri Sep 22 20:58:20 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28631.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28631.json
b/scripts/ci/pr-bot/state/pr-state/pr-28631.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28631.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new c3fbba0a933 Updating config from bot c3fbba0a933 is described below commit c3fbba0a93367b28067376499e3160e209a6cc49 Author: github-actions AuthorDate: Fri Sep 22 20:35:52 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28595.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28595.json b/scripts/ci/pr-bot/state/pr-state/pr-28595.json index e1b3308f259..0046f79aae7 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28595.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28595.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 9408c3aa657 Updating config from bot
9408c3aa657 is described below
commit 9408c3aa6576beff45835a95d8b4843231a91c54
Author: github-actions
AuthorDate: Fri Sep 22 20:06:56 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28626.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28626.json
b/scripts/ci/pr-bot/state/pr-state/pr-28626.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28626.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 8294fb3b28b Updating config from bot
8294fb3b28b is described below
commit 8294fb3b28b485cce739ee725d1ee83f462f1774
Author: github-actions
AuthorDate: Fri Sep 22 20:04:56 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28625.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28625.json
b/scripts/ci/pr-bot/state/pr-state/pr-28625.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28625.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (1b6de5d407b -> f635ade0e70)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 1b6de5d407b [Python BQ] Follow up of #28592: Allow setting a fixed
number of Storage API streams (#28618)
add 0be5a67243d Move release guide from website to new contributor-docs/
home
new f635ade0e70 Merge pull request #28619: Move release guide from website
to new contributor-docs/ home
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../images/cut-release-branch.png | Bin
.../images/release-guide-1.png | Bin
.../images/tag-rc-commit.png | Bin
.../release-guide.md | 344 ++---
.../partials/section-menu/en/contribute.html | 1 -
5 files changed, 172 insertions(+), 173 deletions(-)
rename {website/www/site/static =>
contributor-docs}/images/cut-release-branch.png (100%)
rename {website/www/site/static =>
contributor-docs}/images/release-guide-1.png (100%)
rename {website/www/site/static => contributor-docs}/images/tag-rc-commit.png
(100%)
rename {website/www/site/content/en/contribute =>
contributor-docs}/release-guide.md (82%)
[beam] 01/01: Merge pull request #28619: Move release guide from website to new contributor-docs/ home
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f635ade0e70a8c347d977f6dbd425ba0c4df37d0 Merge: 1b6de5d407b 0be5a67243d Author: Kenneth Knowles AuthorDate: Fri Sep 22 16:04:55 2023 -0400 Merge pull request #28619: Move release guide from website to new contributor-docs/ home .../images/cut-release-branch.png | Bin .../images/release-guide-1.png | Bin .../images/tag-rc-commit.png | Bin .../release-guide.md | 344 ++--- .../partials/section-menu/en/contribute.html | 1 - 5 files changed, 172 insertions(+), 173 deletions(-)
[beam] 01/01: Run arm suite against all validates_container tests
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/armTests
in repository https://gitbox.apache.org/repos/asf/beam.git
commit ae45640e67aa6300284dd48601b27bf0b86ebf95
Author: Danny McCormick
AuthorDate: Fri Sep 22 15:59:37 2023 -0400
Run arm suite against all validates_container tests
---
sdks/python/apache_beam/examples/wordcount_it_test.py | 4
sdks/python/container/run_validatescontainer.sh | 10 --
2 files changed, 4 insertions(+), 10 deletions(-)
diff --git a/sdks/python/apache_beam/examples/wordcount_it_test.py
b/sdks/python/apache_beam/examples/wordcount_it_test.py
index 0a5179e3bf0..bb5859df725 100644
--- a/sdks/python/apache_beam/examples/wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_it_test.py
@@ -112,10 +112,6 @@ class WordCountIT(unittest.TestCase):
def test_wordcount_it_with_use_sibling_sdk_workers(self):
self._run_wordcount_it(wordcount.run, experiment='use_sibling_sdk_workers')
- @pytest.mark.it_dataflow_arm
- def test_wordcount_fnapi_it_arm(self):
-self._run_wordcount_it(wordcount.run, machine_type='t2a-standard-1')
-
def _run_wordcount_it(self, run_wordcount, **opts):
test_pipeline = TestPipeline(is_integration_test=True)
extra_opts = {}
diff --git a/sdks/python/container/run_validatescontainer.sh
b/sdks/python/container/run_validatescontainer.sh
index f09712f74bc..5ee3342a1ef 100755
--- a/sdks/python/container/run_validatescontainer.sh
+++ b/sdks/python/container/run_validatescontainer.sh
@@ -59,7 +59,7 @@ ARCH=${3:-"x86"}
IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk"
CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we
substitute away the dot in the version.
PY_INTERPRETER="python${PY_VERSION}"
-TEST_SUITE_TAG="it_validatescontainer"
+MACHINE_TYPE_ARGS=""
XUNIT_FILE="pytest-$IMAGE_NAME.xml"
@@ -88,12 +88,9 @@ if [[ "$ARCH" == "x86" ]]; then
# Push the container
gcloud docker -- push $CONTAINER:$TAG
elif [[ "$ARCH" == "ARM" ]]; then
- # Note: ARM test suites only run on github actions, where multi-arch Python
SDK containers are already pushed during build.
- # Reset the test suite tag to run ARM pipelines.
- TEST_SUITE_TAG="it_dataflow_arm"
-
# Reset the multi-arch Python SDK container image tag.
TAG=$MULTIARCH_TAG
+ MACHINE_TYPE_ARGS="--machine_type=t2a-standard-1"
else
printf "Please give a valid CPU architecture, either x86 or ARM."
exit 1
@@ -126,7 +123,7 @@ SDK_LOCATION=$2
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
pytest -o log_cli=True -o log_level=Info -o junit_suite_name=$IMAGE_NAME \
- -m=$TEST_SUITE_TAG \
+ -m=it_validatescontainer \
--numprocesses=1 \
--timeout=1800 \
--junitxml=$XUNIT_FILE \
@@ -142,6 +139,7 @@ pytest -o log_cli=True -o log_level=Info -o
junit_suite_name=$IMAGE_NAME \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
+$MACHINE_TYPE_ARGS \
--docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
[beam] branch users/damccorm/armTests created (now ae45640e67a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/armTests in repository https://gitbox.apache.org/repos/asf/beam.git at ae45640e67a Run arm suite against all validates_container tests This branch includes the following new commits: new ae45640e67a Run arm suite against all validates_container tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated: [Python BQ] Follow up of #28592: Allow setting a fixed number of Storage API streams (#28618)
This is an automated email from the ASF dual-hosted git repository.
ahmedabualsaud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1b6de5d407b [Python BQ] Follow up of #28592: Allow setting a fixed
number of Storage API streams (#28618)
1b6de5d407b is described below
commit 1b6de5d407ba4122710ec870c880a1ee443b6d2b
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Fri Sep 22 19:49:41 2023 +
[Python BQ] Follow up of #28592: Allow setting a fixed number of Storage
API streams (#28618)
* expose num streams option; fix some streaming tests
* add test phrase in description
* lint fix
---
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 37 ++
.../io/external/xlang_bigqueryio_it_test.py| 44 ++
sdks/python/apache_beam/io/gcp/bigquery.py | 9 +
.../documentation/io/built-in/google-bigquery.md | 2 +-
6 files changed, 69 insertions(+), 27 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
index d1ee27088c7..1280fcb4e23 100644
---
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Dataflow',
'Run Python_Xlang_Gcp_Dataflow PostCommit', 'Python_Xlang_Gcp_Dataflow
(\"Run Python_Xlang_Gcp_Dataflow PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner. \"Run Python_Xlang_Gcp_Dataflow PostCommit\"')
// Set common parameters.
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
index 438b735fba7..e4bf771be1a 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Direct',
'Run Python_Xlang_Gcp_Direct PostCommit', 'Python_Xlang_Gcp_Direct (\"Run
Python_Xlang_Gcp_Direct PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Direct
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Direct
runner. \"Run Python_Xlang_Gcp_Direct PostCommit\"')
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
index e4461793011..1b9eb309ec4 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
@@ -176,6 +176,13 @@ public class BigQueryStorageWriteApiSchemaTransformProvider
!Strings.isNullOrEmpty(this.getErrorHandling().getOutput()),
invalidConfigMessage + "Output must not be empty if error handling
specified.");
}
+
+ if (this.getAutoSharding() != null && this.getAutoSharding()) {
+checkArgument(
+this.getNumStreams() == 0,
+invalidConfigMessage
++ "Cannot set a fixed number of streams when auto-sharding is
enabled. Please pick only one of the two options.");
+ }
}
/**
@@ -218,11 +225,17 @@ public class
BigQueryStorageWriteApiSchemaTransformProvider
public abstract Boolean getUseAtLeastOnceSemantics();
@SchemaFieldDescription(
-"This option enables using a dynamically determined number of shards
to write to "
+"This option enables using a dynamically determined number of Storage
Write API streams to write to "
+ "BigQu
[beam] branch dependabot/go_modules/sdks/google.golang.org/grpc-1.58.2 deleted (was 5baea42081d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/grpc-1.58.2 in repository https://gitbox.apache.org/repos/asf/beam.git was 5baea42081d Bump google.golang.org/grpc from 1.58.1 to 1.58.2 in /sdks The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated: Bump google.golang.org/grpc from 1.58.1 to 1.58.2 in /sdks (#28607)
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new c2677e28c66 Bump google.golang.org/grpc from 1.58.1 to 1.58.2 in /sdks (#28607) c2677e28c66 is described below commit c2677e28c66e6b91b841c3ad5c843b50b6eb1528 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Sep 22 15:10:58 2023 -0400 Bump google.golang.org/grpc from 1.58.1 to 1.58.2 in /sdks (#28607) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.58.1 to 1.58.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.58.1...v1.58.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index ac7db574791..73b99c8344e 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -59,7 +59,7 @@ require ( golang.org/x/text v0.13.0 google.golang.org/api v0.142.0 google.golang.org/genproto v0.0.0-20230821184602-ccc8af3d0e93 - google.golang.org/grpc v1.58.1 + google.golang.org/grpc v1.58.2 google.golang.org/protobuf v1.31.0 gopkg.in/retry.v1 v1.0.3 gopkg.in/yaml.v2 v2.4.0 diff --git a/sdks/go.sum b/sdks/go.sum index 2b404990716..7e26168a360 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -712,8 +712,8 @@ google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8 google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= -google.golang.org/grpc v1.58.1 h1:OL+Vz23DTtrrldqHK49FUOPHyY75rvFqJfXC84NYW58= -google.golang.org/grpc v1.58.1/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= +google.golang.org/grpc v1.58.2 h1:SXUpjxeVF3FKrTYQI4f4KvbGD5u2xccdYdurwowix5I= +google.golang.org/grpc v1.58.2/go.mod h1:tgX3ZQDlNJGU96V6yHh1T/JeoBQ2TXdr43YbYSsCJk0= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 33190d76cd5 Updating config from bot
33190d76cd5 is described below
commit 33190d76cd522278498c6984f690cf5a145f372e
Author: github-actions
AuthorDate: Fri Sep 22 19:08:26 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28623.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28623.json
b/scripts/ci/pr-bot/state/pr-state/pr-28623.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28623.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 636e7b37749 Updating config from bot
636e7b37749 is described below
commit 636e7b377497065a56ba0a658ea12e3067410136
Author: github-actions
AuthorDate: Fri Sep 22 19:02:08 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28622.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28622.json
b/scripts/ci/pr-bot/state/pr-state/pr-28622.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28622.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 8ddee8c2013 Updating config from bot
8ddee8c2013 is described below
commit 8ddee8c20135ae30223035995737f4308c6df088
Author: github-actions
AuthorDate: Fri Sep 22 18:35:04 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28618.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28618.json
b/scripts/ci/pr-bot/state/pr-state/pr-28618.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28618.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new f77acb0c92e Updating config from bot f77acb0c92e is described below commit f77acb0c92eab5c2f3a6a3a89b01fdb14ddce26c Author: github-actions AuthorDate: Fri Sep 22 15:35:02 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 4b6b9d4b7dc..c39035ddb25 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -9,7 +9,7 @@ "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1695335680296, -"jrmccluskey": 1695144883894, +"jrmccluskey": 1695396898795, "riteshghorse": 1695333945866, "liferoad": 1695324880885 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 94aaa790252 Updating config from bot
94aaa790252 is described below
commit 94aaa790252ea83419b9ac060a5601b2f8f539e4
Author: github-actions
AuthorDate: Fri Sep 22 15:35:00 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28564.json | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28564.json
b/scripts/ci/pr-bot/state/pr-state/pr-28564.json
index 242a48d7d3b..959c289c017 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28564.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28564.json
@@ -1,7 +1,9 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch users/damccorm/gha-metrics deleted (was 4690514bb71)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/gha-metrics in repository https://gitbox.apache.org/repos/asf/beam.git was 4690514bb71 Add all GHA runs to http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e36ff33c482 Updating config from bot
e36ff33c482 is described below
commit e36ff33c482d4f4409d0adc5c14d7924855466da
Author: github-actions
AuthorDate: Fri Sep 22 15:20:09 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28619.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28619.json
b/scripts/ci/pr-bot/state/pr-state/pr-28619.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28619.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (eb36dcac7a9 -> 97dea715f8f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from eb36dcac7a9 Rearrange Java Example tests on Dataflow (#28565) add 97dea715f8f Add all GHA runs to http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1 (#28615) No new revisions were added by this update. Summary of changes: .test-infra/metrics/sync/github/sync_workflows.py | 18 -- 1 file changed, 8 insertions(+), 10 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 61246732bde Updating config from bot
61246732bde is described below
commit 61246732bde43793d97197eee549d6cd08835029
Author: github-actions
AuthorDate: Fri Sep 22 14:34:48 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28614.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28614.json
b/scripts/ci/pr-bot/state/pr-state/pr-28614.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28614.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/github_actions/actions/checkout-4 updated (b21da1217ef -> ca2f7ac4ba9)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/github_actions/actions/checkout-4
in repository https://gitbox.apache.org/repos/asf/beam.git
omit b21da1217ef Bump actions/checkout from 3 to 4
add 04a26da777f [Python BQ] Allow setting a fixed number of Storage API
streams (#28592)
add 426dbd3955e Revert "[Python BQ] Allow setting a fixed number of
Storage API streams (#28592)" (#28613)
add eb36dcac7a9 Rearrange Java Example tests on Dataflow (#28565)
new ca2f7ac4ba9 Bump actions/checkout from 3 to 4
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (b21da1217ef)
\
N -- N -- N
refs/heads/dependabot/github_actions/actions/checkout-4 (ca2f7ac4ba9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/README.md| 7 +--
... => beam_PostCommit_Java_Examples_Dataflow.yml} | 58 +++---
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java17.yml | 2 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
..._Precommit_Java_Examples_Dataflow_Java17.groovy | 2 +-
build.gradle.kts | 1 +
runners/google-cloud-dataflow-java/build.gradle| 25 +++---
.../examples/build.gradle | 40 +--
9 files changed, 80 insertions(+), 59 deletions(-)
copy .github/workflows/{beam_PostCommit_Java_Examples_Dataflow_V2.yml =>
beam_PostCommit_Java_Examples_Dataflow.yml} (65%)
[beam] 01/01: Bump actions/checkout from 3 to 4
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch dependabot/github_actions/actions/checkout-4 in repository https://gitbox.apache.org/repos/asf/beam.git commit ca2f7ac4ba9285db28385d28862ee81ca8cca9e8 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Sep 22 14:10:30 2023 + Bump actions/checkout from 3 to 4 Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/beam_CleanUpGCPResources.yml | 2 +- .github/workflows/beam_CleanUpPrebuiltSDKImages.yml | 2 +- .github/workflows/beam_LoadTests_Go_Combine_Dataflow_Batch.yml | 2 +- .github/workflows/beam_LoadTests_Go_GBK_Dataflow_Batch.yml | 2 +- .github/workflows/beam_LoadTests_Python_Combine_Dataflow_Batch.yml | 2 +- .github/workflows/beam_PerformanceTests_BiqQueryIO_Read_Python.yml | 2 +- .../workflows/beam_PerformanceTests_BiqQueryIO_Write_Python_Batch.yml | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/beam_CleanUpGCPResources.yml b/.github/workflows/beam_CleanUpGCPResources.yml index 7a420822f9d..875e471a775 100644 --- a/.github/workflows/beam_CleanUpGCPResources.yml +++ b/.github/workflows/beam_CleanUpGCPResources.yml @@ -64,7 +64,7 @@ jobs: github.event_name == 'workflow_dispatch' || github.event.comment.body == 'Run Clean GCP Resources' steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Setup repository uses: ./.github/actions/setup-action with: diff --git a/.github/workflows/beam_CleanUpPrebuiltSDKImages.yml b/.github/workflows/beam_CleanUpPrebuiltSDKImages.yml index 3ecf192aded..069f4a05842 100644 --- a/.github/workflows/beam_CleanUpPrebuiltSDKImages.yml +++ b/.github/workflows/beam_CleanUpPrebuiltSDKImages.yml @@ -64,7 +64,7 @@ jobs: github.event_name == 'workflow_dispatch' || github.event.comment.body == 'Run Clean Prebuilt Images' steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Setup repository uses: ./.github/actions/setup-action with: diff --git a/.github/workflows/beam_LoadTests_Go_Combine_Dataflow_Batch.yml b/.github/workflows/beam_LoadTests_Go_Combine_Dataflow_Batch.yml index 423290d3fdc..4813b2c7e9f 100644 --- a/.github/workflows/beam_LoadTests_Go_Combine_Dataflow_Batch.yml +++ b/.github/workflows/beam_LoadTests_Go_Combine_Dataflow_Batch.yml @@ -62,7 +62,7 @@ jobs: job_name: ["beam_LoadTests_Go_Combine_Dataflow_Batch"] job_phrase: ["Run Load Tests Go Combine Dataflow Batch"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Setup repository uses: ./.github/actions/setup-action with: diff --git a/.github/workflows/beam_LoadTests_Go_GBK_Dataflow_Batch.yml b/.github/workflows/beam_LoadTests_Go_GBK_Dataflow_Batch.yml index bfdb19c1f5d..ebc152e9676 100644 --- a/.github/workflows/beam_LoadTests_Go_GBK_Dataflow_Batch.yml +++ b/.github/workflows/beam_LoadTests_Go_GBK_Dataflow_Batch.yml @@ -62,7 +62,7 @@ jobs: job_name: ["beam_LoadTests_Go_GBK_Dataflow_Batch"] job_phrase: ["Run Load Tests Go GBK Dataflow Batch"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Setup repository uses: ./.github/actions/setup-action with: diff --git a/.github/workflows/beam_LoadTests_Python_Combine_Dataflow_Batch.yml b/.github/workflows/beam_LoadTests_Python_Combine_Dataflow_Batch.yml index f7d7a056d59..c62285da6fe 100644 --- a/.github/workflows/beam_LoadTests_Python_Combine_Dataflow_Batch.yml +++ b/.github/workflows/beam_LoadTests_Python_Combine_Dataflow_Batch.yml @@ -62,7 +62,7 @@ jobs: job_name: ["beam_LoadTests_Python_Combine_Dataflow_Batch"] job_phrase: ["Run Load Tests Python Combine Dataflow Batch"] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Setup repository uses: ./.github/actions/setup-action with: diff --git a/.github/workflows/beam_PerformanceTests_BiqQueryIO_Read_Python.yml b/.github/workflows/beam_PerformanceTests_BiqQueryIO_Read_Python.yml index 224689ee908..dc55788e17f 100644 --- a/.github/workflows/beam_PerformanceTests_BiqQueryIO_Read_Python.yml +++ b/.github/workflows/beam_PerformanceTests_BiqQueryIO_Read_Python.yml @@ -62,7 +62,7 @@ jobs
[beam] branch master updated (426dbd3955e -> eb36dcac7a9)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 426dbd3955e Revert "[Python BQ] Allow setting a fixed number of
Storage API streams (#28592)" (#28613)
add eb36dcac7a9 Rearrange Java Example tests on Dataflow (#28565)
No new revisions were added by this update.
Summary of changes:
.github/workflows/README.md| 7 +--
... => beam_PostCommit_Java_Examples_Dataflow.yml} | 58 +++---
...eam_PreCommit_Java_Examples_Dataflow_Java11.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java17.yml | 2 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
..._Precommit_Java_Examples_Dataflow_Java17.groovy | 2 +-
build.gradle.kts | 1 +
runners/google-cloud-dataflow-java/build.gradle| 25 +++---
.../examples/build.gradle | 40 +--
9 files changed, 80 insertions(+), 59 deletions(-)
copy .github/workflows/{beam_PostCommit_Java_Examples_Dataflow_V2.yml =>
beam_PostCommit_Java_Examples_Dataflow.yml} (65%)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new e44ebe83412 Updating config from bot e44ebe83412 is described below commit e44ebe8341229391d33770c2fcc52c8008782755 Author: github-actions AuthorDate: Fri Sep 22 14:05:52 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 3d7bedad6eb..4d689b70e82 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -7,6 +7,6 @@ "kileys": 1674428678843, "apilloud": 1678822446183, "Abacn": 1695375280928, -"bvolpato": 1695105814268 +"bvolpato": 1695391548648 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b4eab153521 Updating config from bot
b4eab153521 is described below
commit b4eab1535212a5e2dcb9c742667a02090dc238ad
Author: github-actions
AuthorDate: Fri Sep 22 14:05:54 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index fcfe98054e5..844ed8f8004 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -1,7 +1,7 @@
{
"label": "io",
"dateOfLastReviewAssignment": {
-"chamikaramj": 1695103578648,
+"chamikaramj": 1695391548648,
"johnjcasey": 1695212037351,
"pabloem": 1691787951165,
"Abacn": 1695375280928,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 54008944b6c Updating config from bot
54008944b6c is described below
commit 54008944b6c847195252352aac70f0a43ca67b93
Author: github-actions
AuthorDate: Fri Sep 22 14:05:50 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28616.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28616.json
b/scripts/ci/pr-bot/state/pr-state/pr-28616.json
new file mode 100644
index 000..57783c81eb0
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28616.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "bvolpato",
+"io": "chamikaramj"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 03561adcf4c Updating config from bot
03561adcf4c is described below
commit 03561adcf4c0c96608c5a3107e717da8b8f3bca8
Author: github-actions
AuthorDate: Fri Sep 22 13:44:25 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28615.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28615.json
b/scripts/ci/pr-bot/state/pr-state/pr-28615.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28615.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] 01/01: Merge pull request #28594: [Release-2.51.0] Cherry pick #28588: Fix sync it framework BigtableResourceManager
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 6306b21b3c38c937f0c256a40bccaf46614a0602 Merge: 20e11a661c8 693380f23d0 Author: Kenneth Knowles AuthorDate: Fri Sep 22 09:40:53 2023 -0400 Merge pull request #28594: [Release-2.51.0] Cherry pick #28588: Fix sync it framework BigtableResourceManager .../apache/beam/it/gcp/bigtable/BigtableResourceManager.java | 10 +- .../beam/it/gcp/bigtable/BigtableResourceManagerUtils.java | 5 +++-- .../beam/it/gcp/bigtable/BigtableResourceManagerTest.java | 3 ++- 3 files changed, 10 insertions(+), 8 deletions(-)
[beam] branch release-2.51.0 updated (20e11a661c8 -> 6306b21b3c3)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git from 20e11a661c8 Set Dataflow container to release version. add 693380f23d0 Fix sync it (#28588) new 6306b21b3c3 Merge pull request #28594: [Release-2.51.0] Cherry pick #28588: Fix sync it framework BigtableResourceManager The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/it/gcp/bigtable/BigtableResourceManager.java | 10 +- .../beam/it/gcp/bigtable/BigtableResourceManagerUtils.java | 5 +++-- .../beam/it/gcp/bigtable/BigtableResourceManagerTest.java | 3 ++- 3 files changed, 10 insertions(+), 8 deletions(-)
[beam] 01/01: Add all GHA runs to http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/gha-metrics
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 4690514bb711fcd6227c1c5dbaa61ff84a3f9548
Author: Danny McCormick
AuthorDate: Fri Sep 22 09:36:33 2023 -0400
Add all GHA runs to
http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1
Also exclude cancelled runs since many runs will fall into that category
---
.test-infra/metrics/sync/github/sync_workflows.py | 18 --
1 file changed, 8 insertions(+), 10 deletions(-)
diff --git a/.test-infra/metrics/sync/github/sync_workflows.py
b/.test-infra/metrics/sync/github/sync_workflows.py
index a83d518af4f..0b4ddfec380 100644
--- a/.test-infra/metrics/sync/github/sync_workflows.py
+++ b/.test-infra/metrics/sync/github/sync_workflows.py
@@ -17,7 +17,7 @@ You can find the cloud function in the next link
https://console.cloud.google.com/functions/details/us-central1/github_actions_workflows_dashboard_sync?env=gen1&project=apache-beam-testing
Pub sub topic :
https://console.cloud.google.com/cloudpubsub/topic/detail/github_actions_workflows_sync?project=apache-beam-testing
Cron Job :
https://console.cloud.google.com/cloudscheduler/jobs/edit/us-central1/github_actions_workflows_dashboard_sync?project=apache-beam-testing
-Writing the latest 10 runs of every postcommit workflow in master branch in a
beammetrics database
+Writing the latest 10 runs of every workflow in master branch in a beammetrics
database
'''
import os
@@ -134,13 +134,11 @@ def fetchWorkflowData():
workflows.append(workflowsPage)
for pageItem in workflows:
for item in pageItem:
-path =item['path']
-isPostCommit = re.search('(.*)postcommit(.*)',path)
-if isPostCommit:
-result = re.search('/(.*).yml', path)
-path =(result.group(1)) + ".yml"
-workflowObject = Workflow(item['id'],item['name'],path)
-WORKFLOWS_OBJECT_LIST.append(workflowObject)
+path = item['path']
+result = re.search('/(.*).yml', path)
+path = (result.group(1)) + ".yml"
+workflowObject = Workflow(item['id'],item['name'],path)
+WORKFLOWS_OBJECT_LIST.append(workflowObject)
url = "https://api.github.com/repos/apache/beam/actions/workflows/";
queryOptions = { 'branch' : 'master', 'per_page' :
GH_WORKFLOWS_NUMBER_EXECUTIONS,
'page' :'1', 'exclude_pull_request':True }
@@ -154,7 +152,7 @@ def fetchWorkflowData():
if item['status'] == 'completed':
workflow.runUrl.append(item['html_url'])
workflow.listOfRuns.append(item['conclusion'])
-else:
+elif item['status'] != 'cancelled':
workflow.listOfRuns.append(item['status'])
workflow.runUrl.append(item['html_url'])
for i in range(0,GH_WORKFLOWS_NUMBER_EXECUTIONS):
@@ -184,4 +182,4 @@ def databaseOperations(connection,fetchWorkflows):
cursor.execute(query)
cursor.close()
connection.commit()
-connection.close()
\ No newline at end of file
+connection.close()
[beam] branch users/damccorm/gha-metrics created (now 4690514bb71)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/gha-metrics in repository https://gitbox.apache.org/repos/asf/beam.git at 4690514bb71 Add all GHA runs to http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1 This branch includes the following new commits: new 4690514bb71 Add all GHA runs to http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (04a26da777f -> 426dbd3955e)
This is an automated email from the ASF dual-hosted git repository. ahmedabualsaud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 04a26da777f [Python BQ] Allow setting a fixed number of Storage API streams (#28592) add 426dbd3955e Revert "[Python BQ] Allow setting a fixed number of Storage API streams (#28592)" (#28613) No new revisions were added by this update. Summary of changes: ...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +- ...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +- ...ueryStorageWriteApiSchemaTransformProvider.java | 37 + .../io/external/xlang_bigqueryio_it_test.py| 38 -- sdks/python/apache_beam/io/gcp/bigquery.py | 9 - .../documentation/io/built-in/google-bigquery.md | 2 +- 6 files changed, 24 insertions(+), 66 deletions(-)
[beam] branch dependabot/pip/sdks/python/container/py310/cryptography-41.0.4 deleted (was a9d89e8551d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/sdks/python/container/py310/cryptography-41.0.4 in repository https://gitbox.apache.org/repos/asf/beam.git was a9d89e8551d Bump cryptography from 41.0.3 to 41.0.4 in /sdks/python/container/py310 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] 01/01: Revert "[Python BQ] Allow setting a fixed number of Storage API streams (#28592)"
This is an automated email from the ASF dual-hosted git repository.
ahmedabualsaud pushed a commit to branch revert-28592-expose_numshards
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 65bf0e2616e78ba6ffe43748d7cd5a1614c79653
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Fri Sep 22 09:13:13 2023 -0400
Revert "[Python BQ] Allow setting a fixed number of Storage API streams
(#28592)"
This reverts commit 04a26da777ff4c0ed9112f07bf0f41a39bc7260d.
---
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 37 +
.../io/external/xlang_bigqueryio_it_test.py| 38 --
sdks/python/apache_beam/io/gcp/bigquery.py | 9 -
.../documentation/io/built-in/google-bigquery.md | 2 +-
6 files changed, 24 insertions(+), 66 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
index 1280fcb4e23..d1ee27088c7 100644
---
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Dataflow',
'Run Python_Xlang_Gcp_Dataflow PostCommit', 'Python_Xlang_Gcp_Dataflow
(\"Run Python_Xlang_Gcp_Dataflow PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner. \"Run Python_Xlang_Gcp_Dataflow PostCommit\"')
+ description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner.')
// Set common parameters.
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
index e4bf771be1a..438b735fba7 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Direct',
'Run Python_Xlang_Gcp_Direct PostCommit', 'Python_Xlang_Gcp_Direct (\"Run
Python_Xlang_Gcp_Direct PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Direct
runner. \"Run Python_Xlang_Gcp_Direct PostCommit\"')
+ description('Runs end-to-end cross language GCP IO tests on the Direct
runner.')
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
index c3eed246723..e4461793011 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
@@ -176,13 +176,6 @@ public class BigQueryStorageWriteApiSchemaTransformProvider
!Strings.isNullOrEmpty(this.getErrorHandling().getOutput()),
invalidConfigMessage + "Output must not be empty if error handling
specified.");
}
-
- if (this.getAutoSharding() != null && this.getAutoSharding()) {
-checkArgument(
-this.getNumStreams() == 0,
-invalidConfigMessage
-+ "Cannot set a fixed number of streams when auto-sharding is
enabled. Please pick only one of the two options.");
- }
}
/**
@@ -225,17 +218,11 @@ public class
BigQueryStorageWriteApiSchemaTransformProvider
public abstract Boolean getUseAtLeastOnceSemantics();
@SchemaFieldDescription(
-"This option enables using a dynamically determined number of Storage
Write API streams to write to "
+"This option enables using a dynamically determined number of shards
to write to "
+ "BigQuery. Only applicable to unbounded data.")
@Nullable
public abstract Boolean getAutoSharding();
-@SchemaFieldDescription(
-"If set, the Storage API sink will default to using this number of
write streams. " +
-"Only applica
[beam] branch revert-28592-expose_numshards created (now 65bf0e2616e)
This is an automated email from the ASF dual-hosted git repository. ahmedabualsaud pushed a change to branch revert-28592-expose_numshards in repository https://gitbox.apache.org/repos/asf/beam.git at 65bf0e2616e Revert "[Python BQ] Allow setting a fixed number of Storage API streams (#28592)" This branch includes the following new commits: new 65bf0e2616e Revert "[Python BQ] Allow setting a fixed number of Storage API streams (#28592)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (0146a8389b4 -> 04a26da777f)
This is an automated email from the ASF dual-hosted git repository. ahmedabualsaud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0146a8389b4 cleanup jobs (#28528) add 04a26da777f [Python BQ] Allow setting a fixed number of Storage API streams (#28592) No new revisions were added by this update. Summary of changes: ...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +- ...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +- ...ueryStorageWriteApiSchemaTransformProvider.java | 37 - .../io/external/xlang_bigqueryio_it_test.py| 38 ++ sdks/python/apache_beam/io/gcp/bigquery.py | 9 + .../documentation/io/built-in/google-bigquery.md | 2 +- 6 files changed, 66 insertions(+), 24 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 3f359cdcae8 Updating config from bot
3f359cdcae8 is described below
commit 3f359cdcae83d4d18ae29617ac9da54375f96fc2
Author: github-actions
AuthorDate: Fri Sep 22 13:06:52 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28609.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28609.json
b/scripts/ci/pr-bot/state/pr-state/pr-28609.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28609.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 964a945d99b Updating config from bot 964a945d99b is described below commit 964a945d99b5049443e27ae6d35c2c5ac893baae Author: github-actions AuthorDate: Fri Sep 22 09:34:44 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 892796945b0..3d7bedad6eb 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -6,7 +6,7 @@ "robertwb": 1695137718373, "kileys": 1674428678843, "apilloud": 1678822446183, -"Abacn": 1694797564637, +"Abacn": 1695375280928, "bvolpato": 1695105814268 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ca38d12dc36 Updating config from bot ca38d12dc36 is described below commit ca38d12dc36284ed0219f58e0f2a54d652834c39 Author: github-actions AuthorDate: Fri Sep 22 09:34:46 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 271656b9ba3..fcfe98054e5 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -4,7 +4,7 @@ "chamikaramj": 1695103578648, "johnjcasey": 1695212037351, "pabloem": 1691787951165, -"Abacn": 1694779650767, +"Abacn": 1695375280928, "ahmedabu98": 1695105814268, "bvolpato": 1695137718373, "manavgarg": 1690826779210
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5444d9595af Updating config from bot
5444d9595af is described below
commit 5444d9595afa564eb2011c939025b4ef8788ed84
Author: github-actions
AuthorDate: Fri Sep 22 09:34:43 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28590.json | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28590.json
b/scripts/ci/pr-bot/state/pr-state/pr-28590.json
index 242a48d7d3b..ee6facddad9 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28590.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28590.json
@@ -1,7 +1,10 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"java": "Abacn",
+"io": "Abacn"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
